[Python for DevOps] Infrastructure testing using `pytest` plugin `Testinfra`!
August 10, 2020
In the last post, we discussed about using
pytest module for writing traditional software tests.
Here, we will be expanding the same for performing Infrastructure testing using
(Note: The notes below are the direct excerpts from the book, Python For DevOps, compiled for the purpose of learning & memory.)
Infrastructure Testing (my favorite part :D)
Testinfra project is a
pytest plug-in for infrastructure testing that relies heavily on fixtures and allows you to write Python tests as if testing code.
The way we explain infrastructure testing is by asking a question: How can you tell that the deployment was successful?
Most of the time, this means some manual checks, such as loading a website or looking at processes, which is insufficient; it is error-prone and can get tedious if the system is significant.
TestInfraproject has all kinds of fixtures to test a system efficiently, and it includes a complete set of backends to connect to servers, regardless of their deployment type: Ansible, Docker, SSH, and Kubernetes are some of the supported connections. By supporting many different connection backends, you can execute the same set of tests regardless of infrastructure changes.
System validation can happen at different levels (with monitoring and alert systems) and at different stages in the life cycle of an application, such as during pre-deployment, at runtime, or during deployment. An application that Alfredo recently put into production needed to handle client connections gracefully without any disruption, even when restarted. To sustain traffic, the application is load balanced: when the system is under heavy loads, new connections get sent to other servers with a lighter load.
When a new release gets deployed, the application has to be restarted. Restarting means that clients experience an odd behavior at best, or a very broken experience at the worst. To avoid this, the restart process waits for all client connections to terminate, the system refuses new connections, allowing it to finish work from existing clients, and the rest of the system picks up the work. When no connections are active, the deployment continues and stops services to get the newer code in. There is validation at every step of the way:
- before the deployment to tell the balancer to stop sending new clients
- and later, verifying that no new clients are active.
If that workflow converts to a test, the title could be something like: make sure that no clients are currently running.
Once the new code is in,
- another validation step checks whether the balancer has acknowledged that the server is ready to produce work once again.
- Another test here could be: balancer has server as active.
- Finally, it makes sure that the server is receiving new client connections—yet another test to write!
Throughout these steps, verification is in place, and tests can be written to verify this type of workflow.
- Create a new virtual environment
validation, and install pytest.
$ python3 -m venv validation $ source testing/bin/activate (validation) $ pip install pytest
testinfra, ensuring that version 2.1.0 is used.
(validation) $ pip install "testinfra==2.1.0"
Because different backend connection types exist, when the connection is not specified directly, Testinfra defaults to certain ones. It is better to be explicit about the connection type and define it in the command line.
- These are all the connection types that Testinfra supports:
- Paramiko (an SSH implementation in Python)
- Kubernetes (via kubectl)
testinfrasection appears in the help menu with some context on the flags that are provided. This is a neat feature from
pytestand its integration with Testinfra. The help for both projects comes from the same command:
(validation) $ pytest --help ... testinfra: --connection=CONNECTION Remote connection backend (paramiko, ssh, safe-ssh, salt, docker, ansible) --hosts=HOSTS Hosts list (comma separated) --ssh-config=SSH_CONFIG SSH config file --ssh-identity-file=SSH_IDENTITY_FILE SSH identify file --sudo Use sudo --sudo-user=SUDO_USER sudo user --ansible-inventory=ANSIBLE_INVENTORY Ansible inventory file --nagios Nagios plugin
Let’s try to understand the infrastructure testing using
pytest with an example.
- Say, there are two servers up and running. To demonstrate the connection options, let’s check if they are running CentOS 7 by poking inside the
/etc/os-releasefile. This is how the test function looks (saved as test_remote.py).
def test_release_file(host): release_file = host.file("/etc/os-release") assert release_file.contains('CentOS') assert release_file.contains('VERSION="7 (Core)"')
- It is a single test function that accepts the
hostfixture, which runs against all the nodes specified.
--hostsflag accepts a list of hosts with a connection scheme (SSH would use
ssh://hostnamefor example), and some other variations using globbing are allowed.
- If we’re testing against more than a couple of remote servers at a time, passing the hosts on the command line becomes cumbersome. This is how it would look to test against two servers using SSH
(validation) $ pytest -v --hosts='ssh://node1,ssh://node2' test_remote.py ============================= test session starts ============================= platform linux -- Python 3.6.8, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 cachedir: .pytest_cache rootdir: /home/alfredo/python/python-devops/samples/chapter16 plugins: testinfra-3.0.0, xdist-1.28.0, forked-1.0.2 collected 2 items test_remote.py::test_release_file[ssh://node1] PASSED [ 50%] test_remote.py::test_release_file[ssh://node2] PASSED [100%] ========================== 2 passed in 3.82 seconds ===========================
- The increased verbosity (with the -v flag) shows that Testinfra is executing the one test function in the two remote servers specified in the invocation.
When setting up the hosts, it is important to have a passwordless connection. There shouldn’t be any password prompts, and if using SSH, a key-based configuration should be used.
Testinfra can consume an SSH configuration file to determine what hosts to connect to. For the previous test run, Vagrant was used, which created these servers with special keys and connection settings. Vagrant can generate an ad-hoc SSH config file for the servers it has created.
(validation) $ vagrant ssh-config Host node1 HostName 127.0.0.1 User vagrant Port 2200 UserKnownHostsFile /dev/null StrictHostKeyChecking no PasswordAuthentication no IdentityFile /home/alfredo/.vagrant.d/insecure_private_key IdentitiesOnly yes LogLevel FATAL Host node2 HostName 127.0.0.1 User vagrant Port 2222 UserKnownHostsFile /dev/null StrictHostKeyChecking no PasswordAuthentication no IdentityFile /home/alfredo/.vagrant.d/insecure_private_key IdentitiesOnly yes LogLevel FATAL
- Exporting the contents of the above output to a file and then passing that to
Testinfraas a flag offers greater flexibility if using more than one host.
(validation) $ vagrant ssh-config > ssh-config (validation) $ pytest --hosts=default --ssh-config=ssh-config test_remote.py
--hosts=defaultavoids having to specify them directly in the command line, and the engine feeds from the SSH configuration.
Ansible is another option if the nodes are local, SSH, or Docker containers. The test setup can benefit from using an inventory of hosts (much like the SSH config), which can group the hosts into different sections. The host groups can also be specified so that you can single out hosts to test against, instead of executing against all.
- For node1 and node2 used in the previous example, this is how the inventory file is defined (and saved as
[all] node1 node2
- If executing against all of them, the command changes to:
$ pytest --connection=ansible --ansible-inventory=hosts test_remote.py
- If defining other hosts in the inventory that need an exclusion, a group can be specified as well. Assuming that both nodes are web servers and are in the nginx group, this command would run the tests on only that one group:
$ pytest --hosts='ansible://nginx' --connection=ansible \ --ansible-inventory=hosts test_remote.py
A lot of system commands require superuser privileges. To allow escalation of privileges, Testinfra allows specifying
--sudoflag makes the engine use sudo when executing the commands, while the
--sudo-usercommand allows running with higher privileges as a different user.The fixture can be used directly as well.
Features and Special Fixtures
So far, the host fixture is the only one used in examples to check for a file and its contents. However, this is deceptive. The host fixture is an all-included fixture; it contains all the other powerful fixtures that Testinfra provides. This means that the example has already used the host.file, which has lots of extras packed in it. It is also possible to use the fixture directly.
In : import testinfra In : host = testinfra.get_host('local://') In : node_file = host.file('/tmp') In : node_file.is_directory Out: True In : node_file.user Out: 'root'
- The all-in-one
hostfixture makes use of the extensive API from Testinfra, which loads everything for each host it connects to.
(Check all the attributes available here.)
- The below are some of the most used ones.
host.ansible: Provides full access to any of the Ansible properties at runtime, such as hosts, inventory, and vars.
host.addr: Network utilities, like checks for IPV4 and IPV6, is host reachable, is host resolvable.
host.docker: Proxy to the Docker API, allows interacting with containers, and checks if they are running.
host.interface: Helpers for inspecting addresses from a given interface.
host.iptables: Helpers for verifying firewall rules as seen by host.iptables.
host.mount_point: Check mounts, filesystem types as they exist in paths, and mount options.
host.package: Very useful to query if a package is installed and at what version.
host.process: Check for running processes.
host.sudo: Allows you to execute commands with host.sudo or as a different user.
host.system_info: All kinds of system metadata, such as distribution version, release, and codename.
host.check_output: Runs a system command, checks its output if runs successfully, and can be used in combination with
host.run: Runs a command, allows you to check the return code,
host.run_expect: Verifies that the return code is as expected.
A frictionless way to start developing system validation tests is to do so while creating the actual deployment. Somewhat similar to Test Driven Development (TDD), any progress warrants a new test.
Here, a web server needs to be installed and configured to run on port 80 to serve a static landing page.
With a vanilla Ubuntu server, start by installing the Nginx package:
$ sudo apt install nginx
- Create a new test file called
test_webserver.pyfor adding new tests after making progress. After Nginx installs, let’s create another test:
def test_nginx_is_installed(host): assert host.package('nginx').is_installed
- Reduce the verbosity in
pytestoutput with the
-qflag to concentrate on failures. The remote server is called
node4and SSH is used to connect to it. This is the command to run the first test.
(validate) $ pytest -q --hosts='ssh://node4' test_webserver.py . 1 passed in 1.44 seconds ## YOU COULD TEST THE SAME ON LOCAL MACHINE WITH THE FOLLOWING COMMAND: (validate) $ pytest -q test_webserver.py . [100%] 1 passed in 0.20s
- The web server needs to be up and running, so a new test is added to verify that behavior.
def test_nginx_is_running(host): assert host.service('nginx').is_running
- The web server should be serving a static landing page on
port 80. Adding another test (in
test_webserver.py) to verify the port is the next step.
- This test is more involved and needs attention to some details.
- It opts to check for TCP connections on port 80 on any IP in the server.
def test_nginx_listens_on_port_80(host): assert host.socket("tcp://0.0.0.0:80").is_listening
- Since there isn’t a built-in fixture to handle HTTP requests to an address, the final test uses the wget utility to retrieve the contents of the running website and make assertions on the output to ensure that the static site renders.
def test_get_content_from_site(host): output = host.check_output('wget -qO- 0.0.0.0:80') assert 'Welcome to nginx' in output
- So, the final run will give the output like:
(validate) $ pytest -v --hosts='ssh://node4' test_webserver.py ================================ test session starts ================================ platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/priyankasaggu119/Desktop/.myhome/python-study plugins: testinfra-2.1.0 collected 4 items test_webserver.py::test_nginx_is_installed[local] PASSED [ 25%] test_webserver.py::test_nginx_is_running[local] PASSED [ 50%] test_webserver.py::test_nginx_listens_on_port_80[local] PASSED [ 75%] test_webserver.py::test_get_content_from_site[local] PASSED [100%] ================================= 4 passed in 0.22s =================================
Testing Jupyter Notebooks with pytest [PENDING]
One easy way to introduce big problems into your company is to forget about applying software engineering best practices when it comes to data science and machine learning. One way to fix this is to use the
nbval (notebook-validation) plug-in for
pytest that allows you to test your notebooks. Take a look at this Makefile:
setup: python3 -m venv ~/.myrepo install: pip install -r requirements.txt test: python -m pytest -vv --cov=myrepolib tests/*.py python -m pytest --nbval notebook.ipynb lint: pylint --disable=R,C myrepolib cli web all: install lint test
- The key item is the
--nbvalflag that also allows the notebook in the repo to be tested by the build server.