OpenStack Unit Testing: a Long Journey Starts With a Small Step

This time I would like to talk about OpenStack unit tests we were working on at Grid Dynamics.

First and furthermost, writing unit tests for OpenStack is a tar pit: if you have started, you cannot resist. The task is really difficult by itself; moreover, creating tests for such quickly developed project is like a life on a volcano.

Our task seemed to be quite trivial. We have a set of projects with different test coverage. We have to cover them at least for 80%. Python coverage utility is at our service.

I have introduced the following infrastructure. We publish our patches on Gerrit that replicates its repositories to another server for easy recovering. Jenkins monitors Gerrit and recalculates coverage after every patch publication. Coverage for each project is gathered to table.

I hoped that coverage calculation is not a big deal, but I was too optimistic. Imagine that we simply use run_tests.sh script that is available for almost all the projects. Than we will face some problems.

1. One does not simply use run_tests.sh. It installs a fresh virtual environment for Python packages using corresponding pip-requires and test-requires files. This environment lacks some necessary packages (e.g., iso8601) and its eventlet is incompatible with Python 2.6 that we use in CentOS. So, I had to prepare a virtual environment with a custom script. So does OpenStack Jenkins.
2. It usually takes a lot of time to download and compile all Python dependencies. It was too long even for a Gentoo user, so I cached the environment using a common env for all the packages.
3. pip is not a reliable utility. During package installation, it is used to erasing all package data except for metainformation. The last point means that the package is considered to be installed while its files are gone, so, you have to uninstall it and install again.

Well, now we have a shiny virtual environment. It’s time to run tests and calculate coverage. Currently, OpenStack unit tests for Nova, Glance, Quantum, and Keystone are slow and run about 10-20 minutes on our testing server. There is a good idea to migrate for nosetests to testr: this will allow to run the tests in parallel (the thing is that nosetests parallelization is incompatible with eventlet/greenlet widely used in OpenStack). Nova already uses testr, but updating other packages is a laborious task. Some tests depend on each other and cannot be run in parallel, here is an example of one such problem I’ve detected and fixed: https://bugs.launchpad.net/quantum/+bug/1125951.

I have used incremental testing to increase the speed. I cache .coverage file that accumulates statistics for previous testing and run only those tests that were added or updated. The latter are determined using git diff --name-only command. Also, incremental testing solves another problem: summarize coverage for all commits for given project. These commits are siblings on commit tree, so, you cannot checkout to the latest one and run all published tests that are not accepted yet.

OpenStack includes the Oslo project – a set of python libraries shared by other projects. Unfortunately, Oslo is not a true library now: its code it copied with a special script to other projects. Thus our coverage statistics would be incorrect if we include uncovered Oslo code to common report. So, I had to implement a blueprint to fix this issue.

All preparations are done, now we are ready for testing!

Our testing process differs from one used in OpenStack Jenkins. It happens that upstream tests fail just because they were not run during verification, but sometimes they pass somehow in OpenStack jobs and fail in my environment. Here is an example: Quantum’s test_policy passes in https://review.openstack.org/#/c/21091/, but they should fail because one necessary configuration parameter is not imported properly (my fix: https://review.openstack.org/#/c/21205/).

So, now OpenStack community has a new team that looks at unit tests from slightly different point of view detecting problems and fixing them.