Testing Python in Docker containers with Jenkins

Oct 24, 2016


As a part of the migration to a new hosting provider at MetaBrainz, we started dockerizing all of our apps. One of the problems that came up is setting up Jenkins to run Python tests inside containers and getting results from them. Here I’m going to describe how we solved this problem. Perhaps it will help you make a similar setup or at least give some ideas.

There are several things to keep in mind:

  1. Python app might depend on additional services like a database or memcache.
  2. We need to have a way to determine if tests that run inside a container pass or fail. There might be other metrics that we need to get from it.
  3. All containers created during a test run need to be shut down and removed.

The process

Making a container for testing

The first step is to create a Docker container that is going to run Python tests. In the examples I’m going to use pytest framework, but the process should be similar for other ones.

Below is the example of running pytest, which generates tests results in JUnit format and a code coverage report.

FROM python

# Application setup...

COPY . /code/ # Copy source coude with all the tests.
CMD py.test --junitxml=/data/test_report.xml \
            --cov-report xml:/data/coverage.xml

If your tests depend on another service like a database to be running, you can use the dockerize utility to wait for that service to start up. In the example below container waits for a service postgresql to start running on port 5432 with a timeout of 60 seconds (or some other). After that service has started, py.test command runs.

FROM python

RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && \
    tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
# Application setup...

COPY . /code/ # Copy source coude with all the tests.
CMD dockerize -wait tcp://postgresql:5432 -timeout 60s \
    py.test --junitxml=/data/test_report.xml \
            --cov-report xml:/data/coverage.xml

Putting dependencies together

Docker Compose can be used to simplify startup and connection of all the services that the application depends on.

version: "2"

    image: postgres

      context: .
      dockerfile: Dockerfile.test
      - postgresql

Service that will run the tests is called app_test in this example. Dockerfile defined above is stored in Dockerfile.test file and compose file is in docker-compose.test.yml.1

Running the tests

It’s time to create a Jenkins project! The type can be a “Freestyle project”, that’s the one I’ve been using. Then add an “Execute shell” build step.

Next I’ll go through each part of the shell script that is used to run tests. Complete script is at the end of this article.

docker-compose -f docker-compose.test.yml \
               -p jenkinsbuild \
               up -d --build --force-recreate
docker logs -f jenkinsbuild_app_test_1

Here’s what’s going on:

  1. We specify the project name (jenkinsbuild) to be able to reliably reference containers.2 It needs to be unique so that we don’t confuse test containers with other ones running on the same host.
  2. up command is run with -d (detached) argument so that we run containers in the background. Otherwise, if there is some other container apart from one that runs the tests and exists afterwards, it will be difficult to know when tests are finished running. For example, PostgreSQL container is not going to exit on its own.
  3. To know when tests actually finish running we can follow the log output of the container with tests - jenkinsbuild_app_test_1.

Getting the results out

In case of pytest, test results can be saved in a form of junit-xml style report (--junitxml=/data/test_report.xml part in a py.test call) which can then be copied into current workspace.3 Same can be done with a coverage report.

docker cp jenkins_app_test_1:/data/test_report.xml .
docker cp jenkins_app_test_1:/data/coverage.xml .

Now those can be specified in related build sections.

JUnit report configuration: JUnit report configuration

Coverage report configuration: Coverage report configuration


Now we just need to shut down all the containers that were created for a test run:

docker-compose -f $COMPOSE_FILE_LOC \
               -p $COMPOSE_PROJECT_NAME \
               down --remove-orphans
docker ps -a --no-trunc  | grep $COMPOSE_PROJECT_NAME \
    | awk '{print $1}' | xargs --no-run-if-empty docker stop
docker ps -a --no-trunc  | grep $COMPOSE_PROJECT_NAME \
    | awk '{print $1}' | xargs --no-run-if-empty docker rm

Bonus: Running Jenkins in a Docker container 📦

If you are going all-in Docker, then you might also want to run Jenkins in a container. There is an official image which allows an easy setup. Except now, if you are executing Jenkins builds in containers, you’ll need a way to manage Docker containers inside another Docker container…

Jérôme Petazzoni suggests4 to mount the Docker socket from a host, which worked well for me. The only thing you need to do is to create your own Dockerfile based on the Jenkins image and install Docker with Compose there.

FROM jenkins:alpine

# Parent container switches to "jenkins" user, so we need to revert that.
# Don't switch back because Docker will stop working.
USER root

RUN apk add --no-cache \
            ca-certificates \
            curl \
            openssl \
            py-pip \
    && rm -rf /var/cache/apk/*

# Installing Docker and Compose...
# See https://hub.docker.com/_/docker/ for updates.
ENV DOCKER_BUCKET get.docker.com
ENV DOCKER_SHA256 626601deb41d9706ac98da23f673af6c0d4631c4d194a677a9a1a07d7219fa0f
RUN set -x \
    && curl -fSL "https://${DOCKER_BUCKET}/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz" -o docker.tgz \
    && echo "${DOCKER_SHA256} *docker.tgz" | sha256sum -c - \
    && tar -xzvf docker.tgz \
    && mv docker/* /usr/local/bin/ \
    && rmdir docker \
    && rm docker.tgz \
    && docker -v
RUN pip install docker-compose

Installation script is mostly copied from the latest official docker image5 at the time of writing. You can check one of the dockerfiles there for updates.


At the end, we have a Jenkins project that is ready to run tests in a Docker container and save results.

This tutorial assumes that there is only one Jenkins build running at a time on one host. I imagine running multiple builds on the same host will not work as well with the setup that is described here.

Appendix: Bash script for Jenkins

This is the complete version of the bash script for the “Execute shell” build step in Jenkins:

#!/usr/bin/env bash

# Modify these two as needed:


# Project name is sanitized by Compose, so we need to do the same thing.
# See https://github.com/docker/compose/issues/2119.
COMPOSE_PROJECT_NAME=$(echo $COMPOSE_PROJECT_NAME_ORIGINAL | awk '{print tolower($0)}' | sed 's/[^a-z0-9]*//g')

# Record installed version of Docker and Compose with each build
echo "Docker environment:"
docker --version
docker-compose --version

function cleanup {
    # Shutting down all containers associated with this project
    docker-compose -f $COMPOSE_FILE_LOC \
                   -p $COMPOSE_PROJECT_NAME \
                   down --remove-orphans
    docker ps -a --no-trunc  | grep $COMPOSE_PROJECT_NAME \
        | awk '{print $1}' | xargs --no-run-if-empty docker stop
    docker ps -a --no-trunc  | grep $COMPOSE_PROJECT_NAME \
        | awk '{print $1}' | xargs --no-run-if-empty docker rm   

function run_tests {
    # Create containers
    docker-compose -f $COMPOSE_FILE_LOC \
                   -p $COMPOSE_PROJECT_NAME \
                   up -d --build --force-recreate

    # List images and containers related to this build
    docker images | grep $COMPOSE_PROJECT_NAME | awk '{print $0}'
    docker ps -a | grep $COMPOSE_PROJECT_NAME | awk '{print $0}'

    # Follow the container with tests...
    docker logs -f $TEST_CONTAINER_REF

function extract_results {
    docker cp ${TEST_CONTAINER_REF}:/data/test_report.xml .
    docker cp ${TEST_CONTAINER_REF}:/data/coverage.xml .

set -e
cleanup            # Initial cleanup
trap cleanup EXIT  # Cleanup after tests finish running


Master Jenkins

  1. I added .test part into Dockerfile and Compose file names in order to not confuse them with other Docker-related files that are used for other purposes (development, production, etc.).
  2. Otherwise Compose uses a directory name, which can be hard to determine. [return]
  3. Putting results into a volume doesn’t work quite well with Jenkins since Docker runs containers as root. There are some tricks to assign another owner, but they seemed way too hacky for my taste. [return]
  4. https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ [return]
  5. Check links to dockerfiles at https://hub.docker.com/_/docker/. [return]