Recently I worked on improving the testing and the Continuos Integration (CI) configuration for a few open-source projects. In particular, I have spent some time on WorldEngine, a world generator written in Python, which uses a C++ extension named plate-tectonics.

There have been several issues, the main two are:

  • the deployment of the application on Windows is problematic
  • the application does not behave in the exact some way on Linux and Mac OS-X

To mitigate these issues I invested some time in writing better tests and improve my usage of Travis (CI for Linux) and start using AppVeyor (CI for Windows). While my solution is still not perfect I feel I am far better covered from regressions on the different platforms and I have a more reliable development process.

Travis

Travis is well-known in the open-source community because of three nice qualities:

  1. It is free for open-source projects
  2. It integrates seamlessly with GitHub
  3. It is very easy to use

Getting started with Travis is very easy: you simply register and connect your GitHub profile, You then select on which projects you want to activate Travis.

At this point you will see a list of your projects. The green or red color used for the project names make immediately evident which projects need to be fixed. You can also take a look at a specific build and see what caused it to fail.

Screen Shot 2015-03-02 at 11.09.46

Every time you push to Github, whether the master or in another branch, a build is started. If the branch you are building is used in a pull request, a badge indicating if the build failed or succeeded will appear in the GitHub interface.

Screen Shot 2015-03-03 at 10.17.11

Setting up Different C++ Compilers for Travis

While having your tests all passing on one platform is good, having them passing on different platforms is great. For example, it is a very good thing to verify if you C++ code compiles correctly both with gcc and clang. This is particularly important if support for C++’11 is needed and it can be affected by using the wrong version of the compiler. You can do that by creating a .travis.yml file containing these lines:

language: cpp
compiler:
    - gcc
    - clang

Now, what you want to do typically is to install specific versions of the compilers, to have a completely controlled environment, and not just using whatever happens to be installed on the machine Travis is offering you. Doing that is pretty simple, you just use apt-get to install whatever you need to install.

before_install:
    - echo "yes" | sudo add-apt-repository ppa:kalakris/cmake
    - echo "yes" | sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
    - sudo apt-get update -qq
    - sudo apt-get install -qq
    - sudo apt-get install cmake
    # g++4.8.1
    - if [ "$CXX" == "g++" ]; then sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test; fi

    # clang 3.4
    - if [ "$CXX" == "clang++" ]; then sudo add-apt-repository -y ppa:h-rayflood/llvm; fi

    - sudo apt-get update -qq

install:
  # g++4.8.1
  - if [ "$CXX" = "g++" ]; then sudo apt-get install -qq g++-4.8; fi
  - if [ "$CXX" = "g++" ]; then export CXX="g++-4.8" CC="gcc-4.8"; fi
  - if [ "$CXX" = "g++" ]; then sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 90; fi
  - if [ "$CXX" = "g++" ]; then sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 90; fi
  - if [ "$CXX" = "g++" ]; then sudo apt-get install libstdc++-4.8-dev; fi

  # clang 3.4
  - if [ "$CXX" == "clang++" ]; then sudo apt-get install --allow-unauthenticated -qq clang-3.4; fi
  - if [ "$CXX" == "clang++" ]; then export CXX="clang++-3.4"; fi

Given you are a smart guy, I am sure you can adapt this example to your specific case.

Setting Up Different Python Versions for Travis

Let’s build our application with a few different versions of Python:

language: python
python:
  - "2.6"
  - "2.7"
  - "3.2"
  - "3.3"
  - "3.4"

In this case we do not manually install Python, but we rely on Travis having the correct versions already installed. To find out more about Python on travis read the Travis documentation about Python support.

A useful trick is to use a different requirements file depending on the python version (damn you Python 2!):

install:
  - if [[ $TRAVIS_PYTHON_VERSION == 2* ]]; then pip install --allow-all-external -r requirements2.txt; fi
  - if [[ $TRAVIS_PYTHON_VERSION == 3* ]]; then pip install --allow-all-external -r requirements3.txt; fi

My Travis Files

These are a couple of complete Travis files I am using.

 plate-tectonics (C++)

language: cpp
compiler:
    - gcc
    - clang

before_install:
    - echo "yes" | sudo add-apt-repository ppa:kalakris/cmake
    - echo "yes" | sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
    - sudo apt-get update -qq
    - sudo apt-get install -qq
    - sudo apt-get install cmake
    # g++4.8.1
    - if [ "$CXX" == "g++" ]; then sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test; fi

    # clang 3.4
    - if [ "$CXX" == "clang++" ]; then sudo add-apt-repository -y ppa:h-rayflood/llvm; fi

    - sudo apt-get update -qq

install:
  # g++4.8.1
  - if [ "$CXX" = "g++" ]; then sudo apt-get install -qq g++-4.8; fi
  - if [ "$CXX" = "g++" ]; then export CXX="g++-4.8" CC="gcc-4.8"; fi
  - if [ "$CXX" = "g++" ]; then sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 90; fi
  - if [ "$CXX" = "g++" ]; then sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 90; fi
  - if [ "$CXX" = "g++" ]; then sudo apt-get install libstdc++-4.8-dev; fi

  # clang 3.4
  - if [ "$CXX" == "clang++" ]; then sudo apt-get install --allow-unauthenticated -qq clang-3.4; fi
  - if [ "$CXX" == "clang++" ]; then export CXX="clang++-3.4"; fi   

script: $CXX --version && cmake . -G 'Unix Makefiles' && make && cd test && make && ./PlateTectonicsTests

WorldEngine (Python, using C++ extensions)

language: python
python:
  - "2.6"
  - "2.7"

before_install:
    - echo "yes" | sudo add-apt-repository ppa:kalakris/cmake
    - echo "yes" | sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
    - sudo apt-get update -qq
    - sudo apt-get install -qq
    - sudo apt-get install cmake
    # g++4.8.1
    - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
    - sudo apt-get update -qq

install:
  # g++4.8.1
  - sudo apt-get install -qq g++-4.8
  - export CXX="g++-4.8" CC="gcc-4.8"
  - sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 90
  - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 90
  - sudo apt-get install libstdc++-4.8-dev
  - echo "CXX=$CXX"
  - echo "version `$CXX --version`"
  - if [[ $TRAVIS_PYTHON_VERSION == 2* ]]; then pip install --allow-all-external -r requirements2.txt; fi
  - if [[ $TRAVIS_PYTHON_VERSION == 3* ]]; then pip install protobuf-py3==2.5.1 && pip install --allow-all-external --allow-unverified protobuf-py3 -r requirements3.txt; fi
  - sudo python setup.py develop

# # command to run tests
script: nosetests

Civs (Clojure)

language: clojure
lein: lein2
script: lein2 test :all

Javaparser (Java)

language: java
install: mvn install -DskipTests=true
script: mvn test
jdk:
  - oraclejdk8
notifications:
  webhooks:
    urls:
      - "https://webhooks.gitter.im/e/ec3127975d8a2b8f11d0"
    on_success: always  # options: [always|never|change] default: always
    on_failure: always  # options: [always|never|change] default: always

# http://lint.travis-ci.org/ validator

Ok, you got the picture. Travis is awesome and you can use it with a lot of different languages. This is easy to get started with (see the Civs script) but it is also flexible and powerful, if you need it to be.

AppVeyor

Recently I was very bugged by problems about building plate-tectonics and WorldEngine on Windows. Luckily Bret (who is maintaining WorldEngine with me) pointed me to AppVeyor which is basically Travis for Windows.

This is out we configured it for our project, so that it can build our library on 6 different versions of Python:

  • Python 2.7, 32 bit
  • Python 2.7, 64 bit
  • Python 3.3, 32 bit
  • Python 3.3, 64 bit
  • Python 3.4, 32 bit
  • Python 3.4, 64 bit
# Taken from: https://packaging.python.org/en/latest/appveyor.html
# and from: https://bitbucket.org/pygame/pygame/pull-request/45/create-python-wheel-builds-using-appveyor/diff
# For now considering only Python 2.7, looking to support soon Python 2.6
# and Python 3 in the future

environment:

  global:
    # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
    # /E:ON and /V:ON options are not enabled in the batch script intepreter
    # See: http://stackoverflow.com/a/13751649/163740
    WITH_COMPILER: "cmd /E:ON /V:ON /C .appveyorrun_with_compiler.cmd"

  matrix:
    - PYTHON: "C:Python27"
      PYTHON_VERSION: "2.7.8"
      PYTHON_ARCH: "32"
      DISTRIBUTIONS: "sdist bdist_wheel"

    - PYTHON: "C:Python33"
      PYTHON_VERSION: "3.3.5"
      PYTHON_ARCH: "32"
      DISTRIBUTIONS: "bdist_wheel"

    - PYTHON: "C:Python34"
      PYTHON_VERSION: "3.4.1"
      PYTHON_ARCH: "32"
      DISTRIBUTIONS: "bdist_wheel"

    - PYTHON: "C:Python27-x64"
      PYTHON_VERSION: "2.7.8"
      PYTHON_ARCH: "64"
      WINDOWS_SDK_VERSION: "v7.0"
      DISTRIBUTIONS: "sdist bdist_wheel"

    - PYTHON: "C:Python33-x64"
      PYTHON_VERSION: "3.3.5"
      PYTHON_ARCH: "64"
      WINDOWS_SDK_VERSION: "v7.1"
      DISTRIBUTIONS: "bdist_wheel"

    - PYTHON: "C:Python34-x64"
      PYTHON_VERSION: "3.4.1"
      PYTHON_ARCH: "64"
      WINDOWS_SDK_VERSION: "v7.1"
      DISTRIBUTIONS: "bdist_wheel"

init:
  - "ECHO %PYTHON% %PYTHON_VERSION% %PYTHON_ARCH%"

install:
  - "powershell appveyorinstall.ps1"
  - "set HOME=%APPVEYOR_BUILD_FOLDER%"

build: off

test_script:
  - "%WITH_COMPILER% %PYTHON%/python setup.py test"

after_test:
  - "%WITH_COMPILER% %PYTHON%/python setup.py %DISTRIBUTIONS%"

artifacts:
  - path: dist*

One feature of AppVeyor is really great: it store artifacts generating during the build.

Screen Shot 2015-03-03 at 10.07.42

We use AppVeyor to generate the Windows binary wheels for our library and then we upload them on Pypi. The project of uploading the files on Pypi is manual at the moment because we want to have some control on it (we do not want just to upload a new version everytime we do a push).

Badges

Badges are nice, and permit to check the status of your project.

It could sound childish but those small virtual stickers can motivate you to fix problems as they arise, because you want to be proud of your green status, on both Travis and AppVeyor.

Screen Shot 2015-03-03 at 10.28.23

Conclusions and Thoughts for Improvements

I think that CI integrations is fundamental to give solid basis to your projects. I sleep better since I have start using it.

However there is still a lot of room for improvements and a few ideas:

  • I am still missing a CI solution for Mac OS-X
  • I could use docker under Travis to verify the building process under different distros
  • I sometimes use Travis in a trial and error fashion: if I do not have access to a Windows machine I just cut a separate branch and push furiously to it, to trigger builds on AppVeyor and collecting feedback on the building process under Windows. This seems silly… but it works for me :)

Bonus

If you want to use Travis with Perl you can read Try Travis CI with your CPAN distributions.

Building Binary Wheels for Windows using Appveyor is an interesting reading for Python developers targeting Windows users.