W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
and why you should assume every change breaks your code
living knowledge
WWU Münster
René Milk
4th November 2015
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
2 /16
Not talking about C ∞ functions
I
completely automatic triggering (nightly, on branch updates, ...)
I
automatic reporting
I
simple visualization
I
wide accessibility
I
metrics
living knowledge
WWU Münster
Continuous
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
3 /16
No quadrature rules contained in this talk either
I
does your software stack still build after changes?
I
compile, run, link, work as a dependency for other software?
I
how about on system X?
living knowledge
WWU Münster
Integration
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
4 /16
and ofc no weak solutions for PDEs
I
automated, repeatable, deterministic, reliable
I
Unit Testing
I
System Testing
I
Performance Testing
living knowledge
WWU Münster
Testing
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
5 /16
I
saves time, no need to argue about possible effects of changes
I
It’s better to err on the side of caution. You will break code from time to time.
I
software is modular, codebases are huge, dependency graphs are complex
I
distributed development: multiple people, branches, features/fixes, ...
I
http://whatever.scalzi.com/2010/06/16/the-failure-state-of-clever/
living knowledge
WWU Münster
assume your every change breaks everybody’s code
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
6 /16
I
free for FOSS
I
integrates with GitHub repositories
I
explicitly supports C++, python, ruby, but if your test/software runs on a standard-ish
Ubuntu 12.04 you’re good
I
triggers on branch push, visual feedback in pull requests and badges
I
very easy to create different configurations
living knowledge
WWU Münster
Tools for CI: Travis
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
7 /16
language: python
python:
- "2.7"
script:
- DISPLAY=:99.0 py.test -k "${PYTEST_MARKER}"
install:
- python setup.py build_ext -i
notifications:
email:
on_success: change
on_failure: change
after_success:
- coveralls
branches:
except:
- gh-pages
living knowledge
WWU Münster
Tools for CI: Travis
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
8 /16
sudo: false
language: cpp
matrix:
include:
- os: linux
compiler: gcc
addons: &gcc49
apt:
sources: [’ubuntu-toolchain-r-test’]
packages: [’g++-4.9’, ’gcc-4.9’]
env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9
- os: linux
compiler: gcc
addons: *gcc49
env: CXX_COMPILER=g++-4.9 COMPILER=gcc-4.9 DELETE="dune-istl"
living knowledge
WWU Münster
Tools for CI: Travis
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
9 /16
living knowledge
WWU Münster
Tools for CI: Travis
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
10 /16
living knowledge
WWU Münster
Tools for CI: Travis
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
11 /16
I
Python framework to build CI systems, not a ready made solution
I
Master/Slave setup
I
good for testing complex stacks on diverse architectures
I
much more freedom than on Travis, at a much higher maintenance and setup cost
living knowledge
WWU Münster
Tools for CI: Buildbot
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
12 /16
I
setup to deploy new masters with only 5 LOC
I
buildbot code for that to work is 2300 LOC
I
git repository tracks individual DUNE-modulesŕepositories -> dynamic build steps ->
make test, result log
I
tied into redmine
I
email on buildstatus change, filter by user in change
living knowledge
WWU Münster
Tools for CI: Buildbot - Example
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
13 /16
from common import config, passwords, genconfig
config.CURRENT_DB = passwords.dune_db_url
import redmine.db.connection
redmine.db.connection.setup(config.CURRENT_DB)
BuildmasterConfig = genconfig.genconfig(
"http://users.../dune-stuff-demos.git",
project="dune-stuff", base_port=11020, use_cmake=True)
living knowledge
WWU Münster
Tools for CI: Buildbot - Example (master config)
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
14 /16
[clang_3.5]
file = config.opts/clang-3.5
cc = clang-3.5
[gcc_5.0]
file = config.opts/gcc-snapshot
cc = gcc-snapshot
[variants]
# name = semicolon seperated list of directory names
# to delete prior to buildbot config
no_fem = dune-fem
minimal = dune-grid;dune-localfunctions;dune-fem;
dune-typetree;dune-pdelab;dune-istl
living knowledge
WWU Münster
Tools for CI: Buildbot - Example (buildbot.opts)
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
15 /16
living knowledge
WWU Münster
Buildbot
,
,
[email protected]
W ESTFÄLISCHE
W ILHELMS -U NIVERSITÄT
M ÜNSTER
Continuous Integration/Testing
16 /16
Testing: What?
Every line of code is unit tested, with function level granularity. System tests cover all
possible software configurations and inputs. Performance testing is run on a wide variety of
benchmark problems, in a restricted, possibly virtualized, environment. All of that on a very
diverse set of computer architectures, operating systems, support library versions.
Not viable in the real world. Configuration space is just too big. Compute time, and
available hardware is limited. Test cycles need to be short to not impede
development.
living knowledge
WWU Münster
utopic
,
,
[email protected]
Continuous Integration/Testing
17 /16
Testing: What?
realistic
Every non-trivial piece of code is unit tested, with algorithm level granularity. System tests
cover the most common configuration options. Benchmark runs are recorded.
,
,
[email protected]
Continuous Integration/Testing
18 /16
Testing: What?
minimal
The code is run for a benchmark problem with known/analytic solution.
,
,
[email protected]
Continuous Integration/Testing
19 /16
Testing: How?
I
Use a test harness: google-test, Boost.Test, py.test
Simplifies selective test running, result gathering, behavior expectation
I
Writing a DUNE module? Use dune-testtools for simple test parameterization
I
record algorithm results, error norms and compare in system tests
I
Make sure exceptions get raised on invalid data input
I
integrate into build system (ie make test)
,
,
[email protected]
Continuous Integration/Testing
20 /16
Testing: How? - Gtest Example
typedef testing::Types<YaspGrid<1>,
YaspGrid<4> Grids;
template <class T>
struct CornerRangeTest : public ::testing::Test {
CornerRangeTest() : grid_prv(0., 1., level) {}
void check() {
for (auto it : grid_prv.grid().leafGridView())
check_range(it, cornerRange(it->geometry()));
for (auto v : DSC::valueRange(T::dimensionworld))
EXPECT_GE(v, 0);
}
};
TYPED_TEST_CASE(CornerRangeTest, Grids);
TYPED_TEST(CornerRangeTest, Misc) { this->check(); }
,
,
[email protected]
Continuous Integration/Testing
21 /16
Testing: What tests are allowed to fail
NONE
except those where the platform requirements cannot be met
,
,
[email protected]
Continuous Integration/Testing
22 /16
Testing: What else is there?
I
Using cmake? Make ctest submit test results to cdash
,
,
[email protected]
Continuous Integration/Testing
23 /16
Thank you for
your attention.
Slides available at http://work.milk.pm
,
,
[email protected]