FIXME: Write the chapter text.

For now, see https://github.com/networkupstools/jenkins-dynamatrix sources (note the README and large comments at start of files may be obsolete, as of this writing — documenting the initial ideas, but the implementation might differ from that over time).

FIXME: Write the chapter text.

For now, see the Jenkinsfile-dynamatrix in the NUT sources (maybe only git), e.g. https://github.com/networkupstools/nut/blob/master/Jenkinsfile-dynamatrix for the practical pipeline preparation and hand-off to library implementation.

Since mid-2021, the NUT CI farm is implemented by several virtual servers courteously provided originally by Fosshost and later by DigitalOcean.

These run various operating systems as build agents, and a Jenkins instance to orchestrate the builds of NUT branches and pull requests on those agents.

This is driven by Jenkinsfile-dynamatrix and a Jenkins Shared Library called jenkins-dynamatrix which prepares a matrix of builds across as many operating systems, bitnesses/architectures, compilers, make programs and C/C++ revisions as it can — based on the population of currently available build agents and capabilities which they expose as agent labels.

This hopefully means that people interested in NUT can contribute to the build farm (and ensure NUT is and remains compatible with their platform) by running a Jenkins Swarm agent with certain labels, which would dial into https://ci.networkupstools.org/ controller. Please contact the NUT maintainer if you want to participate in this manner.

The Jenkinsfile-dynamatrix recipe allows NUT CI farm to run different sets of build scenarios based on various conditions, such as the name of branch being built (or PR’ed against), changed files (e.g. C/C++ sources vs. just docs), and some build combinations may be not required to succeed.

For example, the main development branch and pull requests against it must cleanly pass all specified builds and tests on various platforms with the default level of warnings specified in the configure script. These are balanced to not run too many build scenarios overall, but just a quick and sufficiently representative set.

As another example, there is special handling for "fightwarn" pattern in the branch names to run many more builds with varying warning levels and more variants of intermediate language revisions, and so expose concerns deliberately missed by default warnings levels in "master" branch builds (the bar moves over time, as some classes of warnings become extinct from our codebase).

Further special handling for branches named like fightwarn.*89.* regex enables more intensive warning levels for a GNU89 build specifically (which are otherwise disabled as noisy yet not useful for supported C99+ builds), and is intended to help develop fixes for support of this older language revision, if anyone would dare.

Many of those unsuccessful build stages are precisely the focus of the "fightwarn" effort, and are currently marked as "may fail", so they end up as "UNSTABLE" (seen as orange bubbles in the Jenkins BlueOcean UI, or orange cells in the tabular list of stages in the legacy UI), rather than as "FAILURE" (red bubbles) for build scenarios that were not expected to fail and usually represent higher-priority problems that would block a PR.

Developers whose PR builds (or attempts to fix warnings) did not succeed in some cell of such build matrix, can look at the individual logs of that cell. Beside indication from the compiler about the failure, the end of log text includes the command which was executed by CI worker and can be reproduced locally by the developer, e.g.:

22:26:01  FINISHED with exit-code 2 cmd:  (
22:26:01  [ -x ./ci_build.sh ] || exit
22:26:01
22:26:01  eval BUILD_TYPE="default-alldrv" BUILD_WARNOPT="hard" \
    BUILD_WARNFATAL="yes" MAKE="make"  CC=gcc-10 CXX=g++-10 \
    CPP=cpp-10 CFLAGS='-std=gnu99 -m64' CXXFLAGS='-std=gnu++11 -m64' \
    LDFLAGS='-m64' ./ci_build.sh
22:26:01  )

or for autotools-driven scenarios (which prep, configure, build and test in separate stages — so for reproducing a failed build you should also look at its configuration step separately):

22:28:18  FINISHED with exit-code 0 cmd:  ( [ -x configure ] || exit; \
    eval  CC=clang-9 CXX=clang++-9 CPP=clang-cpp-9 CFLAGS='-std=c11 -m64' \
    CXXFLAGS='-std=c++11 -m64' LDFLAGS='-m64' time ./configure )

To re-run such scenario locally, you can copy the line from eval (but without the eval keyword itself) up to and including the executed script or tool, into your shell. Depending on locally available compilers, you may have to tweak the CC, CXX and CPP arguments; note that a CPP may be specified as /path/to/CC -E for GCC and CLANG based toolkits at least, if they lack a standalone preprocessor program (e.g. IntelCC).

Some further details about the NUT CI farm workers are available in Prerequisites for building NUT on different OSes (or docs/config-prereqs.txt in NUT sources for up-to-date information) and Custom NUT CI farm build agents: LXC multi-arch containers (or docs/ci-farm-lxc-setup.txt in NUT sources for up-to-date information) documentation.