Some links will be in OS-specific chapters below; further reading for this effort included:

According to the fine print in the scary official docs, DigitalOcean VMs can only use "custom images" in one of a number of virtual HDD formats, which should carry an ext3/ext4 filesystem for DigitalOcean addons to barge into for management.

In practice, uploading other images (OpenIndiana Hipster "cloud" image, OmniOS, FreeBSD) from your workstation or by providing an URL to an image file on the Internet (see links in this document for some collections) sort of works. While the upload status remained "pending", a VM could often be made with it soon… but in other cases you have to wait a surprisingly long time, some 15-20 minutes, and additional images suddenly become "Uploaded".

FIXME: Private net, DO-given IPs

One limitation seen with "custom images" is that IPv6 is not offered to those VMs.

Generally all VMs get random (hopefully persistent) public IPv4 addresses from various subnets. It is possible to also request an interconnect VLAN for one project’s VMs co-located in same data center and have it attached (with virtual IP addresses) to an additional network interface on each of your VMs: it is supposed to be faster and free (regarding traffic quotas).

One more potential caveat: while DigitalOcean provides VPC network segments for free inter-communications of a group of droplets, it assigns IP addresses to those and does not let any others be used by the guest. This causes some hassle when importing a set of VMs which used different IP addresses on their inter-communications VLAN originally (on another hosting).

The original OpenIndiana Hipster and OmniOS VMs were configured with the https://github.com/jimklimov/illumos-splitroot-scripts methodology and scripting, so there are quite a few datasets dedicated to their purposes instead of a large one.

There are known issues about VM reboot:

Wondering if there are QEMU HW watchdogs on DigitalOcean that we could use…

Initial attempt, using the OpenIndiana cloud image ISO:

The OI image could be loaded… but that’s it — the logo is visible on the DigitalOcean Recovery Console, as well as some early boot-loader lines ending with a list of supported consoles. I assume it went into the ttya (serial) console as one is present in the hardware list, but DigitalOcean UI does not make it accessible and I did not find quickly if there are any REST API or SSH tunnel into serial ports.

It probably booted and auto-installed, since I could see an rpool/swap twice the size of VM RAM later on, and the rpool occupied the whole VM disk (created with auto-sizing).

The VM can however be rebooted with a (DO-provided) Recovery ISO, based at that time on Ubuntu 18.04 LTS with ZFS support — which was sufficient to send over the existing VM contents from original OI VM on Fosshost. See above about booting and preparing that environment.

As the practically useful VM already existed at Fosshost.org, and a quick shot failed at making a new one from scratch, in order to only transfer local zones (containers), a decision was made to transfer the whole ZFS pool via snapshots using the Recovery ISO.

First, following up from the first experiment above: I can import the ZFS pool created by cloud-OI image into the Linux Recovery CD session:

The import and subsequent inspection above showed that the kernel core-dump area was missing, compared to the original VM… so adding per best practice:

To receive ZFS streams from the running OI into the freshly prepared cloud-OI image, it wanted the ZFS features to be enabled (all were disabled by default) since some are used in the replication stream:

On the original VM, stop any automatic snapshot services like ZnapZend or zfs-auto-snapshot, and manually snapshot all datasets recursively so that whole data trees can be easily sent over (note that we then remove some snaps like for swap/dump areas which otherwise waste a lot of space over time with blocks of obsolete swap data held by the pool for possible dataset rollback):

#origin# zfs snapshot -r rpool@20231210-01
#origin# zfs destroy rpool/swap@20231210-01&
#origin# zfs destroy rpool/dump@20231210-01&

On the receiving VM, move existing cloudy rpool/ROOT out of the way, if we would not use it anyway, so the new one from the original VM can land (for kicks, we can zfs rename the cloud-image’s boot environment back into the fold after replication is complete). Also prepare to maximally compress the received root filesystem data, so it does not occupy too much in the new home (this is not something we write too often, so slower gzip-9 writes can be tolerated):

#recovery# zfs rename rpool/ROOT{,x} ; \
    while ! zfs set compression=gzip-9 rpool/ROOT ; do sleep 0.2 || break ; done

Send over the data (from the prepared screen session on the origin server); first make sure all options are correct while using a dry-run mode, e.g.:

### Do not let other work of the origin server preempt the replication
#origin# renice -n -20 $$

#origin# zfs send -Lce -R rpool/ROOT@20231210-01 | mbuffer | \
    ssh root@recovery "mbuffer | zfs recv -vFnd rpool"

With sufficiently large machines and slow source hosting, expect some hours for the transfer.

Note that one of the benefits of ZFS (and the non-automatic snapshots used here) is that it is easy to catch-up later to send the data which the original server would generate and write during the replication. You can keep it actually working until the last minutes of the migration.

After the large initial transfers complete, follow-up with a pass to stop the original services (e.g. whole zones either from OS default grouping or as wrapped by https://github.com/jimklimov/illumos-smf-zones scripting) and replicate any new information created on origin server during this transfer (and/or human outage for the time it would take you to focus on this task again, after the computers were busy for many hours…)

If the new VM does boot correctly, log into it and:

As of this writing, the NUT CI Jenkins controller runs on DigitalOcean — and feels a lot snappier in browsing and SSH management than the older Fosshost.org VMs. Despite the official demise of the platform, they were alive and used as build agents for the newly re-hosted Jenkins controller for over a year until somebody or something put them to rest: the container with the old production Jenkins controller was set to not-auto-booting, and container with worker was attached to the new controller.

The Jenkins SSH Build Agent setups involved here were copied on the controller (as XML files) and then updated to tap into the different "host" and "port" (so that the original definitions can in time be used for replicas on DO), and due to trust settings — the ~jenkins/.ssh/known_hosts file on the new controller had to be updated with the "new" remote system fingerprints. Otherwise, the migration went smooth.

Similarly, existing Jenkins swarm agents from community PCs had to be taught the new DNS name (some had it in /etc/hosts), but otherwise connected OK.

Added replicas of more existing VMs: OmniOS (relatively straightforward with the OI image).

The original OmniOS VM used ZFS, so its contents were sent-received similarly to the OI VM explained above.

Added replicas of more existing VMs: FreeBSD 12 (needed to use a seed image, tried an OpenIndiana image first but did not cut it — the ZFS options in its rpool were too new, so the older build of the BSD loader was not too eager to find the pool).

The original FreeBSD VM used ZFS, so its contents were sent-received similarly to the OI VM explained above.

Added a replica of OpenBSD 6.5 VM as an example of relatively dated system in the CI farm, which went decently well as a dd stream of the local VM’s vHDD into DO recovery console session:

#tgt-recovery# mbuffer -4 -I 12340 > /dev/vda

#src# dd if=/dev/rsd0c | time nc myHostingIP 12340

…followed by a reboot and subsequent adaptation of /etc/myname and /etc/hostname.vio* files.

I did not check if the DigitalOcean recovery image can directly mount BSD UFS partitions, as it sufficed to log into the pre-configured system.

One caveat was that it was originally installed with X11, but DigitalOcean web-console did not pass through the mouse nor advanced keyboard shortcuts. So rcctl disable xenodm (to reduce the attack surface and resource waste).

FWIW, openbsd-7.3-2023-04-22.qcow2 "custom image" did not seem to boot. At least, no activity on display and the IP address did not go up.

Spinning up the Debian-based Linux builder (with many containers for various Linux systems) with ZFS, to be consistent across the board, was an adventure.

After the root pool was prepared and the large tree of datasets defined to handle the numerous LXC containers, abuild home directory, and other important locations of the original system, rsync -avPHK worked well to transfer the data.

Numerous containers with an array of Linux distributions are used as either Jenkins SSH build agents or swarm agents, as documented in chapters about LXC containers.

Arch Linux containers prepared by procedure above include only a minimal footprint, and if you missed the -P pkg,list argument, they can lack even an SSH server. Suggestions below assume this path to container:

:; ALTROOT=/srv/libvirt/rootfs/jenkins-archlinux-amd64/rootfs/

Let pacman know current package database:

:; grep 8.8.8.8 $ALTROOT/etc/resolv.conf || (echo 'nameserver 8.8.8.8' > $ALTROOT/etc/resolv.conf)
:; chroot $ALTROOT pacman -Syu
:; chroot $ALTROOT pacman -S openssh sudo
:; chroot $ALTROOT systemctl enable sshd
:; chroot $ALTROOT systemctl start sshd

This may require that you perform bind-mounts above, as well as "passthrough" the /var/cache/pacman/pkg from host to guest environment (in virsh edit, and bind-mount for chroot like for /proc et al above).

It is possible that virsh console would serve you better than chroot. Note you may have to first chroot to set the root password anyhow.

Emulated-CPU container builds are CPU-intensive, so for them we define as few capabilities as possible: here CI is more interested in checking how binaries behave on those CPUs, not in checking the quality of recipes (distcheck, Make implementations, etc.), shell scripts or documentation, which is more efficient to test on native platforms.

Still, we are interested in results from different compiler suites, so specify at least one version of each.

The reduced set of labels for QEMU workers looks like:

qemu-nut-builder qemu-nut-builder:alldrv
NUT_BUILD_CAPS=drivers:all NUT_BUILD_CAPS=cppunit
OS_FAMILY=linux OS_DISTRO=debian11 GCCVER=10 CLANGVER=11
COMPILER=GCC COMPILER=CLANG
ARCH64=ppc64le ARCH_BITS=64

For contrast, a "real" build agent’s set of labels, depending on presence or known lack of some capabilities, looks something like this:

doc-builder nut-builder nut-builder:alldrv
NUT_BUILD_CAPS=docs:man NUT_BUILD_CAPS=docs:all
NUT_BUILD_CAPS=drivers:all NUT_BUILD_CAPS=cppunit=no
OS_FAMILY=bsd OS_DISTRO=freebsd12 GCCVER=10 CLANGVER=10
COMPILER=GCC COMPILER=CLANG
ARCH64=amd64 ARCH_BITS=64
SHELL_PROGS=sh SHELL_PROGS=dash SHELL_PROGS=zsh SHELL_PROGS=bash
SHELL_PROGS=csh SHELL_PROGS=tcsh SHELL_PROGS=busybox
MAKE=make MAKE=gmake
PYTHON=python2.7 PYTHON=python3.8

This is a typical use-case for tightly integrated build farms under common management, where the Jenkins controller can log by SSH into systems which act as its build agents. It injects and launches the agent.jar to execute child processes for the builds, and maintains a tunnel to communicate.

Methods below involving SSH assume that you have configured a password-less key authentication from the host machine to the abuild account in each guest build environment container. This can be an ssh-keygen result posted into authorized_keys, or a trusted key passed by a chain of ssh agents from a Jenkins Credential for connection to the container-hoster into the container. The private SSH key involved may be secured by a pass-phrase, as long as your Jenkins Credential storage knows it too. Note that for the approaches explored below, the containers are not directly exposed for log-in from any external network.

'

This approach allows remote systems to participate in the NUT CI farm by dialing in and so defining an agent. A single contributing system may be running a number of containers or virtual machines set up following the instructions above, and each of those would be a separate build agent.

Such systems should be "dedicated" to contribution in the sense that they should be up and connected for days, and sometimes tasks would land.

Configuration files maintained on the Swarm Agent system dictate which labels or how many executors it would expose, etc. Credentials to access the NUT CI farm Jenkins controller to register as an agent should be arranged with the farm maintainers, and currently involve a GitHub account with Jenkins role assignment for such access, and a token for authentication.

The jenkins-swarm-nutci repository contains example code from such setup with a back-up server experiment for the NUT CI farm, including auto-start method scripts for Linux systemd and upstart, illumos SMF, and OpenBSD rcctl.

Another aspect of farm management is that emulation is a slow and intensive operation, so we can not run all agents and execute builds at the same time.

The current solution relies on https://github.com/jimklimov/conflict-aware-ondemand-retention-strategy-plugin to allow co-located build agents to "conflict" with each other — when one picks up a job from the queue, it blocks neighbors from starting; when it is done, another may start.

Containers can be configured with "Availability ⇒ On demand", with shorter cycle to switch over faster (the core code sleeps a minute between attempts):

Also, the "executors" count should be reduced to the amount of compilers in that system (usually 2) and so avoid extra stress of scheduling too many emulated-CPU builds at once.

As part of the jenkins-dynamatrix optional optimizations, the NUT CI recipe invoked via Jenkinsfile-dynamatrix maintains persistent git reference repositories that can be used to cache NUT codebase (including the tested commits) and so considerably speed up workspace preparation when running numerous build scenarios on the same agent.

Such .gitcache-dynamatrix cache directories are located in the build workspace location (unique for each agent), but on a system with numerous containers these names can be symlinks pointing to a shared location.

To avoid collisions with several executors updating the same cache with new commits, critical access windows are sequentialized with the use of Lockable Resources plugin. On the jenkins-dynamatrix side this is facilitated by labels:

DYNAMATRIX_UNSTASH_PREFERENCE=scm-ws:nut-ci-src
DYNAMATRIX_REFREPO_WORKSPACE_LOCKNAME=gitcache-dynamatrix:SHARED_HYPERVISOR_NAME