Example list of packages for Debian-based systems may include (not necessarily is limited to):

:; apt install lxc lxcfs lxc-templates \
    ipxe-qemu qemu-kvm qemu-system-common qemu-system-data \
    qemu-system-sparc qemu-system-x86 qemu-user-static qemu-utils \
    virt-manager virt-viewer virtinst ovmf \
    libvirt-daemon-system-systemd libvirt-daemon-system \
    libvirt-daemon-driver-lxc libvirt-daemon-driver-qemu \
    libvirt-daemon-config-network libvirt-daemon-config-nwfilter \
    libvirt-daemon libvirt-clients

# TODO: Where to find virt-top - present in some but not all releases?
# Can fetch sources from https://packages.debian.org/sid/virt-top and follow
# https://www.linuxfordevices.com/tutorials/debian/build-packages-from-source
# Be sure to use 1.0.x versions, since 1.1.x uses a "better-optimized API"
# which is not implemented by libvirt/LXC backend.

Prepare LXC and LIBVIRT-LXC integration, including an "independent" (aka "masqueraded) bridge for NAT, following https://wiki.debian.org/LXC and https://wiki.debian.org/LXC/SimpleBridge

Prepare /home/abuild on the host system (preferably in ZFS with lightweight compression like lz4 — and optionally, only if the amount of available system RAM permits, with deduplication; otherwise avoid it); account user and group ID numbers are 399 as on the rest of the CI farm (historically, inherited from OBS workers)

Edit the root (or whoever manages libvirt) ~/.profile to default the virsh provider with:

LIBVIRT_DEFAULT_URI=lxc:///system
export LIBVIRT_DEFAULT_URI

If host root filesystem is small, relocate the LXC download cache to the (larger) /srv/libvirt partition:

:; mkdir -p /srv/libvirt/cache-lxc
:; rm -rf /var/cache/lxc
:; ln -sfr /srv/libvirt/cache-lxc /var/cache/lxc

Monitor deployed container rootfs’es with:

:; du -ks /srv/libvirt/rootfs/*

(should have non-trivial size for deployments without fatal infant errors)

Mass-edit/review libvirt configurations with:

:; virsh list --all | awk '{print $2}' \
   | grep jenkins | while read X ; do \
     virsh edit --skip-validate $X ; done

Mass-define network interfaces:

:; virsh list --all | awk '{print $2}' \
   | grep jenkins | while read X ; do \
     virsh dumpxml "$X" | grep "bridge='lxcbr0'" \
     || virsh attach-interface --domain "$X" --config \
        --type bridge --source lxcbr0 ; \
   done

Verify that unique MAC addresses were defined (e.g. 00:16:3e:00:00:01 tends to pop up often, while 52:54:00:xx:xx:xx are assigned to other containers); edit the domain definitions to randomize, if needed:

:; grep 'mac add' /etc/libvirt/lxc/*.xml | awk '{print $NF" "$1}' | sort

Make sure at least one console device exists (end of file, under the network interface definition tags), e.g.:

    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>

Populate with abuild account, as well as with the bash shell and sudo ability, reporting of assigned IP addresses on the console, and SSH server access complete with envvar passing from CI clients by virtue of ssh -o SendEnv='*' container-name:

:; for ALTROOT in /srv/libvirt/rootfs/*/rootfs/ ; do \
    echo "=== $ALTROOT :" >&2; \
    grep eth0 "$ALTROOT/etc/issue" || ( printf '%s %s\n' \
        '\S{NAME} \S{VERSION_ID} \n \l@\b ;' \
        'Current IP(s): \4{eth0} \4{eth1} \4{eth2} \4{eth3}' \
        >> "$ALTROOT/etc/issue" ) ; \
    grep eth0 "$ALTROOT/etc/issue.net" || ( printf '%s %s\n' \
        '\S{NAME} \S{VERSION_ID} \n \l@\b ;' \
        'Current IP(s): \4{eth0} \4{eth1} \4{eth2} \4{eth3}' \
        >> "$ALTROOT/etc/issue.net" ) ; \
    groupadd -R "$ALTROOT" -g 399 abuild ; \
    useradd -R "$ALTROOT" -u 399 -g abuild -M -N -s /bin/bash abuild \
    || useradd -R "$ALTROOT" -u 399 -g 399 -M -N -s /bin/bash abuild \
    || { if ! grep -w abuild "$ALTROOT/etc/passwd" ; then \
            echo 'abuild:x:399:399::/home/abuild:/bin/bash' \
            >> "$ALTROOT/etc/passwd" ; \
            echo "USERADDed manually: passwd" >&2 ; \
         fi ; \
         if ! grep -w abuild "$ALTROOT/etc/shadow" ; then \
            echo 'abuild:!:18889:0:99999:7:::' >> "$ALTROOT/etc/shadow" ; \
            echo "USERADDed manually: shadow" >&2 ; \
         fi ; \
       } ; \
    if [ -s "$ALTROOT/etc/ssh/sshd_config" ]; then \
        grep 'AcceptEnv \*' "$ALTROOT/etc/ssh/sshd_config" || ( \
            ( echo "" ; \
              echo "# For CI: Allow passing any envvars:"; \
              echo 'AcceptEnv *' ) \
            >> "$ALTROOT/etc/ssh/sshd_config" \
        ) ; \
    fi ; \
   done

Note that for some reason, in some of those other-arch distros useradd fails to find the group anyway; then we have to "manually" add them.

Let the host know and resolve the names/IPs of containers you assigned:

:; grep -v '#' /etc/lxc/dnsmasq-hosts.conf \
   | while IFS=, read N I ; do \
    getent hosts "$N" >&2 || echo "$I $N" ; \
   done >> /etc/hosts

Q: Container won’t start, its virsh console says something like:

Failed to create symlink /sys/fs/cgroup/net_cls: Operation not permitted

A: According to https://bugzilla.redhat.com/show_bug.cgi?id=1770763 (skip to the end for summary) this can happen when a newer Linux host system with cgroupsv2 capabilities runs an older guest distro which only knows about cgroupsv1, such as when hosting a CentOS 7 container on a Debian 11 server.

Q: Server was set up with ZFS as recommended, and lots of I/O hit the disk even when application writes are negligible

A: This was seen on some servers and generally derives from data layout and how ZFS maintains the tree structure of blocks. A small application write (such as a new log line) means a new empty data block allocation, an old block release, and bubble up through the whole metadata tree to complete the transaction (grouped as TXG to flush to disk).