Building controlled Linux environments for specialised products

§ 01The cost of a package nobody chose

A general-purpose distribution installed with its defaults ships several hundred packages that nobody on the product team chose. A stock Ubuntu Server cloud image carries on the order of 600 packages before the product's own dependencies go in; a default desktop install, well over 1,800.¹ Each one arrives with its own maintainer, its own update cadence and its own CVE history, and a good share of them install something that starts at boot: a timer, a socket, a daemon waiting for hardware the product does not have.

The habit is to file the reduction under housekeeping. I treat it as a product decision, because every one of those packages is paid for in four currencies at once. Robustness: a unit that is present can fail, and a unit that fails on one machine in fifty is a field call. Reproducibility: the more that is installed, the more of the boot depends on ordering, timers and network state you never specified. Attack surface: every listening socket is one you now have to patch, and every setuid binary is a privilege boundary you are now responsible for. Support: an image you cannot enumerate is an image where “what changed” is an investigation instead of a diff.

The bar I set is not a package count. It is that I can produce the list of everything that runs on the device, say why each entry is there, and rebuild the same list from a text file a year later. The practice page shows the shape with a synthetic grid, 1,842 packages down to 312; that ratio is the right order of magnitude for a product that needs a kernel, an init, a network stack, a runtime and its own service, but the number is a consequence of the method, not a target of it.

§ 02A base image that is a definition, not a machine

The build must be a function of files in a repository and nothing else: not a golden VM someone configured once, not whatever the mirror happened to serve the day the build ran. Three things make that true on a Debian-family base. The package set is a lock file with exact versions. The repository is a dated snapshot, so apt resolves against the same index every time. And the timestamps inside the image are fixed with SOURCE_DATE_EPOCH, so two builds of the same definition produce the same bytes, or a diff that explains why not.

#!/usr/bin/env bash
# build.sh: the image is a function of this file and lock/packages.lock
set -euo pipefail
SNAP=20260701T000000Z                 # dated apt snapshot: same index every build
export SOURCE_DATE_EPOCH=1751328000     # fixed timestamps inside the image
PKGS=$(paste -sd, lock/packages.lock)   # one "name=version" per line, essentials included

mmdebstrap --variant=custom --architectures=amd64 \
  --include="$PKGS" \
  --dpkgopt='path-exclude=/usr/share/doc/*' \
  --dpkgopt='path-exclude=/usr/share/man/*' \
  --dpkgopt='path-exclude=/usr/share/locale/*' \
  --hook-dir=hooks/ \                    # masks units, drops machine-id + host keys, seeds /etc
  noble rootfs/ \
  "https://snapshot.ubuntu.com/ubuntu/$SNAP"

# the manifest is what went in; the squashfs is what ships; both are hashed
dpkg-query --admindir=rootfs/var/lib/dpkg -W -f='${Package}=${Version}\n' \
  | sort > out/manifest.txt
mksquashfs rootfs/ out/root.squashfs -comp zstd -noappend -all-root \
  -mkfs-time "$SOURCE_DATE_EPOCH" -all-time "$SOURCE_DATE_EPOCH"
syft dir:rootfs/ -o cyclonedx-json > out/sbom.cdx.json
sha256sum out/manifest.txt out/root.squashfs out/sbom.cdx.json > out/SHA256SUMS

The recipe builds from --variant=custom, which means nothing goes in unless the lock file names it, essential packages included. That is deliberate: minbase is a distribution's opinion about what is minimal, and the product's opinion is different. The hooks directory does the surgery that turns a distribution into a product: it masks the units on the deny-list, removes /etc/machine-id and every SSH host key so they are generated on first boot rather than cloned across the fleet, and writes the handful of /etc files the product actually owns. The manifest, the SBOM and the hashes go into the release next to the image; §06 explains what they buy.

Pinning is half of it. The lock file has to move on purpose. A scheduled job resolves the current snapshot against the same package names, diffs the resulting manifest against the committed one and opens a change with the diff and the matching CVE list attached. Nothing reaches the image without that diff being read by a person. It is slower than apt upgrade on a golden machine, and it is the difference between a product you can support for years and an artefact you can only rebuild by hand.

§ 03The allow-list of services

A package list says what is installed; it does not say what runs. On systemd the running set is decided by what is enabled, what is static but pulled in through .wants directories, what sockets and timers activate lazily, and what the generators synthesise from /etc/fstab and the kernel command line. Auditing it means asking the image, then asking a booted device, and reconciling the two.

#!/usr/bin/env bash
# audit-units.sh: fail the build if anything is enabled that is not on the allow-list
set -euo pipefail
ROOT=${1:-/}
ALLOW=allow/units.txt      # unit<TAB>reason, one per line

# 1. static: what the image enables, no boot needed (static units count: sockets, timers, .wants)
systemctl --root="$ROOT" list-unit-files --state=enabled,static --no-legend \
  | awk '{print $1}' | sort > /tmp/enabled.txt
cut -f1 "$ALLOW" | sort > /tmp/allowed.txt

if comm -23 /tmp/enabled.txt /tmp/allowed.txt | grep -q .; then
  echo "units enabled without a reason:" >&2
  comm -23 /tmp/enabled.txt /tmp/allowed.txt >&2
  exit 1
fi

# 2. dynamic, on a booted device: every LISTEN socket must map to an allowed unit and address
ss -Hltunp | awk '{print $1, $5, $7}'            # proto  local-addr:port  users:(("proc",pid,fd))
systemctl list-units --type=service,socket,timer --state=running,active,waiting --no-legend

# 3. exposure: anything above 7.0 that runs as root and touches the network gets a sandbox
systemd-analyze security --no-pager | awk 'NR>1 && $2+0 > 7.0'

The static half runs in CI against the unpacked root, so a lock update that drags in a new package with a new timer fails the build before anyone boots it. The dynamic half runs on a lab device for every release candidate: ss lists every socket in LISTEN, and each line has to map to a unit on the allow-list with the address it is expected to bind. Anything on 0.0.0.0 that is not the product's own API is a defect. systemd-analyze security gives each service an exposure score; I do not chase every unit to the floor, but a score above 7 for something that runs as root and talks to the network means ProtectSystem=strict, PrivateTmp=, CapabilityBoundingSet= and a SystemCallFilter= before it ships.

What usually goes: the distribution's unattended upgrades and their timers, because updating is the product's job (§05); snapd, cloud-init, ModemManager, avahi, cups, the MOTD generators, apport, the fwupd refresh timer, polkit when nothing needs it. What stays, and is written down with its reason: systemd-journald with a size cap, systemd-timesyncd or chrony pointed at a source the product controls, systemd-networkd or NetworkManager (one, never both), sshd bound to the management interface, the hardware watchdog, and the product's own units.

§ 04Read-only root, explicit state

Once the set is known, the next step is to stop it changing in the field. The root filesystem is a squashfs image mounted read-only. Anything writable is writable on purpose, and it lives in one of three places: a tmpfs for what may be lost at reboot, a small state partition for what the machine owns (machine-id, host keys, the network settings a technician entered, the health record of the current slot), and a data partition for what the product owns. /etc gets an overlay whose upper directory sits on state, so the persistent delta against the image stays small and readable; overlayroot mounts it on Ubuntu, a short initramfs hook does the same elsewhere.

The discipline this imposes is useful in itself. Every path a service writes to becomes a decision, and StateDirectory=, LogsDirectory= and systemd-tmpfiles make it explicit in the unit rather than implicit in a script. Packages whose post-install steps assume they can regenerate something under /etc or /usr at first boot show up immediately, because the write fails; they either get a build-time hook or they do not ship. What you get is a device whose state after a power cut is the image plus a small, enumerable delta, instead of whatever a year of apt and hand edits produced.

§ 05A/B updates and rollback

An update is a whole new image, not a package transaction. Two root partitions of identical size, one active and one inactive; the updater verifies the signature and the hash of the new image, writes it to the inactive slot, adds a boot entry for that slot with a try counter, and reboots. If the new slot fails to reach the product's own health check within three attempts, the boot loader falls back to the previous slot without anyone's help.² The state and data partitions are shared, so anything on data must stay readable by release N−1 for as long as N can still be rolled back to it. That is a rule the product code inherits from the update design, and it belongs where the developers will read it.

# disk.sfdisk: GPT, both roots the same size so any image fits either slot
label: gpt
1 : size=256M, type=uefi,  name=esp
2 : size=2G,   type=linux, name=root-a
3 : size=2G,   type=linux, name=root-b
4 : size=64M,  type=linux, name=state     # /etc upper dir, machine-id, host keys, slot health
5 :            type=linux, name=data      # product data, shared by both slots, survives every update

# /boot/loader/entries/product-b+3.conf: written by the updater; "+3" = three tries left
title    product (slot B)
linux    /EFI/product/b/vmlinuz
initrd   /EFI/product/b/initrd.img
options  root=PARTLABEL=root-b ro rootfstype=squashfs roothash=<verity root hash> \
         overlayroot=device:dev=/dev/disk/by-partlabel/state panic=10

# /etc/systemd/system/boot-good.service: a boot is good only when the product answers
[Unit]
Description=Confirm this boot once the product answers
Requires=product.service
After=product.service
Before=boot-complete.target
FailureAction=reboot            # a bad slot must consume its tries, not hang
[Service]
Type=oneshot
ExecStart=/usr/bin/curl -fsS --max-time 5 http://127.0.0.1:8080/health
[Install]
WantedBy=boot-complete.target   # systemd-bless-boot runs after this target; if we fail, it never does

The table compares the payload and the failure behaviour of the strategies I have used or evaluated. The assumptions: a compressed root image of about 400 MB, releases that change roughly 10 % of the blocks between versions, and a fleet on metered links where each megabyte per device is paid for. Payload is a fraction of the full image; it moves with the change ratio between releases, not with the product.

In-place package updates are what a general-purpose distribution gives you, and the last row is why I do not ship them on a device: rollback needs filesystem snapshots you would have to design in anyway, and a power cut between unpack and configure leaves dpkg in a state a remote technician cannot reason about. A/B with a block delta keeps the atomicity and cuts the payload by whatever fraction of the image actually changed; OSTree reaches the same numbers with content-addressed objects and fits better when the image is many small independent components. On a specialised product with one primary service I default to plain A/B and add the delta when fleet size or link cost justifies the extra moving part.

§ 06Proving the image is what you think it is

Three artefacts answer three questions. The package manifest, sorted and hashed, answers what is installed; the diff between two releases' manifests is the change log for the whole operating system, and it is what feeds the SOUP record when the product is a medical device (the note on legal constraints as inputs covers that record). The squashfs hash answers what bytes were shipped, and it is what the updater verifies before it writes a slot. Neither answers what is being read now, on a device in the field, a year on: for that the squashfs is formatted with dm-verity and the root hash travels on the kernel command line, which is itself part of the signed boot entry, so a block that changed since build time fails to read instead of failing quietly.

Reproducibility is checked, not assumed. CI builds the release twice, on two runners, and compares the manifest hash and the squashfs hash. The manifest is bit-identical every time. The squashfs is identical when the lock file, the snapshot and SOURCE_DATE_EPOCH are pinned and mksquashfs is told to fix its timestamps; when it is not, diffoscope on the two trees points at the package that embeds a build date or a random seed, and that package is either fixed or listed as a known non-reproducible with a reason next to it. The SBOM comes out of the same build, from the same root, so it cannot describe an image other than the one it ships with. What I look at on every release is therefore short: two hashes that match, a manifest diff that has been read, an SBOM diff against the last release, and the unit audit that came back empty.

§ 07Failure modes

The list below is the class of things that go wrong on controlled images and that a developer's machine will never show you, with what the design above does about each. None of them is exotic; all of them are invisible until a device is alone in the field.

• Cloned identity. A machine-id, SSH host keys or a DHCP client identifier baked into the image gives every device the same identity: the DHCP server hands out the same lease twice and the log collector cannot tell them apart. Removed at build, generated on first boot, kept on state.
• Time at first boot. A board without an RTC battery boots in 1970, no TLS certificate is valid yet, and every HTTPS fetch fails until the clock is set. The updater is ordered After=time-sync.target, and the time source is one the product controls, not a pool that the customer's firewall may block.
• The overlay that fills up. A volatile /var/log or an /etc upper directory grows until memory or the state partition is full, weeks after deployment. RuntimeMaxUse= on journald, size limits on every tmpfs, and free space on state as part of the health probe.
• A health check that passes while the product fails. A boot-good that only checks the service is active blesses a crash-looping slot. The check calls the product's own endpoint, and only that, and it can fail: FailureAction=reboot, so a bad slot consumes its tries instead of hanging on them.
• Kernel and modules out of step. Two slots sharing one /boot and one kernel means slot B runs slot A's modules. Each boot entry points at its own kernel and initramfs, and firmware for a board revision that only exists in the field is a lock-file entry, not a file someone copied.
• Data that N−1 cannot read. The operating system rolls back cleanly and the product will not start, because the schema on data moved forward with N. Migrations are forward-compatible for one release, and CI tests the rollback, not only the upgrade.
• A hidden write to the read-only root. A post-install script, or the product itself, writes under /usr at runtime; it works on the developer's writable root and fails silently on the device. CI boots the read-only image and scans journalctl -p err before the candidate is called one.