Qualify a board in bootloader mode

A build that passes the build checks is not a board that rolls back. The checks prove the configuration. Only a bad update on the real board proves the rollback.

Qualify every board and every bootloader build before you ship it. This page is the procedure we followed on the QEMU targets, and what to repeat on yours.

Use real artifacts only

Test with real, signed artifacts made from your own root file system. Apply a fault to a disposable copy, check the fault there, and only then sign the artifact. Never deploy a hand-assembled or synthetic artifact to a real board. Delete every pending deployment for a board before you power-cycle it. A pending test deployment can install itself at the next start.

Before you start

You need:

• A serial console on the board. Without one you cannot see the bootloader fall back.
• Control of the power, such as a relay. Some failures need a power cycle.
• A way to recover the board: a card reader, a USB download mode or an SD swap.
• Your server with no pending deployment for the board.
• Someone at the bench for the first run.

Stage 1: classify the board

Answer these before you build:

1. Which bootloader boots the board: U-Boot or GRUB?
2. Where does it keep its environment? On U-Boot: which device, which offsets, and are there two copies? On GRUB: which partition?
3. Can U-Boot’s environment be written from Linux with fw_setenv?
4. Is the partition table MBR or GPT? Is there room for two root file systems and a data partition?
5. Does the kernel command line carry a positive panic=? Is there a hardware watchdog?
6. How do you recover a bricked unit?

If the board boots through neither U-Boot nor GRUB, bootloader mode does not apply.

Stage 2: build and read-only checks

Build with the variables for your board. Fix every message the build prints.

Boot the image and look, without changing anything. These are the checks we recorded at the start of every qualification run:

cat /etc/otapulse/boot_mode
cat /proc/cmdline
systemctl is-system-running
systemctl --failed

You should see:

• bootloader from the first command.
• A root= that names one of your two root partitions, and a positive panic=<n> on the command line.
• running from systemctl is-system-running, and no failed unit.

On U-Boot, read the environment:

fw_printenv

The output must show upgrade_available=0, a bootcount, a bootlimit above zero and an altbootcmd. A bootlimit of 0 means the bootloader never decides that an update failed. The qualified images use bootlimit=1. The active slot variable is also in the list. Its name is on the attribution page.

On GRUB, the environment is two copies on the EFI partition. It must show upgrade_available=0 and a bootcount. The agent reads it for you. If you need to read it by hand, the tool names are on the attribution page.

On the first boot of a fresh U-Boot image the console prints Loading Environment from MMC... *** Warning - bad CRC, using default environment, then Saving Environment to MMC... Writing to MMC(0)... OK. That is expected once. Every later boot prints Loading Environment from MMC... OK.

Stage 3: a first real update, both ways

Deploy a real signed artifact built from your image, and install it. The image build writes one (see Build an image), and the first update guide shows how to deploy it. Then deploy again to go back.

Check after each install:

• The running root partition changed (root= in /proc/cmdline).

• upgrade_available is back to 0 after the agent commits.

• The server shows the deployment as successful.

• The device reports the new artifact name. Read it on the device with:

  OTAPULSE_DATASTORE_DIR=/data/otapulse soc-ota-agent show-artifact

  Without OTAPULSE_DATASTORE_DIR the command reads an empty default datastore and prints unknown. The service sets this variable itself.

• systemctl --failed is empty on the first boot of the freshly written slot.

The three update state scripts that native images use to switch slots are switched off in bootloader mode. On a bootloader-mode device they log a line and exit before they touch anything. That line goes to /var/log/messages, not to the journal.

The second slot is not trustworthy until a real update has written it. Our factory images ship a filled-looking second slot, so a file system there proves nothing. Use the reported artifact name to know a slot holds a real update.

Stage 4: prove the rollback

Run each of these on the board with the console open. For each one, check that the bootloader returned to the original slot, upgrade_available is 0, and the server shows the deployment as failed. Then install a healthy update. It must commit.

Bad update	What it proves	What you should see
The new root file system cannot be read by the bootloader (zeroed superblock)	The bootloader falls back with no help from Linux. The kernel never runs.	U-Boot prints a load failure and resetting ..., then Warning: Bootlimit (...) exceeded. Using altbootcmd. GRUB prints an error (for example error: disk ... not found.) and flips back on the next pass. At least two new bootloader banners appear.
The root file system mounts but has no usable init	The kernel panic timeout resets the board.	Kernel panic - not syncing: No working init found. and Rebooting in <n> seconds, with <n> equal to your OTAPULSE_PANIC_TIMEOUT_S.
The agent is masked, so userspace is up but the update can never commit	The boot deadline reboots a stuck trial boot.	The device stays up until about 20 minutes of uptime, then reboots. The bootloader reverts.
The data seed fails (device type on /data differs from the image)	The agent cannot start, and the guard reboots at once.	On U-Boot the console shows Failed to start OTA-Pulse persistent data seeding. The device reboots well before 20 minutes.
A standalone soc-ota-agent install that waits for its own reboot	The guard must not reboot a device that is merely waiting.	After more than 20 minutes the device has the same boot id. No bootloader banner appears.

Use a value of OTAPULSE_PANIC_TIMEOUT_S that is not the default 10 for the “no usable init” test. Then the console line Rebooting in <n> seconds proves your value reached the kernel, and not a kernel default. The qualified images used 7.

Our test suite builds these artifacts from the device’s own root file system. Do not assemble them by hand.

Note

These cases take time on a real board, and longer on a slow one. On the emulated targets the whole pass took 1.5 to 2.5 hours. About 8 minutes of each arm64 revert case was the host building a 1 GB artifact.

How each bootloader rolls back

	U-Boot	GRUB
Environment	Two redundant copies on the storage device. The agent reads them with fw_printenv and writes them with fw_setenv.	Two copies on the EFI partition.
Flag and counter	upgrade_available, bootcount, bootlimit, altbootcmd	upgrade_available, bootcount (GRUB keeps no bootlimit)
What reverts	Boot count passes bootlimit, so U-Boot runs altbootcmd and boots the old slot.	On the next pass with upgrade_available=1 and bootcount non-zero, GRUB flips back to the old slot.
Needs a reset	Yes. A kernel panic resets through panic=<n>.	Yes. The same.
Qualified on	QEMU arm64, SD card, MBR	QEMU x86-64, UEFI, GPT

On QEMU every revert showed 3 new bootloader banners on the console. A healthy install showed at least one.

Boot deadline

The bootloader reverts only when the board resets. A hung userspace with a running kernel does not reset it. A small guard on every bootloader-mode image covers that. It never writes the bootloader environment and never decides what a rollback is. It only turns two stuck situations into a plain reboot, and the bootloader does the rest.

• Deadline. If the bootloader environment still says upgrade_available=1 20 minutes after the boot, on a boot that started as the trial boot of an update, the guard reboots the device. It logs the reason at error level first:

  journalctl -p err -t otapulse-bl-guard

  The journal is volatile. Copy it off the device before the reboot if you need it.

• Data seed failure. If otapulse-data-seed fails, the agent cannot start. On an uncommitted update the guard reboots at once, so the update is reverted and does not sit dark until the deadline. Fix a device-type mismatch by reflashing the data partition, or by removing /data/otapulse/device_type if the change is intended.

The guard is careful, because a wrong reboot would loop a healthy device:

• It never acts on an environment it cannot read. It logs at error level instead, and the timer unit shows as failed.
• It does nothing when bootcount is 0. That is an install that is waiting for its own reboot.
• It reboots at most once per boot.
• Consecutive guard reboots are counted on the data partition and stop at 5. The count is cleared the first time upgrade_available=0 is seen.

Limits of the boot deadline

• A /data mount failure is not covered. On U-Boot the environment description lives in /data/u-boot, so the environment is unreadable then. The system most likely drops to emergency mode, where timers do not run. Only the kernel panic path and a watchdog can rescue that boot.
• No rollback and no writable /data. Then there is no persistent reboot count, and the deadline reboots once per boot, about every 20 minutes, with no end. With a writable /data it stops after 5.
• Update control maps. A control map that pauses the agent before the commit for longer than 20 minutes is turned into a reboot, and then a rollback. No server feature creates such a pause today. Raise the timer’s OnBootSec or disable the timer before you introduce one.
• A trial boot with an unreadable environment gets no trial marker. The deadline then does not rescue it.

Limits of our qualification

These came out of the QEMU runs. Check them on your hardware.

• Emulation. The runs used QEMU without KVM. The arm64 image carried softdog.soft_margin=600 as a bench accommodation, because the emulated SD card stalled the guest during the agent’s large read-back. A hardware image must keep the default watchdog margin, and you should watch the agent’s read-back time on your own storage.
• x86-64 console. The serial console printed kernel messages but no systemd unit status. The console list ends with the screen, so systemd writes status there. Putting the serial console last on the kernel command line is the likely fix. We have not applied it.
• x86-64 GRUB timing. On the fast x86-64 row, the check that the environment pointed at the new slot before the reboot was skipped in 3 of the 4 revert cases. The proof for those cases is the console path, the bootloader banner count, and the state after the revert.
• Inactive slot identity. We recorded the running slot’s file hashes and reported artifact at start, after the first install and at the end of every run. We did not hash the inactive slot before and after each case.
• Manual provisioning was booted live on arm64 only.
• Late-run failed unit. After the 30 minute standalone-install case on arm64, the emulated storage starved the guest long enough for a diagnostic logging unit to miss its 2 second watchdog ping, and systemd marked it failed. The guard cases check that no unit has failed at their start.