the brief
A US vendor of wireless mesh devices was shipping OpenWrt-based firmware that it couldn’t fully account for. Its images carried 20 to 30 private patches spread across batman-adv, the Linux Wi-Fi stack, Wi-Fi drivers and OpenWrt core, and there was no reliable way to tell which of them had gone into any given build.
We gave the vendor a standard, automated build procedure in which every image can be traced back to the exact code it was built from, and then worked to shrink the private patch set by moving changes upstream.
the workstation problem
Firmware images were built on the senior developer’s workstation, from whatever state the code base happened to be in at that moment. There was no standard procedure, no version stamp in the image, and no record of which patches had been applied, so development carried on while images went out to customers with nothing to tie them back to the code that produced them.
As long as there was only one image in the field this was manageable. Once different customers were running different images, it stopped being manageable at all.
the same bug, again
When customers reported bugs, the first question, which code is this device actually running, had no answer. A critical misbehaviour couldn’t be traced to the change that introduced it, which also meant there was no way to tell what the right fix was, or whether a later image already contained it.
Every report became a fresh round of debugging, discovery and fixing, often for a bug that had already been found and fixed before, just in an image nobody could identify any more.
one source of truth
We started by cataloguing every private modification and moving all of them into version control as an ordered, documented patch set, so that the state of the code base was defined in one place rather than on one machine.
On top of that we automated the build, so that images are produced by a standard procedure rather than by hand, and every image is stamped with the exact version/commit of the code and patches it was built from. A bug report from a customer now points to a precise state of the code base, and the same source always produces the same firmware.
closing the gap with upstream
A private patch set is a cost the vendor pays on every update: each new OpenWrt or batman-adv release means reapplying, rebasing and retesting every change, and each backported fix risks conflicting with them.
So we measured how far the vendor’s tree had drifted from upstream and started moving patches out of it. Changes of general value went upstream to batman-adv, the Wi-Fi stack, the drivers and OpenWrt, and changes that no longer served a purpose were dropped. Around 50% of the patches were sent upstream: some were merged, some were reworked after valuable feedback received by the maintainers.
results
Every image traceable. Each firmware build maps to an exact, versioned state of the code base, so a customer report leads straight to the code behind it.
Bugs fixed once. Problems can be traced to the change that caused them, fixed at the source, and the fix verifiably reaches every image built afterwards.
A lighter patch set. With fewer private changes to carry, OpenWrt and batman-adv updates stopped being a project of their own, and critical upstream fixes could be taken in far more quickly.
what it took
Reproducible builds sound like a tooling problem, but most of this work was about the code itself: knowing what each of those patches did across batman-adv, the Wi-Fi stack, the drivers and OpenWrt core, deciding which belonged upstream, and getting them accepted there. We work in all of those projects routinely, which is what allowed the vendor’s firmware to move closer to upstream rather than just being built more neatly.
if this sounds familiar
If you’re not sure which code is running on your devices in the field, or every upstream update costs you weeks of rebasing, that’s the problem we solve.
Just book a call and let’s discuss.
More about our wireless and mesh work · Our open-source contributions