Updating a Fleet You Can't Reach
#kairair#ota#security#firmware#building-in-public#cryptography
David Olsson
The first station I gave away broke a comfortable assumption.
Up to that point, every board was on my desk. If the firmware needed changing, I plugged in a USB cable and reflashed it. Simple, total control, five seconds away at all times.
Then a station went to live in someone else's backyard, and the cable didn't reach anymore.
That's the moment a "collection of gadgets" has to actually become a fleet. A fleet is defined by a hard fact: you cannot physically touch most of it. If a station needs new firmware — a bug fix, a new sensor's support, a security patch — it has to receive that update over the air, apply it to itself, and reboot into it, with nobody standing there to catch it if it goes wrong.
Over-the-air updates. Easy to say. The trouble is what you've just built if you're careless.
the door you just installed
Think about what an OTA update actually is. A device on the internet is willing to download a chunk of code and run it as its own firmware. That's not a feature; that's the most dangerous thing a device can possibly do.
If a station will install any firmware that shows up looking official, then anyone who can impersonate the server — or just trick the device into fetching from the wrong place — owns that station. Not "reads its data." Owns it. Runs whatever they want on a computer sitting in someone's home, wired to their network.
An OTA system without verification isn't a convenience. It's a remote-code-execution backdoor you shipped on purpose, to every device, in every backyard.
So the real problem was never "how does a station download an update." It was: how does a station know an update is really from us, and really intact, before it trusts it with its own life.
a signature the device can check alone
The answer is asymmetric cryptography, used the boring, correct way.
There's one key pair for the whole network. A private key that only the server has, and a matching public key that gets baked into the firmware itself — it ships inside every station, in a file the device carries from the day it's flashed.
When the server prepares a firmware update, it signs the image with the private key. Signing is a mathematical operation that produces a chunk of bytes — a signature — that could only have been generated by someone holding the private key, and that's bound to the exact contents of that image. Change one byte of the firmware and the signature no longer matches.
The station, receiving the update, does the reverse. It has the public key. It checks the signature against the image it just downloaded. If the signature verifies, the device knows two things at once: the image came from the holder of the private key (us), and it hasn't been altered in transit. If it doesn't verify — wrong signer, corrupted download, a man in the middle — the device refuses the image and stays on the firmware it already trusts.
On top of that there's a plain integrity check — a hash of the contents and the expected size — so a merely corrupted download gets caught even before the cryptography weighs in. Belt and suspenders. The device would rather stay on old, working firmware forever than run something it can't prove.
The nice property is that the station verifies alone. It doesn't call home to ask permission. It doesn't trust the network path. It holds the public key, it does the math, and it decides. A station in a backyard with a flaky connection is still perfectly safe, because safety lives in the device, not the pipe.
how an update actually reaches a station
With that foundation, the delivery is almost mundane, which is the point.
An operator uploads a new firmware build. The server signs it and files it under a release channel. From there, stations get it one of two ways: they can quietly poll — check in and ask "is there anything newer for me?" — or an operator can push it, to a whole channel at once or to a single named device, from the admin panel. Either way, the same verification runs on the device before anything is installed. Push and poll are just when the image arrives. The signature check is whether it's allowed to matter.
That's a fleet you can manage without a van full of USB cables. Fix a bug once, sign it, push it, and every station proves the fix is authentic before running it.
the day the key got out
Now the honest part, because a build-in-public series that only tells the triumphs is lying by omission.
That whole scheme has a single point of absolute trust: the private key. It's the thing that says "this is really KAIRair." And the uncomfortable corollary is that if the private key ever leaks, everything downstream of it is compromised — because now someone else can sign an image your stations will happily accept.
We had a scare. During development, the signing key was exposed — pasted somewhere it never should have been. Nothing in the field was harmed, but the key had to be treated as burned. You don't get to hope a leaked key is fine. You assume the worst and you rotate.
Rotating a signing key is not a small thing, and this is where the earlier design decisions came due. The public key is baked into the firmware. So generating a new key pair means every station in the field is still carrying the old public key — and will keep trusting images signed by the old, now-retired private key until it's reflashed over USB with the new one. Rotate carelessly and you can lock yourself out of your own fleet, or leave a retired key trusted in the field.
So we did it deliberately. A fresh key pair was generated on the production server itself, so the new private key was born in the one place it's allowed to live and never travelled. The old key was retired. The new public key went into the firmware. And the standing consequence is now written down in plain language where I can't forget it:
If the private key is lost or regenerated, the whole fielded fleet rejects OTA until every device is re-flashed over USB with the new public key.
That sentence is a leash, on purpose. It's why the key gets backed up off the box. It's why "just regenerate it" is never a casual answer. The security that makes a station refuse a forged update is the same security that makes the key precious: the device's stubbornness protects you from attackers and from yourself.
what it buys
What you get, at the end of all that care, is the thing that makes a backyard network survivable: the ability to improve a station you will never physically see again, safely.
A sensor gets better support. A bug gets fixed. A new measurement becomes possible. You sign it once, and it rolls out to the fleet — and not one device runs a byte of it without first proving, on its own, that it came from you and arrived intact.
That's the difference between a gadget and infrastructure. A gadget you maintain by hand. Infrastructure maintains itself, and refuses anything it can't verify.
Part five of the KAIRair build series. Previous: Many Boards, Many Sensors, One Network. Next: sun, battery, USB — the four station types, why a weather station's hardest problem is power, and what "calibration mode" really means.