the complaint
“The client is connected. It has an IP. Nothing goes through.”
A Wi-Fi router manufacturer in Hong Kong had been hearing this from customers for a long time. Some stations, some networks, no pattern. Broadcast traffic kept flowing while unicast to the same station died silently. The logs showed a healthy association and nothing else, because as far as the stack was concerned nothing had gone wrong.
Our best estimate is that the bug had shipped with the product from the beginning: one or two years of intermittent complaints that nobody had managed to close.
The cause turned out to be in ath9k, the mainline Linux driver for Atheros (now Qualcomm) chipsets. In the two months after our fix went out in firmware, this class of complaint dropped by about 90%.
client profile
industry
Wi-Fi router manufacturing
size
Small-sized, USA-based vendor
focus
Delivering high-quality Wi-Fi networking products to a global customer base
why nobody had caught it
Three things kept this bug alive for years.
It never announced itself. No error, no timeout, no counter moving. You cannot grep for silence.
It refused to be reproduced. It only happened on live customer networks, under real traffic with a real mix of clients, and it vanished the moment anyone tried to recreate it on a bench. That removes the entire normal debugging workflow in one go.
And it sat well below the layer most teams can work at. The symptom appeared at the application level; the cause was in driver and chipset behaviour. A team without driver expertise can stare at the effect for years without ever getting close to it, which is more or less what had happened.
going into production to find it
The bug only existed in the field, so that’s where we worked.
We picked the customer networks that reported the problem most often and instrumented a few dozen access points across them, streaming live packet captures over SSH to a central collection point. Real traffic, real users, production networks, no disruption.
Then came the slow part.
the theories that didn't survive
We went through tens of hypotheses over several weeks. The plausible ones all died under the data.
A rogue DHCP server, quietly changing a client’s address underneath it, would explain traffic stopping while the association stayed up. The captures said otherwise.
Spurious deauthentication frames would explain a station dropping out. But the affected stations weren’t being deauthenticated. They stayed associated the whole time.
Interference or channel jamming would explain degradation. It does not explain broadcast reaching a station perfectly while unicast to that same station stops dead.
Each dead end narrowed the field a little. That’s the real shape of this kind of work: weeks of removing explanations, with no guarantee the next one is the answer.
the one that held
Eventually the captures pointed somewhere specific. The chipset was deleting Pairwise Transient Keys for certain clients during connection, when it had no business doing so.
Once you see it, everything fits. Lose the unicast key on one side and unicast frames can’t be decrypted or delivered, while broadcast, encrypted with the group key, carries on untouched. The association stays up. DHCP already finished. Nothing logs an error, because from the stack’s point of view nothing failed.
That is precisely the complaint the customers were filing.
days, not weeks, once we knew
Finding the bug took weeks. Fixing it took days.
ath9k is mainline kernel code and working at that level is ordinary for us, so a proof of concept was ready within a few days of the theory forming. The hard part had already happened.
proving it in the field
An intermittent bug can go quiet on its own, so a fix that looks successful for a week proves nothing.
We rolled the proof of concept out one access point at a time across the instrumented networks, watching each one for recurrence before widening. Only once the silence had held long enough to mean something did the fix go into a firmware release for the entire customer base.
Two months later, complaints of this type were down roughly 90%.
what this actually took
The client’s engineers had lived with this bug for a long time, and so had the people they’d brought in before us. Not through lack of effort. It was simply out of reach from where they were working.
Closing it needed three things at once: the ability to instrument live production networks without breaking them, the patience to burn through dozens of explanations across weeks of capture data with no guaranteed payoff, and enough 802.11 and driver knowledge to recognise a key lifecycle problem from its downstream symptoms, then fix it in the driver.
We work inside the Linux wireless stack, in cfg80211, mac80211 and the drivers underneath, as routine work rather than as a special occasion.
if this sounds familiar
If you have an intermittent field failure your team can’t reproduce, that’s the kind of problem we take on.
Just book a call and let’s discuss.
More about our wireless and mesh work · Our open-source contributions to the Linux wireless stack