Project: Soft Rains Entry 13

Changing the router with the house switched on

A phased cutover, an authentication detour, and twenty minutes where nothing could resolve a hostname.

Everything in this project has been sitting on the router the internet company gave me, which I have wanted to replace since the day I moved in, and which I had been putting off because replacing it means taking the network down while the house is running on the network. I finally bought a GL.iNet Flint 3. It turned up on Monday and I did the swap on Tuesday, which is a gap entirely explained by wanting to write the plan down first.

It arrived after a week in transit during which I wrote a scheduled job to poll the courier's tracking page twice a day and log the status into my own system, because apparently I could not simply look at the page like a person. That job is now retired. I am choosing not to examine what it says about me.

An authentication detour

I wanted to drive the whole configuration over the API rather than clicking through a web interface, so that every setting would be in version control and I could read it back afterwards to prove it took.

The login is a challenge-response, and every example I could find hardcodes md5-crypt for hashing the password. Mine kept failing. It turns out this firmware uses sha256-crypt, and, crucially, the challenge response tells you which one it wants: the algorithm is right there in the reply and everyone, including me, was ignoring it and assuming. So I wrote a small pure-Python sha256crypt, checked it against the published test vectors before trusting it with anything, and dispatched on whatever the challenge actually declares rather than on what the examples assume. Then it logged in first try.

Doing it in phases, which is not like me

I wrote a runbook first. An actual document, with a verification step after each phase and a rollback. I do not normally do this for my own house and I am now convinced by it, because the thing about swapping a router is that the tool you would use to fix a mistake is the thing you have just unplugged.

Phase one was everything that could be done while the old gateway was still serving the house: address reservations for every device that needs a stable one, the DHCP pool, the option that hands out my own DNS server, IPv6 switched off. All committed over the API and all read back afterwards to confirm.

Two details I would repeat. I held the new radios off through the whole flip, so there was never a window where two access points were broadcasting the same network name and fighting over clients, which the written plan had allowed for and which would have been miserable. And I armed a small agent on the Mac to bring its wired connection back automatically if the swap stranded it, since the Mac is how I was driving all of this and losing it mid-cutover would have meant finishing the job by crawling behind furniture with a laptop. It fired once, at 16:51:51, exactly when the cable moved, and then I retired it.

One casualty, which I had predicted in the runbook and still managed to be mildly annoyed by: the Hue bridge was wired into the old gateway rather than the switch, so every light in the house went away until I recabled it.

Which prompted the other job I did today, and the one I should have done months ago. Everything that this house actually depends on now lives in one place. The Mac mini has come out of the living room console, where it had been sitting next to the Apple TV since the day I bought it, and gone downstairs into the guest bedroom alongside the new router and the Hue bridge. One room, one set of cables, one thing to look at when something is wrong.

There is a joke in there that I only noticed while I was carrying it down. The guest bedroom is the room whose light switches I was changing last week when I threw the breaker and took the whole house down for sixty eight minutes without noticing. The machine now lives in the room that killed it. At least this time I know which breaker it is, and the machine has since learned to boot on its own.

Twenty minutes of a very confusing outage

Then the interesting failure. With everything moved across, the Mac could not resolve hostnames. Not slowly, not intermittently, at all, through the normal system calls that every application uses. Meanwhile querying DNS directly with a command line tool worked perfectly, instantly, every time.

The obvious suspect was my own DNS filtering, which had just become the whole house's resolver about ninety seconds earlier. It was innocent. The giveaway was that pointing at a well known public resolver failed in exactly the same way, so the problem was not what was answering, it was that the system was not really asking.

What had actually happened is that macOS still had the network service order from phase one, when I had deliberately put Wi-Fi first, and the resolver had wedged itself against the wrong interface through all the churn of things coming up and down. Setting the order back to wired-first, which was where it was always supposed to end up, rebuilt the resolver instantly. Twenty minutes, no data lost, one car watchdog timeout that recovered by itself.

That lesson is now written into the runbook in capitals for next time: restore the service order immediately after the swap, before touching anything else.

What I got out of it

The payoff is that my own DNS filtering is now the whole house's resolver, handed out by the router itself rather than configured device by device, which means it covers the things I cannot install software on. It is carrying about a million blocking rules across a general list, a bigger one, and a malware and phishing list I added the same evening. Everything on the network gets it, including guests, including the television.

The genuinely satisfying part is smaller than that. Every setting on this router was applied from a script, read back to confirm, and committed. When it dies, or when I do this again in three years having forgotten everything, the configuration is a file I can read rather than a memory of which tabs I clicked.