I was changing the switches in the guest bedroom on Sunday, which meant throwing the breaker, which meant, as I worked out about a second too late, cutting power to the living room console the Mac mini lives in, a whole floor above me. It had not occurred to me that a bedroom down here and a cabinet up there would share a circuit. They are now filed in my head as one thing.
That part did not worry me. I finished the switches, put the breaker back on, saw the light on the front of the mini come up, and went back to tidying the room. The machine does one job. Power returns, it boots, the house comes back. That was the whole of my thinking and I did not check it.
The house did not come back. No lights on motion, no switch automations, no car service, nothing, for sixty eight minutes by the logs, until I sat down in front of it and logged in. The moment the desktop appeared every service started up perfectly and carried on as if nothing had happened.
That is the whole bug. Every service in this project was set up as a user agent, which is the easy and obvious way to run something on a Mac, and a user agent needs a logged-in graphical session to exist in. No login, no house. It had never come up because the machine had never lost power before.
What I changed
Automatic login is on now, which I am mildly unhappy about and have accepted, because this machine sits in a cabinet and does one job. Sleep is disabled and it starts back up on its own when power returns, which is the setting I should have found the day I first plugged it in. The services that genuinely do not need a user session have moved to being system daemons instead. I also know which breaker it is on.
The general lesson is that I had tested every part of this except the one that matters, which is what happens when it all stops at once. The detail I keep coming back to is that I was in the house the entire time, twenty feet away, and still did not notice for an hour. An hour of that on a Sunday afternoon is nothing. The same hour at three in the morning, with no lights on the stairs, is how somebody falls down them.
There is a version of this entry where I am annoyed about the outage, except I threw the breaker myself, so there is nobody to be annoyed at. Mostly what I felt was that I had crossed a line without noticing. When I started I wrote down that the worst case was a light being the wrong brightness, and I meant it, and it was true at the time because nothing depended on any of it. Somewhere in the last few months the stairs stopped having a working manual habit attached to them. Nobody in the house reaches for the switch any more. That is the automation succeeding, and it is also exactly how you end up with a house worse than the dumb one when the clever part is down. I do not have a tidy answer to it. I have a machine that boots itself now, and a note to keep an eye on which habits I am letting the house replace.
Watching the watcher
So I added the boring layer I should have started with. A snapshot every fifteen minutes that records what is running, which ports are answering, how long the machine has been up, and how stale each of the event logs is, because a log that has stopped growing is the earliest sign that something has wedged. It appends one line of JSON per run and never touches anything.
On top of that there is a review that runs early every morning, reads the last day of those snapshots, restarts anything obviously stuck, and writes up what it found. It is allowed to do very little: no reboots, no network changes, a hard cap on how many things it can change in one run, and anything it cannot fix twice in a row gets escalated to me rather than retried forever. I would rather it did nothing and told me than got clever at five in the morning.
Worth knowing if you build something similar: asking the system for a list of running services is not the same as asking whether the process is actually alive. I had several that reported as loaded and configured while their process was long gone. Checking for the process directly is the only version I now trust.
And one for the road
I finally put all of this in version control the same week. The first commit included a file of credentials, because I wrote the ignore rules and the commit in the same ten minutes and got the order wrong. I caught it before it went anywhere and rewrote the commit, so no harm done, but it is a good illustration of the actual risk profile of this hobby. The house is not going to hurt me. The habits I bring to it might.