Run and Reliability

What happens to an automation six months after it launches

Automation that matched the work exactly at launch stops matching it two quarters later. Nothing breaks loudly. The inputs drift, an upstream form gains a field, and a customer finds it before a dashboard does.

Uptime shows none of this. The service was available the whole time.

So the monitoring sits on the decisions rather than on the process. Every automated decision is logged with its inputs, its output and its confidence, a sample of them is read by a person on a fixed schedule, and the rate at which a decision is later reversed by a human is the number that gets watched.

A run and reliability engagement is mostly a set of standing commitments:

• every decision logged with the inputs it saw

• a human sample reviewed on a fixed schedule

• alerts on accuracy as well as on uptime

• a scheduled retune, booked before it is needed

Drift usually arrives through a door nobody was watching. A supplier changes a document layout, a team starts using a field for something else, and the classifier carries on returning confident answers about a world that has moved.

Retuning is therefore ordinary maintenance rather than a rescue. Thresholds move on evidence from the review sample, categories are added when the queue shows a new shape, and every change is recorded against the version of the system that made it.

Rollback belongs to the same discipline. Every model, prompt and rule set is versioned, so a retune that makes things worse is undone in an afternoon rather than argued over for a fortnight while the errors accumulate.

The system still working in month nine is not the one that launched best. It is the one somebody kept measuring, and retuned before a customer had to point the problem out.