智能监控提升系统运维

A new level of operational excellence is possible. Our team uses Claude Tag heavily, which has helped us monitor each component of the system.

For example, we created Slack channels where we tasked Claude with being obsessively focused on the health of a single component. We call these Claudes “superfans.

We have a scion-daemon superfan, a scion-cache superfan, an Antique merge queue superfan, and so on.

They have access to our telemetry and logs, and act like an experienced engineer who has nothing better to do than to refresh charts and look for anything strange.

We also have traditional alerts for all of our systems, but our superfans can detect subtle and novel issues that make these alerts seem one-dimensional by comparison. This is so obviously the path forward for software systems operations.

Modern software systems, like the LLMs that write them, are becoming more “grown” than “designed”.

Yes, human engineers often still give the high-level design and shape to the system, and yes, software systems (for now) are much more interpretable than LLMs – since you can just sit down and read the code, if you are so inclined.

However, with the volume of code being written and shipped increasing exponentially, the amount of actual eyes-on-code knowledge that any human operator of a system has is decreasing. Systems monitoring was never easy.

There is a whole industry of observability SaaS companies – Sentry, Datadog, Splunk, Honeycomb, to name a few. Observability is expensive. Doing it well requires a specialized human skill set and relies on building experience operating real systems at scale.

One can really only develop these skills by working in industry. This is a domain that takes years to train junior engineers on. As with seemingly everything in the software industry in 2026, system monitoring is poised for a step change due to agents. I…

am surprised I have not seen more people write about this, especially in the current milieu of “agent frenzy”. It’s right there in front of us all, and it works.

There have been a bunch of startups trying to do the “AI SRE”, but very little writing from practitioners. But you don’t really need a full “AI SRE” to improve operational excellence, nor do you need a built-out SaaS product to do so.

As of the Opus 5.5 / Astra class of models, with a bit of care, it’s entirely possible to make a good agent-driven monitoring system that “just works”.

Case study: change-point monitoring I’ll ground this discussion in a problem that is near to my heart: production monitoring of change points, like binary releases and config pushes.

The idea is that you want to do specialized, intensive monitoring around critical points of change to a system; these change points are more likely to trigger a regression, and thus if you monitor these events closely, you can limit its impact.

To quote the clichéd line from the Google SRE book, “~70% of outages are due to changes in a live system”. This is a wicked problem. I’ve been working on it in some form since 2023, and it’s been hard to get traction on.