Building an in‑house AI SRE system can give companies a clearer view of their own infrastructure, according to executives at Chronosphere, a Palo Alto Networks subsidiary.
Why traditional SRE teams are feeling the strain
Modern development pipelines now push code at a speed that outpaces many site reliability engineers. Automated coding tools generate large volumes of machine‑written code, and when failures occur, the underlying logic is often opaque to human operators. Sorting out the root cause, fixing the issue, and preventing recurrence can consume valuable time while downtime remains equally costly.
The case for an internal AI‑driven SRE
Advances in large language models have improved the ability of AI to perform root‑cause analysis, though benchmarks such as OpenRCA still show room for growth. The Chronosphere team argues that companies should first attempt to build their own AI SRE before turning to third‑party solutions. Their reasoning is that creating a custom agent forces an organization to map its systems thoroughly, producing a Markdown knowledge base that the AI can reference when incidents arise.
The process of gathering this information becomes both a learning exercise and a practical asset. “Collecting the data is the journey and the destination,” Heinrich explained on The New Stack podcast. The resulting documentation can feed directly into the AI’s diagnostic routines, reducing the time it takes to pinpoint failures.
Companies that try this approach often discover gaps in their telemetry stack. They may need a dedicated service to aggregate logs and traces, plus an observability platform capable of correlating that data. Chronosphere offers both a telemetry solution and an observability tool, alongside a pre‑built AI SRE product that can be used after the internal effort.
While the idea of building an AI SRE from scratch sounds daunting, the initial investment can pay off by deepening internal knowledge. A modest internal team can prototype an agent, test it against real incidents, and refine the knowledge base over time.
Related: Striking the balance between freedom and growth
One possible outcome is a more resilient system that can automatically suggest remediation steps during off‑hours, when staff may be scarce. This could lessen the need for on‑call engineers to scramble through unfamiliar code, especially in organizations that rely heavily on automated code generation.
Potential pitfalls and next steps
Developing an AI‑powered SRE is not without challenges. Companies must allocate resources for data collection, model training, and ongoing maintenance. There is also the risk of over‑reliance on the AI, which could obscure deeper architectural problems if not monitored closely. Moreover, the OpenRCA benchmark indicates that current models still miss many subtle failure scenarios.
Nevertheless, the Chronosphere executives suggest that the learning curve itself justifies the effort. By forcing teams to document system behavior, organizations gain a clearer picture of dependencies and failure modes that might otherwise remain hidden.
In practice, a firm might start with a pilot project focused on a single high‑impact service. After establishing a reliable data pipeline and a basic AI agent, the team can expand the scope to cover additional components. Success in the pilot can inform decisions about scaling up or integrating commercial AI SRE solutions later.
The trade‑off is short‑term effort for long‑term gains.
Looking ahead, the rise of AI‑generated code will likely push more firms toward automated reliability tooling. Those that invest early in internal capabilities may find themselves better positioned to adapt as the technology matures. The short‑term commitment of engineering effort may lead to a longer‑term reduction in incident response time.
