Your most senior engineers are burning out on alerts that all look the same. Here’s a better way to address these challenges.
It’s 2:47 AM. A Slack ping lands in #incidents. An AI Engineer on the team writes: “prod API returning 502s, users can’t log in.” Somewhere, a Platform Engineer opens their laptop and starts digging: pod logs, HPA status, the last few deploys, any timestamp that might line up. 20 minutes later, they find it. A misconfigured autoscaler had throttled pods too hard after the deploy. The fix takes thirty seconds.
The investigation took 23 minutes, on a good night. Nobody would call that a disaster. It’s just Tuesday, and then Wednesday, and then every on-call shift after that.
This is the L1 triage loop: urgent but rarely novel investigation work that keeps landing on your most experienced people. The problem is almost never something new, which is exactly why it drains so much time and creates so much frustration. We had the same challenges. It was eating a serious chunk of our Software Engineering team’s week before we built a way out of it.
The Bottleneck We Stopped Accepting
Almost every organization faces their version of this same challenge. An AI Engineer hits an issue and posts in Slack. A Platform Engineer, in the middle of something else, drops what they are doing to run the same mental checklist they’ve run dozens of times: pod status, logs for the right service, recent deploys, whether a node is under pressure, and the relevant GitHub Actions runs.
None of that knowledge is secret. It’s a learnable set of questions cross-referenced across a few systems. The reason it costs so much isn’t that any single step is hard. It’s the volume of these alerts, plus the productivity drain of context-switching, plus the fact that the answers they need are scattered across six tools that each take some expertise to navigate.
For teams spread across the globe, the challenge is even greater. Incidents don’t respect time zones. A production outage at 2 AM in Bangalore lands in the middle of the San Francisco workday, while the engineer who knows that service best is asleep somewhere else. Follow-the-sun coverage looks clean on an org chart. However, in practice, you must either maintain large overlapping teams in every region or accept that whoever gets paged at 3 AM might be the wrong person for the incident.
So how do global companies address these problems?
They tend to land on one of two bad options. They either over-staff the rotation to guarantee round-the-clock coverage, or they under-staff it and burn out the people covering the gaps. Neither approach is workable long-term, and both get harder as the platform grows.
The L1 loop is the junk mail of engineering work: necessary to process, but it shouldn’t be consuming your most experienced people’s time.
At RapidCanvas, we estimated our team was spending 40–60% of incident response time on investigations that followed predictable patterns. We built rc-infra-agent to handle that work — and to be the first responder that never sleeps, regardless of in what time zone the alert lands.
What the Agent Does
rc-infra-agent accepts a plain-language description of a problem — from Slack, from a CLI, or from an API call — and autonomously investigates across your infrastructure. It selects which tools to run, in what order, correlates the findings, and returns a plain-language diagnosis with evidence.
The agent covers 68 tools across 6 platform modules:
The Design Decision That Made Platform Engineers Trust It
Our first instinct was to build something fully autonomous — investigate, diagnose, and fix. We resisted it. Not because we didn’t trust the LLM’s reasoning, but because we knew engineers wouldn’t trust a system that could change production state without their knowledge. The first time an autonomous fix does something unexpected, you’ve lost the team’s trust forever.
So we drew a hard line: every tool is classified as either read-only (runs immediately, no approval needed) or needs approval (proposed to a human, never executed until explicitly confirmed). Investigation is fully autonomous. Remediation is not.
Design principle: The agent can investigate everything. It fixes nothing without a human clicking Approve. Only Software Engineers in the designated approver group can authorize state-changing actions like restarting deployments, scaling services, and rebooting VMs.
Human oversight was a critical design decision. Engineers stopped worrying about the agent running rogue. It created trust in the diagnoses because the tool’s relationship to their systems was read-only by default. Trust was built incrementally: first the investigations were right, then the proposed fixes were sound, and eventually approving them became reflex rather than a source of anxiety.
Why Sophisticated Architecture Lost to a Clear System Prompt
Our first version of the investigation engine had hand-coded logic for common failure patterns. If the query mentioned “502,” check ingress first. If “OOM,” pull pod resource metrics. We built routing tables, scoring heuristics, and a complex orchestration layer to sequence the right tools in the right order.
It was brittle. Every new service type, every new failure pattern, required new code. The investigation coverage was only as wide as our engineering team’s ability to predict future failure modes and encode them as rules.
We rebuilt with a different philosophy: the LLM is the reasoning engine, not our orchestration code. We provide a well-written system prompt with platform knowledge, including service-to-log mappings, cluster registry, known failure heuristics, and a set of tools. The LLM decides which tools to call and in what order, based on the evidence it finds along the way.
The model is the reasoning engine. Context is what makes it reason about your infrastructure.
The result was dramatic. Investigation code went from ~3,000 lines to ~300. Coverage expanded from a fixed set of encoded patterns to anything the LLM can reason about given the right tools and context. The system prompt became the architecture.
The Context Layer That Made It Specific to Us
Generic tool access alone wasn’t enough to make the agent useful. An agent that can query Kubernetes logs is valuable, but an agent functions better when it knows:
- Which Kubernetes cluster to look at for a given RapidCanvas service URL
- Which log identifier maps to which microservice
- What failure pattern typically presents as a 504 on the FastAPI inference endpoint
That insight turns an investigation from “here are some logs” into a diagnosis.
We encoded our platform knowledge explicitly: a map of every microservice to its GitHub repo and log identifier, a cluster registry covering all seven environments (GCP/AWS/Azure, prod through test), URL-to-cluster routing so a pasted app URL automatically resolves to the right context, and investigation heuristics derived from six months of recurring incidents.
This is the institutional knowledge that competitors can’t replicate. The tooling is generic. The context is ours. It helps create a competitive moat that grows over time.
The Cost-Effective Way to Resolve Incidents
The most expensive incident response system is one built without the right context — a generic agent that blindly queries every system, returns walls of logs, and calls tool after tool hoping the model finds the answer. That approach burns LLM spend fast and still produces noisy, hard-to-act-on results.
Building it right flips that equation. The agent needs precise platform context. For example:
- Which service maps to which cluster
- Which log pattern signals which failure
- Which tool to call first for a given symptom class
That context enables the agent to reach a correct diagnosis in far fewer steps. Fewer tool calls means less context consumed, which means lower cost per investigation. The quality of the context directly determines the efficiency of the agent.
The same principle applies to LLM selection. Not every investigation requires the most capable model. For routine L1 triage where the tools do the heavy lifting, a well-chosen mid-tier model handles the reasoning effectively. Reserving higher-capability models for complex cross-system correlation, in situations where genuine reasoning across partial signals matters, keeps costs proportional to the task. Right-sizing the model to the problem is an important design decision.
On the human side, the shift from investigation to validation compounds these savings further. Platform Engineers reviewing a diagnosis rather than producing one can handle significantly more incident volume without a proportional increase in headcount or on-call burden. A solution with the right context, the right model, and the right constraints is a cost-effective one.
What We’d Tell Teams Starting This Today
If your organization is looking for a solution to address this challenge, here are some pieces of advice:
- Start with investigation only. Read-only tools build trust before write tools can be considered. Ship the diagnosis layer first; earn the right to propose fixes later.
- Human approval is a key feature. Human approval steps build trust. Without this critical oversight step, you will struggle to create adoption.
- Let the LLM do the reasoning. Hand-coded investigation logic doesn’t scale. A well-contextualized model with the right tools will out-cover any orchestration logic you hand-write.
- Platform context matters more than tool breadth. An agent with 10 tools and deep domain knowledge outperforms one with 50 tools and generic prompts. Encode what your team knows.
- Put it where the alerts already land. A Slack-native agent removes all adoption friction. If engineers have to open a new tool to get a diagnosis, most won’t bother.
Platform Engineers are not on-call to be search engines for log data. They’re on-call because complex systems fail in complex ways that require human judgment to navigate. The goal of rc-infra-agent is to handle the legwork like the data collection, correlation, and pattern-matching so that when a human does engage, they’re engaging with a diagnosis, not starting a scavenger hunt.
For RapidCanvas, mean time to acknowledge dropped from minutes to seconds. Proposed fixes are reviewed and approved in context, in Slack, without switching tools. Platform Engineers are reviewing diagnoses instead of producing them.
The L1 triage loop still exists. It just no longer requires a human to run it.
If you would like to learn how RapidCanvas can help you build AI that transforms your business, visit our website, book a meeting with our team, or read verified reviews on G2.
