Modern digital products are judged by what happens after they go live. The speed of feature delivery matters, but users don’t differentiate between product innovation and product reliability. They don’t notice whether an issue is caused by a new release, an infrastructure dependency, a background worker, or an edge case in user behavior.
Users only care that a product functions as expected. This is why support and site reliability engineering (SRE) teams are re-evaluating how AI harness engineering can support production operations.
Harness engineering in software delivery typically focuses on development, testing, and release workflows. However, this approach is equally relevant to production operations. The harness does not perform SRE tasks itself; rather, it provides the structured environment for the AI agent, including tools, context, instructions, and guardrails for human review.
I applied this approach to a specific support and SRE scenario: production log anomaly detection. While the use case was straightforward, its impact was significant. We were not merely automating a manual task, but enabling a workflow that was previously too manual and labor-intensive to execute consistently.
Before this AI agent workflow was supported by a harness, many issues would only surface when a client noticed and reported them. With the agentic workflow in place, an AI agent now reviews production logs on a schedule, flags potential anomalies, and allows engineers to investigate before users are affected.
AI harness engineering for SRE is valuable not as a replacement for engineers, but as a practical step toward proactive incident management.
What AI harness engineering means in support and SRE
In this context, AI harness engineering involves designing an operating environment that enables AI agents to reliably support production workflows.
That environment usually includes:
- Access to operational tools, such as log platforms, monitoring systems, APIs, or CLIs
- Context about the product, infrastructure, and expected system behavior
- Instructions that define what the agent should and should not treat as anomalous
- A reporting workflow, such as Slack or an incident management tool
- Human review points
- A feedback loop for refining prompts, context, and logging quality over time
It’s important to make this distinction because harness engineering does not resolve incidents on its own. Instead, it enables AI agents to operate within SRE workflows by providing sufficient context and structure.
The harness gives the AI agent the same resources a new engineer would need: access, documentation, examples, constraints, and a clear task. Without these elements, the agent operates without direction.
Why this matters: Some reliability work was previously too manual to justify

AI can automate existing tasks, but a key lesson from this project is its ability to make new operational workflows practical.
In many production systems, there are large volumes of error logs that do not necessarily indicate that something is broken. Unexpected user behavior may create an error. A known edge case may produce a warning. A noisy integration may generate logs that look concerning but have no real business impact.
Treating every error as an incident creates alert fatigue. Ignoring logs entirely creates the opposite risk: genuine issues may sit unnoticed until users or clients report them.
Manual daily review of this data is rarely feasible because it is time-consuming, repetitive, and low-yield. This is exactly the type of workflow AI agents can support with the right harness.
In our case, the AI agent analyzes production logs, applies contextual instructions, and reports potential anomalies to engineers. A human still validates the findings, but the initial daily review no longer requires manual log searches. A significant shift, as it enables teams to perform reliability work that was previously impractical.
AI for incident detection: How we built an agent-based anomaly detection workflow
The first use case I implemented was AI for incident detection through agent-based anomaly detection in production logs.
The setup was intentionally lightweight. I instructed an AI agent to use the Loki log CLI and provided context on what should not be considered an anomaly. The agent runs daily in our CI environment and reports any suspicious findings in Slack. I then review the report and determine if action is needed.

This is not a fully autonomous incident response system. It is a focused, human-reviewed agentic workflow that supports earlier detection.
This approach is useful because it is bounded, practical, and easy to improve over time.
Reducing MTTD in production through earlier anomaly detection
One of the clearest benefits of this approach is the potential to reduce Mean Time To Detect (MTTD).
Many teams already use dashboards, alerts, and threshold-based monitoring. Those tools remain essential, but they do not catch everything. Some production issues do not immediately create visible user impact. Others appear as subtle patterns in logs before they escalate into a larger failure.
For example, a background transaction processor in our system lost access to an AWS S3 bucket. Since this affected a background worker, users would not necessarily notice it straight away. By the time a client report came in, the issue could already have affected a meaningful number of transactions.
With agent-based anomaly detection, such issues can be identified earlier. The AI agent detects unusual log patterns related to access failures and flags them for review before they become apparent to users. This changes the operating model.
Instead of:
User report → support ticket → engineering investigation → fix
The workflow becomes:
Anomaly detected → engineer review → investigation → fix before wider impact
That is the practical value of proactive incident management. It gives teams a chance to act before customers become the monitoring system.
How agentic automation supports proactive incident management
Detection is only the first step. An effective AI harness can also support incident triage by reducing the amount of manual context-gathering engineers need to do.
Incident response often slows down because engineers must switch between tools such as logs, dashboards, deployment history, documentation, tickets, infrastructure systems, and messaging threads. Investigation is usually quick once the right context is available; the delay is often in gathering that context.
An AI agent equipped with a well-designed harness can help by surfacing relevant context earlier. Depending on the maturity of the setup, agents can be structured to:
- Correlate anomalies with recent deployments
- Surface relevant log excerpts
- Reference runbooks stored in markdown repositories or Atlassian Confluence
- Identify likely affected services
- Analyze project code and suggest possible causes for review
- Route findings into incident workflows
- Generate a proposed fix for a developer to review
The important phrase here is for review.
The goal is not to remove human judgment from incident response, but to reduce manual investigation so engineers can make better decisions faster. Even modest improvements in triage speed can reduce Mean Time To Resolve (MTTR), shorten escalation paths, and lower the operational burden on support and engineering teams.

Signal quality has to be engineered
One of the biggest risks in AI-assisted support and SRE workflows is noise.
A poorly tuned agent that floods Slack with false positives does not improve reliability; it creates a new monitoring burden. Signal quality must be engineered into the harness.
In practice, this requires maintaining context over time. I keep a list of non-anomalies and update prompts or instructions when the agent reports false positives, so it can better interpret similar patterns in the future. I also update the application code as needed to avoid unnecessary error logs.
There are four important practices here.
First, define known non-anomalies. If there are benign errors, expected edge cases, or routine warnings, the agent needs to know that.
Second, tune prompts continuously. False positives are not just noise; they are feedback on the harness.
Third, improve the logs themselves. If applications log too many unnecessary errors, the AI agent has a weaker signal to work with.
Fourth, keep humans in the loop. The agent supports detection, but engineers validate findings and decide what action to take.
It’s an evolving reliability practice, not a set-it-and-forget-it automation.
Where teams can start with AI harness engineering for SRE
The good news is that this kind of workflow doesn’t require a large-scale transformation program to get started. A focused log-anomaly-detection harness can often be piloted quickly. In our case, the first useful version took about two days to integrate at a sufficient level. While timelines may vary, this shows the entry point can be practical.
For teams starting out, I recommend beginning with a narrow focus.
Begin with one production signal source. Logs are often the easiest place to start because they already contain rich information about system behavior.
Choose a bounded use case. Start with anomaly detection before attempting autonomous remediation.
Use existing tooling. If your team already uses platforms such as Grafana and Loki, build on what is already available.
Give the AI agent strong context. Assume the agent knows nothing about your system unless you tell it.
Schedule structured reviews. The workflow improves when engineers review findings and refine the context over time.
Treat it as an experiment. Measure the quality of detections, track false positives and iterate.
This approach minimizes risk and provides a practical path toward proactive incident management.
The readiness test: Would a new engineer have enough context?
The biggest implementation mistake is assuming the AI agent can compensate for missing operational knowledge. It cannot do so.
A simple readiness test is this:
If you hired a new engineer unfamiliar with your system and assigned this task, would they have enough context and tools to complete it without repeated clarification? If not, an AI agent likely will not either.
If documentation is fragmented, tools are hard to access, and the task is unclear, the agent will struggle. If a human engineer would need several follow-up questions before starting, the AI agent needs better context to contribute reliably.
Teams need three foundations in place.
Context
AI agents need usable context about the system, expected behavior, known issues, and operational boundaries. I increasingly prefer markdown-based repositories for this context, as structured files are easier for agents to process than fragmented documentation.
Tooling
Operational tools should be agent-friendly. APIs, CLIs, and emerging model context protocols (MCPs) make it easier to connect agents to real workflows. If a tool is difficult to wire into an agentic workflow, teams may need to reconsider its fit for future operating models.
Operational understanding
Teams need a clear understanding of agent workflows to structure instructions effectively. The agent requires a defined task, examples, constraints, and escalation points. Without these, the workflow becomes unpredictable.
Common mistakes to avoid in AI harness engineering for SRE
Early AI harness engineering efforts often fail for common reasons.
A common mistake is treating agents as autonomous before they have enough context. Agents do not magically understand your architecture, product behavior, or incident priorities. The harness must provide this information.
Another mistake is attempting to automate too much too soon. A focused anomaly detection workflow is a better starting point than a broad autonomous remediation system.
Poor-quality signals lead to poor outcomes. If logs are noisy, inconsistent, or filled with unnecessary errors, the agent will struggle to distinguish real anomalies from routine events.
Teams should avoid overengineering around current tooling. Agentic workflows and best practices are evolving quickly. The most durable investments are likely context sources, tool accessibility, and operational patterns, rather than rigid workflow architecture.
Finally, teams should not assume automation will immediately reduce support load. In this case, we were not previously monitoring log anomalies manually, so the workflow did not replace an existing process. Its initial value was improved reliability; workload reduction may follow as the workflow matures.
What mature agentic SRE workflows could look like
It is too early to define a universal mature state for AI harness engineering in support and SRE. The field is moving too quickly, and best practices are still changing.
However, the likely direction is clear. In the next one to two years, product delivery teams will need broader AI strategies that connect development, support, and operations through shared context and reusable tooling.
A more mature setup may include:
- Continuous anomaly detection across multiple production signals
- Agent-assisted incident investigation
- Shared operational context across engineering, support and platform teams
- AI-supported runbook execution with human approval
- Preventive issue identification across systems
- Triage workflows that connect logs, code, deployments and incidents
The long-term opportunity is not isolated automation, but a unified operating model for production reliability. This operating model begins with a practical question: Where is operational work still too manual, repetitive, or low-yield for humans to perform consistently, yet valuable enough for AI agents to support?
Why AI harness engineering for SRE matters to engineering leaders
For CTOs, VPs of Engineering, platform leaders, and support leaders, the key question is not just whether AI can automate tasks, but whether AI harness engineering can improve production resilience.
Early use cases suggest it can, especially where teams start with bounded, human-reviewed workflows.
Agentic workflows can help teams detect emerging issues earlier, improve incident triage, reduce MTTD in production, and establish preventive reliability practices. They also help teams move beyond reactive support models, where customers are often the first to notice production failures.
The most important shift is that AI agents enable previously impractical workflows. For example, daily production log anomaly review is now feasible with an AI agent, whereas most teams would not assign an engineer to do this manually. Harness engineering is strategically important because it transforms AI from a general capability into a structured production workflow that can be reviewed, improved, and trusted over time.
The future of SRE is proactive incident management, not reactive support
Support and SRE have traditionally been measured by how effectively teams respond to incidents. AI harness engineering introduces a different ambition: helping teams identify and address issues before they become user-facing.
This does not require fully autonomous operations or waiting for a mature future state. Teams can start now with focused, low-risk workflows such as agent-based anomaly detection, learn through iteration, and expand as signal quality improves.
In modern digital systems, reliability is part of the product experience. A well-designed AI harness gives agents the tools, context, and guardrails needed to support that experience, not by replacing engineers but by helping them act earlier.
For engineering leaders, the real opportunity is moving from reactive incident response to proactive reliability, one practical workflow at a time.
How Star can help
At Star, we help product, platform, and engineering teams move from AI experimentation to production-ready workflows that deliver measurable operational value. If your team is exploring AI harness engineering for SRE, support, or broader product delivery, we can help you identify the right first use case, design the agentic workflow, connect the necessary tools and context, and build the guardrails needed for safe, human-reviewed adoption.
Contact our experts to explore how AI harness engineering could improve reliability across your production operations.
FAQ
AI harness engineering for SRE is the design of structured workflows that enable AI agents to support site reliability engineering tasks. The harness gives the agent access to the right tools, context, instructions, permissions, and review points, so it can assist with tasks such as anomaly detection, incident triage, and proactive monitoring.








