Your AI Agent Returned a 200. That Doesn't Mean It Worked.

A support agent, powered by an LLM, tells a customer their refund has been processed.

It hasn't.

The agent called the refund tool. The tool call contained a malformed amount field. The API rejected it. The model received the response, interpreted the interaction as successful, and confidently told the customer their refund would arrive in three to five business days.

No exception was thrown. Latency was normal. The dashboard showed a healthy 200. Your monitoring did exactly what you built it to do: it confirmed that the request completed. It never checked whether the answer was true.

That distinction is becoming one of the most important production problems in AI systems.

With traditional software, we got very good at answering "did the system execute successfully?"

With AI agents, we increasingly need to answer a harder question:

Did the system actually accomplish what it was supposed to accomplish?

Those are not the same question, and the gap between them is where the real incidents live.

The assumption your production stack was built around

Think about what monitoring has traditionally meant.

Error rate.

Latency percentiles.

Throughput.

Availability.

CPU and memory.

Database health.

Queue depth.

Every one of these is valuable, and none of them are going away.

But they mostly tell you something about the execution of the system:

Did the request reach the service?

Did the service respond?

Did the dependency respond?

Did the operation complete within the expected time?

For deterministic software, that gets you remarkably far.

A function receives an input, follows a defined set of instructions, and produces an output. If the implementation is correct, the output should be correct for the defined input. When something breaks mechanically, there's usually a way to observe it: an exception, a failed assertion, a timeout, a non-success status code.

Of course, traditional software can still produce incorrect business results. A bug can return a perfectly valid value that happens to be wrong.

But with an AI agent, incorrectness becomes much harder to distinguish from success.

The agent can execute every step cleanly and still misunderstand the user's intent.

It can call the right tool with the wrong arguments.

It can retrieve the wrong information and confidently build an answer around it.

It can produce a perfectly formatted response that contains a fabricated detail.

And the system reports all of this as healthy, because mechanically, nothing failed.

That's the shift.

The production problem isn't that monitoring stopped working. It's that execution health and outcome correctness are no longer the same thing.

Wrong but healthy

"AI can hallucinate" has become such a familiar phrase that it's lost some of its usefulness.

The production question is more specific:

What does an AI failure actually look like when nothing technically fails?

The right tool, wrong arguments.

An agent needs to issue a refund. It calls the correct API but sends the wrong amount or currency field. The API rejects the operation. The model sees a response come back and generates a reassuring summary anyway.

The request completed.

The business operation didn't.

The answer to the wrong question.

A customer asks something ambiguous. The agent chooses one interpretation, produces a detailed and fluent answer, and moves on.

No exception.

No timeout.

No failed dependency.

Just a misunderstanding, delivered with total confidence.

A plausible detail that isn't true.

The agent states a policy number, a date, a compatibility claim, a business rule, something that sounds perfectly reasonable and has no basis in the available data.

Nothing in the infrastructure layer necessarily notices, because there's nothing to notice from that layer's point of view.

A different answer tomorrow.

You test a scenario repeatedly and it works.

Later, a slightly different context, conversation history, model version, prompt, or tool result causes the agent to take another path.

The system didn't become unhealthy.

Its behavior simply changed, and that's a much harder thing to catch with a threshold alert.

These aren't infrastructure failures in the traditional sense.

They're correctness failures, and that distinction matters more than it might sound like it does.

Three different meanings of "success"

One way to make this concrete is to stop treating success as a single metric.

For an agent, there are at least three levels.

Transport success.

Did the request complete?

HTTP 200

The service responded.

Good.

But that tells you very little about what happened next.

Execution success.

Did the system execute the requested operation?

Refund API → accepted → refund created

Better.

Now you know the tool actually did something.

But even this may not be enough.

Outcome success.

Did the system accomplish what the user actually needed?

Customer requested refund
→ correct refund created
→ correct amount
→ correct account
→ customer informed accurately

That's the level the user ultimately cares about.

A 200 proves transport success.

A successful tool call proves execution success.

Neither necessarily proves outcome success.

That is the production gap.

And it isn't unique to AI.

What AI does is make the gap much harder to ignore, because the system itself can generate a convincing explanation of why everything went well, even when it didn't.

Why this changes the monitoring problem

There's a temptation to respond with:

"Just add better logging."

You should absolutely add better logging.

But logging alone doesn't solve the problem.

Suppose your trace shows:

POST /refund
200 OK
Latency: 412ms

That's useful.

Now suppose you also log the tool call itself:

Tool: CreateRefund
Amount: 1500
Currency: INR
Result: Rejected

Much better.

But your agent still tells the customer:

"Your refund has been processed."

You now have excellent visibility into what happened, and you still haven't automatically detected that the final answer was wrong.

That's because the missing signal isn't another infrastructure metric.

It's semantic correctness: some way to determine whether what the system said and did actually matches what was supposed to happen.

Sometimes that check can be deterministic.

Sometimes it requires a second model, a human reviewer, or asking the user to confirm.

The implementation varies by domain.

The production requirement doesn't.

You need a signal for correctness, separate from a signal for completion.

What production monitoring for agents should actually watch

None of this requires throwing away what you already know about production engineering.

The same disciplines still apply.

You just have to point them at the right layer.

Sample real outputs continuously

Don't treat evaluation as something that happens before launch.

Sample a percentage of real conversations and evaluate them on an ongoing basis:

Was the answer correct?

Was the user's intent understood?

Was the response grounded in the available information?

Did the agent actually complete the requested task?

A second model can provide rubric-based scoring for some workflows. High-stakes ones may need a human in the loop.

The important part is that evaluation becomes a permanent production process, not a launch checklist item.

Your uptime monitoring doesn't stop after deployment.

Your output evaluation shouldn't either.

Log the entire tool-call chain, not just the final response

Record the:

• User request
• Agent decision
• Tool selected
• Arguments sent
• Tool's actual response
• Agent's interpretation of that response
• Final answer given to the user

The gap between the tool's real result and the model's interpretation of it is often where the incident lives.

In the refund example, that gap is the entire bug.

If you only retain the sentence:

"Your refund has been processed."

you've thrown away the exact evidence you'd need to explain why the system said it.

Measure outcome correctness as its own metric

Not whether the model returned something.

Not whether the answer was well formed.

Not even whether the tool call succeeded.

Ask whether the system accomplished the intended outcome.

For a support agent, that might mean the correct policy was applied.

For a coding agent, it might mean the tests pass and the requested behavior actually exists.

For a retrieval system, it might mean the answer is genuinely supported by the source material.

The definition changes by domain.

The principle doesn't.

Version prompts like code

Prompts are production dependencies, and they drift the same way config drifts.

Someone changes an instruction in March to fix one complaint.

Someone adds another in April.

A third person edits a tool description in May.

By August, nobody remembers why the system behaves differently than the version that worked in January.

If you can't answer what the agent's instructions looked like last Tuesday, you can't debug a regression that started last Tuesday.

Put prompts in version control.

Review changes to them.

Be able to roll them back.

Treat meaningful prompt changes with the same discipline you apply to code changes.

Separate request success from task success

These look identical when everything is working and become very different the moment something goes wrong.

A system can post a 99.99% request success rate while its actual task completion rate sits meaningfully lower.

If you only track the first number, the dashboard looks excellent while users are quietly having a bad experience.

Give uncertainty somewhere to go

An agent that isn't sure shouldn't only have two options:

Answer
Fail

There should be a third path, something closer to:

I'm not confident enough to answer this safely.

That can trigger:

• A clarifying question
• Human review
• A deterministic check
• A safer fallback

This only works if the escalation path is real.

An uncertainty signal nobody monitors is just another unread log line.

A hypothetical incident

Consider a mid-sized fintech team that deploys an internal agent to help support staff draft responses to billing disputes.

The agent reads the customer's message, retrieves account history through a tool call, and produces a draft for a human representative to review before sending.

The first month looks great.

Response times fall.

Support staff like the tool.

The infrastructure dashboards stay clean.

No meaningful increase in latency or error rate.

No dependency outages.

Everything green.

Then, during a routine review, a support lead notices several drafts referencing a "loyalty discount" that doesn't actually exist as company policy.

The model didn't invent this from nothing.

A handful of historical support transcripts, sitting in the context the agent had access to, included a one-off manual discount a manager had approved years earlier for a single customer.

The model treated that isolated example as if it were a general rule, because nothing in its instructions told it that one example isn't a policy.

The infrastructure never noticed any of this.

The agent never crashed.

The retrieval system worked exactly as designed.

The model responded fluently, every single time.

The only reason the problem surfaced at all is that a human happened to read a batch of responses closely, out of habit, not because any system flagged them.

That's not really a story about a bad model.

It's a story about a production system with excellent infrastructure observability and almost no output observability.

The system could tell the team whether it was running.

It couldn't tell them whether it was right.

This isn't a replacement for traditional observability

The answer to all of this isn't abandoning APM, logs, traces, health checks, or alerts.

You still need all of them.

And an agent sitting on top of broken infrastructure is still broken infrastructure.

What's changed is that those signals are no longer sufficient on their own.

Think of it as adding a layer:

Infrastructure health
        ↓
Execution health
        ↓
Task correctness
        ↓
Business outcome

Traditional observability is exceptionally good at the first two.

Agentic systems force a more deliberate investment in the last two.

That's not AI replacing production engineering.

It's production engineering applied one layer deeper than it used to need to go.

Testing once isn't the same as knowing

With a traditional bug, you can usually reproduce the failure, fix the root cause, add a regression test, and gain strong confidence that the specific failure won't recur.

That confidence comes from the fact that, for a given piece of deterministic logic, a passing regression test provides a strong guarantee about that specific behavior going forward.

With an agent, a successful test is evidence, not a permanent guarantee.

The model may behave differently with slightly different phrasing.

A tool may return a different result.

The context may change.

The prompt may change.

The underlying model itself may change, sometimes without you touching anything, because a provider updated the model behind the API you're calling.

Your application may add another tool that changes the agent's available paths.

The right response isn't pretending you can eliminate that uncertainty.

It's designing the production system around the fact that it exists:

• Continuous evaluation
• Observability that goes beyond infrastructure
• Controlled and versioned changes
• Explicit escalation paths
• Deterministic verification wherever it's actually possible

The organizational problem hiding underneath

There's also an ownership question that tends to go unanswered until something goes wrong.

Who reviews sampled outputs?

Who owns the evaluation criteria?

Who maintains prompt versions?

Who investigates a sudden drop in task correctness when every infrastructure metric stays green?

Who gets involved when the agent starts repeatedly producing plausible but unsupported answers?

If the honest answer to any of these is:

"Nobody, currently."

the system isn't production-ready just because the deployment happens to be stable.

This is the same lesson production engineering has taught for years in other contexts.

If a payment service matters, somebody owns its reliability.

If a database matters, somebody owns its health.

If an agent is making decisions users depend on, somebody has to own whether those decisions are actually correct, on an ongoing basis, not just at launch.

AI doesn't remove operational ownership.

It expands what operational ownership has to mean.

The one-sentence version

A 200 proves your system answered.

It doesn't prove your system succeeded.

That gap between a request that completed and a task that actually got done is where the real production incidents live.

More from the Series

• Silence Is a Design Decision
• CQRS Simplified the Design. It Complicated Production.
• You Are Fixing the Wrong Thing

The True Code of Production Systems is a series about the decisions that only become visible when something breaks in production.
Read the full series at The True Code of Production Systems