Video Transcript (English)

Picture this: your agent is live in production. Your monitoring dashboards say everything is fine: error rates are flat, latency is within SLOs, throughput looks normal.

But one out of every twenty-five users is quietly getting a wrong answer. There are no error codes, no alerts, but incorrect information is provided.

This is the defining challenge of agentic AI in production: the failures that matter most are the ones your existing monitoring stack was never designed to catch.

In this lesson, we’ll draw a clear line between monitoring and observability, and see why agentic systems have needs that land firmly on the observability side of that line.

Monitoring starts from a simple idea: you decide in advance which signals matter, set thresholds, and wait to be alerted if those thresholds are crossed.

You might track CPU usage, error rate, request latency, queue length. For each, you define what “normal” looks like and when to page the team.

Monitoring shines at catching the failures you already expected. When a disk is full, when an error rate spikes, when there’s an uptick in request timeouts. All of the things you plan for.

Observability takes a different approach. Instead of predefining every question, you design your system so you can ask new questions later, using the data it emits. It allows you to deal with the unexpected.

But what about the failures you didn't anticipate, the ones you didn't know to build a threshold for? That's where observability comes in.

When something unexpected happens in production, you want to be able to ask, “What happened? Where? When? And why?” and be able to find an answer.

With observability, you understand system behavior by examining logs, metrics, and traces, and you can do this in near real-time.

For deterministic services, the same input almost always produces the same output. Code paths are fixed, contracts between components are explicit, and state is limited.

When something breaks, you can often reproduce it locally: send the same request, trigger the same bug, and capture the stack trace.

An AI agent operates by different rules. Execution is non-deterministic: the same user query can produce different outputs on different runs.

The agent dynamically chooses tools, calls other services, maybe even coordinates with other agents, all at runtime. This means two runs could vary drastically in their context and tool usage, amongst other things.

The output of a non-deterministic system is shaped by the model’s reasoning, the retrieved context, and the tools it calls.

State is also more complex in agentic AI systems, because conversation history, retrieved documents, and intermediate results all influence what happens next.

The result is a black box. You can see what went in and what came out, but everything in between is invisible unless you instrument it deliberately.

Let’s see what this looks like when something goes wrong.

A user gets a wrong answer from your agent. From the outside, it looks like a single failure: bad output.

But that failure could have started in at least four different places.

The model might have misinterpreted the task and hallucinated a response.

A tool might have timed out, returned stale data, or responded in a format the agent didn’t quite handle correctly.

Retrieval might have pulled back irrelevant or low-quality documents, so the agent reasoned from bad context.

Or the problem could exist at the infrastructure level, such as rate limits, slow databases, resource contention under load.

It’s also worth noting that these issues don’t always occur in isolation. A single, significant error, like a tool returning incorrect data, may surface quickly and even self-correct. But multiple smaller errors occurring simultaneously can quietly compound into a bad output with no obvious single cause.

This counterintuitive dynamic is part of what makes agentic AI systems a challenge to debug.

Each of these error sources leads to a different remediation path, but from the outside they all look identical: one wrong answer.

Without observability, you can’t tell which category you’re dealing with. You’re left guessing whether to tune prompts, change tools, adjust infra, or all three.

A 4% failure rate may sound small, but at a million requests a day, that’s forty thousand requests with poor output every day.

Most of those users will never file a ticket. They’ll just stop trusting the system, and likely stop using it. Or, they could assume it is correct, and make costly decisions based on this misinformation.

Silent errors at scale erode confidence in ways that are hard to measure and even harder to repair.

When something finally does surface as an incident, lack of observability means you can’t reconstruct what happened.

Every incident analysis becomes speculation. Every fix is a guess. Your troubleshooting, remediation process, and fixes all slow down or break entirely.

With strong observability, that process is much more efficient. You see problems in near real time and can trace them back to a specific model version, model plan, process, tool, or deployment.

Let’s recap what we’ve learned:

Monitoring tells you what’s going on in your system.

Observability uses logs, metrics, and traces to help you understand what went wrong, when and where it occurred, and why.

Because agent failures are novel, non-deterministic, and often silent, monitoring alone isn’t enough.

Observability allows you to see problems and trace them, so you're no longer working with a black box.

Great job! We’ve explored what observability is and why it matters. Next, we’ll go deeper into what makes agent failures so difficult to identify, and how those failures can quietly compound across multi-step workflows.