Everything is green. So why is production broken?

There is a particular kind of production incident that is harder to explain than a server going down. Everything looks normal. CPU is within range. Memory is stable. Kubernetes says the Pods are healthy. The database is available. No major alerts are firing… And customers are still complaining that the application is slow, unreliable or simply not working the way it should.

This is where the difference between system health and user experience becomes important.

A production system can be technically operational while failing to deliver the experience users expect from it.

Google's reliability guidance makes the distinction explicitly: infrastructure metrics such as CPU utilization or server errors do not necessarily tell you whether users can successfully use the service. Reliability should be measured through outcomes such as request success, latency and error rates from the user's perspective.

That sounds obvious.

In practice, it changes the way teams operate.

Green dashboards can hide real problems

Imagine an e-commerce API with a 99.99% availability target.

The infrastructure dashboard is green. The service is technically available.

But one important checkout endpoint has become significantly slower because of a downstream dependency.

Most requests eventually succeed. The overall error rate remains low. nothing crosses the alert threshold.

From the infrastructure's perspective, everything is fine.

From the customer's perspective, checkout is broken.

This is one of the reasons SRE practices emphasize Service Level Indicators and Service Level Objectives. Google's SRE guidance describes SLOs as a way to create an objective, user-focused way of discussing reliability, with error budgets providing a framework for balancing reliability against other engineering work.

The important part is the word objective.

“Production feels unstable” is difficult to operationalize.

“99.9% of checkout requests must complete successfully within 800 milliseconds.” gives a team something it can measure, investigate and improve.

Good reliability metrics also need to reflect the service's actual purpose.

A background processing system might care about job completion time.

An API might care about latency and successful requests.

A consumer application might care about login success, page load time or transaction completion.

The right SLI depends on what users are actually trying to accomplish.

This is also where observability becomes more useful than a collection of dashboards. AWS recommends aligning observability with business and customer outcomes, including application telemetry, user experience telemetry and dependency telemetry.

The goal is not to collect everything.

It is to know which signals tell you when the system is failing users.

Reliability is a system property

Once teams start measuring user-facing outcomes, another realization usually follows: reliability rarely belongs to one component.

A checkout can depend on the frontend, API, authentication service, payment provider, database, queue and several internal services.

Every component can be healthy on its own while the complete user journey is unhealthy.

This is why distributed systems require a broader view of reliability.

AWS describes resiliency as the ability of a workload to recover from infrastructure or service disruptions and continue operating through issues such as transient network failures or changes in demand. Its reliability guidance also emphasizes testing failure scenarios rather than assuming that a system will behave correctly simply because each component appears healthy.

The practical implication is important: reliability needs to be designed into the relationships between components, not only into the components themselves.

Timeouts, retries, fallbacks, dependency isolation, recovery procedures… They all matter.

And the team needs to know what happens when one of those mechanisms fails.

This is where incident response becomes part of engineering rather than an emergency activity that happens after engineering is finished.

A useful incident process asks:

• What happened?
• Who was affected?
• What signal should have told us earlier?
• Which dependency failed?
• How did the system behave under stress?
• What allowed the incident to spread?
• What would make recovery faster next time?

The objective of a postmortem should be learning, not assigning blame.

Google's SRE practices explicitly connect reliability with postmortems, progressive rollouts, isolation strategies and continuous improvement.

Reliability is what users experience

The most useful reliability dashboard may therefore have surprisingly little to do with the infrastructure dashboard. A service can be running at 40% CPU utilization and still be failing users; a cluster can have healthy Pods while requests are timing out, and a database can be fully available while the application built on top of it is effectively unusable. The closer reliability measurements are to the actual user journey, the more meaningful they become.

That does not make infrastructure metrics irrelevant. CPU, memory, disk, network and container health remain valuable signals for understanding what is happening inside a system. They simply answer a different question. Infrastructure metrics tell you how the system is behaving, while user-facing reliability tells you whether the system is delivering what it promised. Good operations need both perspectives, and when they disagree, the user-facing signal deserves closer attention.

For teams building cloud-native systems, reliability is ultimately less about keeping every dashboard green and more about making sure the system continues to do what people depend on it to do. That is a much harder standard, but also a much more useful one. To go deeper into the signals that help teams understand system behavior, see Observability in DevOps: Metrics, Logs and Traces.