Grafana Dashboards for Operators, Not Executives

A Grafana dashboard built for operators answers one question fast: is this service healthy, and if not, where is the problem? Most dashboards fail that test. They are walls of pretty panels built to impress in a review, useless at 3 a.m. when someone is paged and has thirty seconds to find the fault. Designing for the operator under pressure, not the executive in a meeting, is the whole discipline.

The tell is the panel count. A dashboard with fifty graphs is decoration; nobody scans fifty graphs while production is down. A dashboard with a tight golden-signals row and a clear path to drill deeper is a tool.

Why dashboard design matters

A dashboard is read at two very different moments: calmly, during a review, and frantically, during an incident. Most dashboards are unconsciously designed for the calm moment, which is exactly the wrong audience, because the calm moment is not when the dashboard has to earn its keep.

The incident is when it matters. Under pressure, an operator needs to go from “something is wrong” to “the database is saturated” in seconds. A dashboard that buries that signal in a grid of low-value panels actively slows the response. This post is part of the Observability series and pairs with Jaeger Tracing for Cross-Service Debugging.

What are the four golden signals?

The four golden signals are latency, traffic, errors, and saturation. Latency is how long requests take (watch the tail, not the average), traffic is request volume, errors is the failure rate, and saturation is how full your constrained resources are. Together they summarize whether a service is healthy, which makes them the correct top row of any operator dashboard.

They come from Google’s SRE practice, and they are popular because they generalize: almost any service’s health can be read from those four numbers. If those four look good, the service is almost certainly fine; if one is bad, it points you at the category of problem.

Signal	Question it answers	Watch for
Latency	How long do requests take?	p99/p99.9 tail, not the mean
Traffic	How much demand?	Sudden spikes or drops
Errors	What fraction fail?	Rate and sudden changes
Saturation	How full are resources?	The most constrained resource first

What makes a good Grafana dashboard?

A good Grafana dashboard answers a specific question fast and contains nothing that does not serve that question. It leads with the golden signals, uses a consistent time range across panels so they are comparable, labels axes and units clearly, and ruthlessly omits anything that does not help someone act. Clarity under pressure is the only metric that matters.

The design principles that follow from “answer fast”:

• One dashboard, one job. A service-health dashboard is not also a capacity-planning dashboard. Separate audiences, separate boards.
• Most important panels top-left. Eyes start there; put the golden signals where they land first.
• Consistent time windows. If panels show different ranges, you cannot correlate a spike across them.
• Units and thresholds visible. A number without a unit or a “good vs bad” threshold makes the operator do math under stress.

How many panels should a dashboard have?

As few as answer the dashboard’s question, usually a handful, not dozens. A fifty-panel dashboard cannot be scanned during an incident, so the detail is effectively invisible when it is needed. Lead with the golden-signal panels and push everything else into linked drill-down dashboards that you open only once the top-level board points you there.

The structure that scales is a hierarchy: a top-level health dashboard with the golden signals and a handful of key indicators, linking down to detailed dashboards per subsystem. The operator starts at the top, sees which signal is bad, and clicks into the relevant detail. Depth on demand, not depth by default.

Should dashboards and alerts use the same metrics?

Yes, always. The metric that pages you must be front and center on the dashboard you open when that page fires. If the alert references one number and the dashboard shows different ones, the operator wastes precious time reconciling them. Alerts and the operator dashboard should tell a single, consistent story.

This alignment is what makes a page actionable. The ideal flow is: the alert fires on a golden signal, the operator opens the linked dashboard, and the very first panel shows the metric that fired, in context, with its threshold marked. There is no translation step. The dashboard and the alert are two views of the same truth, which is also why dashboards should be the thing you tune alongside your SLOs rather than an afterthought.

How do you organize dashboards across many services?

Use a hierarchy and a template, not one bespoke dashboard per service built by hand. At scale, the winning pattern is a single standardized service-health dashboard, parameterized by a variable that selects the service, so every service is viewed through the same consistent lens. Above it sits a fleet-level overview; below it sit subsystem drill-downs.

This solves the problem that kills observability at scale: dashboard sprawl. When every team hand-builds its own dashboards, you end up with hundreds of inconsistent boards, and an on-call engineer paged for an unfamiliar service has to learn a new layout under pressure. A templated dashboard means the golden signals are always in the same place, whatever service you are looking at, so muscle memory works across the whole fleet.

The structure that scales is three tiers: a fleet overview showing the health of all services at a glance, a per-service health dashboard generated from one template, and detailed subsystem dashboards linked from each. Operators move down the tiers as they narrow the problem. Building dashboards as code (versioned, templated, reviewed) rather than clicking them together by hand is what keeps that structure consistent as the number of services grows.

Why do executive and operator dashboards need to be separate?

The instinct to build one dashboard that serves everyone is reasonable and produces something that serves nobody. The two audiences want opposite things, and the conflict is structural rather than a matter of taste.

An operator is looking at a dashboard because something may be wrong right now. They need high resolution over a short window, percentiles rather than averages, breakdowns by dimension, and enough detail to localise a fault. Their session lasts minutes and ends in an action.

An executive is looking to understand a trend. They need low resolution over a long window, aggregates rather than distributions, business framing rather than technical, and a small number of numbers with clear direction. Their session ends in a decision about priorities.

Trying to merge them produces the familiar bad outcome: a dashboard with a business KPI at the top and forty technical panels below it, where the executive cannot find the number they came for and the operator scrolls past a revenue chart during an outage. Neither audience trusts it, so both build their own anyway — usually in a hurry, usually badly.

Build them separately and accept the small duplication. They can share underlying queries; what differs is aggregation, time range, and framing. The operator dashboard optimises for time-to-diagnosis. The executive dashboard optimises for time-to-understanding. Those are different optimisation targets and they produce different layouts.

One practical consequence worth stating: if leadership regularly watches an operator dashboard, the operator dashboard will get worse. Panels get added because they look reassuring, thresholds get softened because red is uncomfortable in a review, and the instrument slowly becomes a presentation. Giving leadership a dashboard designed for them is what protects the one on-call depends on.

How do you keep dashboards useful over time?

Dashboards decay faster than almost anything else in a system, because nothing breaks when they go stale. Code that rots fails a test; a dashboard that rots just quietly stops helping, and nobody notices until an incident.

The decay is predictable. Metrics get renamed and panels go blank. Services are split and a dashboard covers half a system. Someone adds a panel during an incident and never removes it. Thresholds set for last year’s traffic no longer mean anything. After a year, the dashboard is a historical record of past incidents rather than an instrument for the current system.

Four practices arrest it:

Define dashboards as code. JSON in a repository, reviewed and deployed like anything else. Click-built dashboards have no history, no review, and no way to answer “who changed this and why.” As-code also makes the next practice possible.

Template rather than duplicate. Forty services should not have forty hand-built dashboards. One parameterised template with the service as a variable means an improvement to the template improves every service at once, and a new service gets a good dashboard for free. Bespoke dashboards are for the handful of systems with genuinely unusual shapes.

Review after every incident. The postmortem should answer one dashboard question: what did we need and not have? That is the highest-quality signal available for what to add, because it comes from a real moment of need rather than from someone imagining one. It is also the right time to delete a panel that nobody looked at.

Delete aggressively. A panel showing “No data” should be removed the day it is noticed. Stale panels are worse than absent ones because they train responders to ignore empty panels, and an empty panel sometimes means the exporter is down.

The health check for a dashboard is simple and worth running occasionally: hand it to an engineer from another team and ask them to tell you whether the service is healthy. If they cannot, the dashboard works only for its author — and the author is not always the person on call at 3 a.m.

A cheap habit that compounds: put a one-line description at the top of every dashboard stating what the service does and what “healthy” looks like in plain language. It costs a sentence and removes the single largest obstacle for a responder who has never seen this service before, which during a cross-team incident is most of them. The same applies per panel — a short subtitle explaining what the metric means beats a title that only repeats the query name. Treat those descriptions as part of the dashboard’s contract with its readers, and review them whenever the underlying service changes shape, because a confidently wrong description is worse than none at all.

The broader principle behind every practice in this section: a dashboard is a user interface for a person under stress, and it should be judged the way any interface is judged — by whether a first-time user succeeds at the task. Almost every rule here follows from taking that framing seriously rather than treating the dashboard as a place to display metrics you happen to collect.

How should a dashboard be laid out for an incident?

Panel choice gets discussed; layout rarely does. Yet layout is what determines whether a responder reaches a conclusion in thirty seconds or scrolls for five minutes.

The organising principle is top-down, narrowing: the top of the dashboard answers “is something wrong,” the middle answers “where,” and the bottom answers “why.” A responder should be able to stop reading as soon as they have their answer.

• Row 1 — is this real? Two or three panels, no more: request rate, error rate, and latency for the service as a whole, with the SLO threshold drawn on. This row alone should settle whether to keep going or go back to bed.
• Row 2 — where is it? The same signals broken down by the dimension that localises problems in your system: endpoint, dependency, region, or version. This is the row that turns “the service is slow” into “the service is slow on one endpoint.”
• Row 3 — why? Saturation and resource panels: connection pool utilisation, queue depth, CPU, memory, GC. These are causes, and they are only useful once you know what you are explaining.
• Row 4 — context. Deploy markers, config changes, dependency status. Frequently the fastest possible answer to “why,” and frequently at the bottom because it is a confirmation rather than a search.

Two layout rules that matter more than they sound. Time ranges must be linked across panels, so that comparing two graphs never involves squinting at differing axes. And deploy annotations belong on every panel, not on their own. “It started at 14:02” plus a deploy marker at 14:01 is an entire investigation completed in one glance, and it only works if the marker sits on the graph you were already looking at.

What are the most common dashboard anti-patterns?

Most bad dashboards are bad in the same handful of ways.

Averages. A mean latency hides everything that matters. If 1% of requests take ten seconds and 99% take fifty milliseconds, the average looks fine and 1% of your users are furious. Show p50, p95, and p99 — the shape of the distribution is the information.

Every metric, because it was available. A dashboard with sixty panels is a data dump, not an instrument. It is built by adding a panel each time someone wants something and never removing one. The discipline is subtractive: a panel earns its place by having changed a decision at least once.

Panels with no threshold. A graph of CPU with no line marking “too high” requires the reader to already know what normal is. Draw the threshold, the SLO, or the capacity limit directly on the panel so the answer is visual rather than remembered.

Dashboards that only make sense to their author. The tell is unlabelled panels titled with a raw metric name. If a responder from another team cannot use it during an incident, its value evaporates exactly when it is needed, because the author is not always the one on call.

Stale panels pointing at deleted metrics. Panels showing “No data” quietly train responders to ignore empty panels, which is precisely the wrong instinct when an empty panel means the exporter is down. Delete them.

Alerting off one dashboard query and paging off another. When the alert and the dashboard disagree about what “error rate” means, every incident starts with reconciling the two. Define the query once and reference it in both.

An operator-dashboard checklist

Before you call a dashboard done:

• It answers one clear question (service health, or one subsystem), not five.
• The golden signals (latency tail, traffic, errors, saturation) are the top row.
• Every panel would change what an operator does during an incident; the rest are cut or moved to drill-downs.
• Panels share a consistent time range and have clear units and thresholds.
• The metric behind each alert is visible on the dashboard the alert links to.
• A new on-call engineer can read service health from it in under thirty seconds.
• Detail lives in linked drill-down dashboards, not crammed onto the top-level view.

What should you not put on an operator dashboard?

Leave off anything that does not change what an operator does in the next few minutes: vanity metrics, raw counters with no rate or threshold, business KPIs that belong on an executive dashboard, and any panel that is there to look comprehensive rather than to be acted on. If a graph cannot trigger a decision during an incident, it is clutter on the board that matters most.

The specific offenders show up again and again. Cumulative totals (total requests ever) instead of rates tell an operator nothing about right now. Panels with no unit or no good-versus-bad threshold force mental math under stress. Business metrics like signups or revenue belong on a different dashboard for a different audience; mixing them into a health view dilutes the signal an on-call engineer is scanning for. And the worst offender is the panel added “just in case,” which is never the panel anyone looks at during a real incident.

The discipline is subtractive. Every panel should justify its place by answering “what would an operator do differently because of this.” Anything that cannot answer that question moves to a drill-down or comes off the board entirely. A lean dashboard is not a less-thorough dashboard; it is one that respects the thirty seconds an operator actually has.

What I’d do differently

The dashboards I regret are the impressive ones: dense grids of panels that looked authoritative in a review and that nobody could actually use when the system was on fire. They optimized for the wrong audience, the observer who wants to feel informed, instead of the operator who needs to act.

If I were building dashboards again, I would design every one for the worst moment: a tired engineer, paged at 3 a.m., who needs the answer in seconds. That constraint forces the right choices, golden signals on top, ruthless panel pruning, alerts and dashboards in lockstep, and detail one click away. A dashboard that serves that moment serves every other moment too. The traces that complement these dashboards during debugging are covered in Jaeger Tracing for Cross-Service Debugging.

Sources

• Google SRE Book, Monitoring Distributed Systems (the four golden signals): sre.google/sre-book/monitoring-distributed-systems
• Grafana, Best practices for dashboard management: grafana.com/docs/grafana/latest/dashboards/build-dashboards/best-practices
• Grafana, Common observability strategies: grafana.com/docs/grafana/latest/fundamentals