Readiness Probes That Don't Lie

Most Kubernetes readiness probes lie. They return 200 because the process is running, not because the service can actually serve a request, so Kubernetes confidently routes traffic to a pod that immediately throws errors. A readiness probe that does not reflect real readiness is worse than no probe at all, because it makes failure look like health.

A truthful readiness probe answers one question: can this specific pod serve a real request right now? Not “did the process start,” not “is the binary alive,” but “would a request succeed if I sent one.” Writing the probe to answer that, and no more, is the whole skill.

Why readiness probes matter

In Kubernetes, the readiness probe is the gate between your pod and live traffic. Pass it and the pod joins the Service endpoints and starts receiving requests. Fail it and the pod is pulled from rotation, no traffic, no restart.

That makes the readiness probe a load-bearing piece of your reliability, and a deceptively easy one to get wrong. The default instinct, a handler that returns 200, technically “works” and quietly defeats the entire mechanism. This post is part of the Kubernetes operations series.

What is the difference between a readiness and liveness probe?

A readiness probe decides whether a pod receives traffic: failing it removes the pod from the Service endpoints but never restarts it. A liveness probe decides whether a pod is restarted: failing it kills and recreates the container. Readiness asks “can I serve right now,” liveness asks “am I broken beyond recovery.”

Conflating the two is one of the most common and damaging Kubernetes mistakes. If you put dependency checks in a liveness probe, a transient database blip will restart every pod instead of simply pausing traffic, turning a brief degradation into a self-inflicted restart storm.

	Readiness probe	Liveness probe
Controls	Whether the pod gets traffic	Whether the pod is restarted
On failure	Removed from Service endpoints	Container killed and recreated
Answers	”Can I serve right now?"	"Am I unrecoverably broken?”
Dependency checks	Sometimes, carefully	Almost never
Failure during a DB blip	Pauses traffic (good)	Restart storm (bad)

There is also the startup probe, which protects slow-starting containers by holding off liveness and readiness checks until the app has finished booting. Use it for anything with a long warmup (a JVM service, a large model load) so a slow start is not mistaken for a failure.

The startup probe exists to resolve a genuine conflict. Steady-state liveness settings should be tight enough to catch a wedged process reasonably quickly, but boot can legitimately take far longer than any healthy steady-state check. Without a startup probe you must loosen liveness enough to cover the slowest possible boot, and then live with those loose settings forever. The startup probe lets you keep tight steady-state values while granting a generous one-time budget: set a short periodSeconds with a large failureThreshold, so the total allowance is long but the transition to serving happens as soon as the app is genuinely up. Prefer it to a large initialDelaySeconds, which is a fixed guess that is simultaneously too long on fast boots and too short on slow ones.

Why do readiness probes give false positives?

Because the probe checks that the process is up, not that it can serve. The classic offender is a /healthz handler that returns 200 unconditionally. It passes while the database connection pool is empty, the cache is cold, or a critical downstream is unreachable, so Kubernetes sends real traffic straight into errors.

A truthful probe verifies the things the pod genuinely needs to serve. If the service cannot function without its database connection pool being initialized, the readiness probe should reflect that the pool is ready, not merely that the HTTP server is listening.

// LYING readiness probe: passes as long as the process runs
http.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK) // says "ready" even with no DB, cold cache
})

// TRUTHFUL readiness probe: reflects real ability to serve
http.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
    if !app.DBPoolReady() || !app.CacheWarm() {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Should a readiness probe check dependencies like the database?

Check the critical dependencies the pod truly cannot serve without, but do it carefully. The danger is a shared dependency: if every pod’s readiness probe checks the same database, one database blip fails all of them at once, removes the entire service from rotation, and converts a partial degradation into a full outage.

The balance is to distinguish what is local from what is shared. Checking your own initialized connection pool is safe, because it is per-pod. Checking that the shared database is reachable on every probe couples every pod’s fate to that dependency, which can be exactly the wrong behavior under stress.

A more resilient pattern is to keep serving in a degraded mode when a non-critical dependency is down, rather than failing readiness. Fail readiness only for dependencies without which a request genuinely cannot succeed, and even then consider whether removing every pod at once is better or worse than serving degraded responses.

The decision, as a rule you can apply

The useful reframing is to stop asking “is this dependency important?” and start asking “if this check fails on every pod simultaneously, is removing the entire service the correct response?”

Almost always, the answer is no — and that single question resolves most cases:

• A shared database is unreachable. Removing every pod means callers get connection refused instead of a 503 with a Retry-After. You have destroyed your own ability to shed load gracefully, degrade partially, or serve cached reads. Worse, you have removed the pods that would have recovered instantly when the database came back, and now they must pass probes again before traffic returns, extending the outage past the actual fault.
• A per-pod resource is not ready — your own connection pool, a loaded local cache, a warmed JIT. Failing readiness is exactly right, because the condition is genuinely local and other pods are unaffected.

The distinction is local versus shared, not important versus unimportant. Check things whose failure is specific to this pod. For shared dependencies, prefer serving an honest error over vanishing from the load balancer.

There is a second-order effect worth naming: readiness failures caused by a shared dependency are correlated, and correlated failure is what turns degradation into an outage. Independent failures are absorbed by capacity headroom; correlated ones are not, because every replica fails at the same instant for the same reason. Any check that can fail identically across all replicas at once deserves a much higher bar than one that cannot.

If you do check a shared dependency, at least make it stateful rather than instantaneous: require the dependency to have been failing for some sustained window before flipping readiness, and use a failureThreshold high enough that a single slow query cannot depopulate the service. A probe that reacts to one bad response is a probe that amplifies every blip into an incident.

Can a bad liveness probe cause an outage?

Yes, and it is a classic self-inflicted one. A liveness probe that is too aggressive, or that checks external dependencies, restarts healthy pods during load spikes or transient blips. The restarts shed capacity exactly when you need it most, which increases load on the survivors, which fails more probes: a restart storm.

Liveness should detect only states a restart can actually fix: a deadlocked process, an unrecoverable internal error, a wedged event loop. It should never fail because a dependency is slow or because the pod is briefly busy. When in doubt, make the liveness probe more lenient and the readiness probe more precise.

The sharpest test for whether something belongs in a liveness probe is a single question: would restarting the container fix this? A deadlock, yes. A leaked resource that only a fresh process reclaims, yes. A slow database, no — the new container will be just as unable to reach it, and you have now paid a cold start on top of the original problem.

Applied honestly, that test eliminates most liveness probes entirely. A well-written service that crashes on unrecoverable errors and lets Kubernetes restart it via its normal restart policy often needs no liveness probe at all. This is a legitimate and underused configuration: omitting the liveness probe is frequently safer than having one, because a probe that never fires adds nothing, while a probe that misfires under load takes capacity offline precisely when it is scarcest.

The restart-storm dynamic is worth spelling out because it is self-reinforcing. Load rises, probe latency rises with it, some pods miss the liveness timeout, those pods restart, their traffic redistributes onto the remaining pods, which pushes those pods’ probe latency higher, which restarts more of them. Nothing was wrong with the application. The probe manufactured the outage, and it accelerates as it goes.

That is also why liveness failureThreshold should be generous. Requiring five or six consecutive failures over a longer period means a genuine deadlock is still caught within a minute or two, while a transient load spike passes harmlessly. You lose almost nothing in detection time and remove the entire class of self-inflicted restart cascade.

How do you tune probe timing?

Set the timing so a healthy pod is never marked unhealthy and a truly broken one is caught quickly. The four knobs that matter are initialDelaySeconds (or better, a startup probe), periodSeconds, timeoutSeconds, and failureThreshold. The most common failure is a timeoutSeconds set so tight that a pod under load misses the deadline and gets pulled or restarted while it is actually fine.

A practical approach: use a startup probe to cover boot time instead of a long initialDelaySeconds, so slow starts do not force you to loosen the steady-state checks. Keep periodSeconds short enough to react quickly (a few seconds) but not so short that probes add meaningful load. Set timeoutSeconds generously relative to your real probe latency under load, because a probe that does a tiny bit of work can blow a 1-second timeout during a traffic spike. And set failureThreshold high enough that a single transient miss does not flap the pod out of rotation.

The asymmetry to remember: for readiness, err toward reacting fast so you stop sending traffic to a struggling pod. For liveness, err toward patience so you do not restart pods that are merely busy. Tuning both the same way is how a load spike turns into a restart storm.

A readiness probe checklist

Before you ship a probe, confirm:

• Readiness reflects real ability to serve, not just that the process started.
• Liveness detects only unrecoverable states; it does not check external dependencies.
• A startup probe guards any slow-booting container so warmup is not read as failure.
• Shared-dependency checks will not remove the entire service on a single blip.
• The pod degrades gracefully where it can, instead of failing readiness for every minor issue.
• Probe timeouts, periods, and failure thresholds are tuned (a too-tight timeout fails healthy pods under load).
• During a graceful shutdown, the pod fails readiness first so it drains traffic before terminating.

That last point matters for clean deploys: on shutdown, fail readiness immediately so Kubernetes stops sending new requests, then finish in-flight work, then exit. It is the difference between a deploy that drops zero requests and one that drops a burst on every rollout.

Why do requests still fail during a rolling deploy?

This is the probe problem that survives everything above, and it catches teams who did all the other work correctly. You fail readiness on shutdown, you handle SIGTERM, you drain in-flight work — and every deploy still produces a small burst of connection errors.

The cause is a race, and it is structural rather than a bug in your code. When a pod is deleted, two things happen in parallel, not in sequence:

1. The kubelet sends SIGTERM to your container.
2. The pod’s removal from the Service endpoints propagates outward — to the endpoints controller, then to every node’s kube-proxy or ingress, then into their forwarding rules.

Step 2 is eventually consistent and takes a nonzero amount of time across a real cluster. Step 1 is immediate. So if your application exits promptly on SIGTERM, it dies while some proxies still believe it is a valid backend, and they keep sending it requests. Those are your errors. Failing readiness at shutdown does not fix this by itself, because the readiness result has to traverse the same propagation path.

The fix is to make the pod deliberately outlive its own removal:

• Add a preStop hook that simply sleeps — commonly 5 to 15 seconds. The container keeps serving normally during this window while endpoint removal propagates. This is the single highest-value line of YAML in most deployments.
• Only after the hook completes does Kubernetes send SIGTERM. Then drain: stop accepting new work, finish in-flight requests, close connections, exit.
• Set terminationGracePeriodSeconds larger than preStop duration plus your longest realistic in-flight request, or the kubelet will SIGKILL you mid-drain and undo the whole exercise.

The counterintuitive part worth internalizing: shutting down fast is the bug. A pod that exits in 100 ms on SIGTERM is a pod that drops requests on every single rollout. Deliberate slowness at shutdown is what makes deploys invisible to users.

A useful check: run a steady load generator against the service and do a rolling restart. If you see any non-zero errors, you have this race. Zero errors under load during a rollout is an achievable bar, and it is the only real proof that the drain sequence is correct.

Which probe type should you use: httpGet, exec, or tcpSocket?

The mechanism matters more than teams expect, because two of the three have failure modes that only appear under load.

Type	How it works	Use it when	Watch out for
httpGet	Kubelet issues an HTTP request to a path and port	Almost always — it is the default choice for any HTTP service	The handler must be cheap; anything doing real work will time out under load
tcpSocket	Kubelet opens a TCP connection	Non-HTTP protocols where a connection implies health	Connecting proves a listener exists, nothing more — it is the weakest signal and lies readily
exec	Runs a command inside the container	Genuinely no network endpoint, or checking on-disk state	Forks a process every period, in every pod; under memory pressure this is a real cost and can itself cause failures
gRPC	Native gRPC health-checking protocol	gRPC services, in modern Kubernetes	Requires the service to implement the standard health service

The exec probe is the one that surprises people. A probe running every 5 seconds across 200 pods is 40 process spawns per second, continuously, forever. On a memory-constrained node it competes with the workload it is supposed to be protecting. Prefer httpGet unless you genuinely cannot expose an endpoint.

The tcpSocket probe deserves the same skepticism as an unconditional 200: a bound socket does not mean the application behind it can serve. It is a liveness signal at best, and it makes a poor readiness check for anything with dependencies.

What probe timing values should you actually use?

Concrete starting points, to be adjusted against measured probe latency rather than copied blindly:

Setting	Readiness	Liveness	Reasoning
periodSeconds	5	10–20	Readiness should react quickly; liveness has no reason to be eager
timeoutSeconds	2–3	3–5	Must exceed p99 probe latency under load, not idle latency
failureThreshold	2–3	5–6	Readiness can flap out cheaply; a liveness false positive costs a restart
successThreshold	1–2	1 (fixed)	Requiring two successes damps flapping back into rotation
initialDelaySeconds	0	0	Use a startup probe instead of guessing a delay

Two rules behind the numbers. First, liveness is always more lenient than readiness — the asymmetry is the entire point, because the cost of a wrong answer differs by orders of magnitude. Second, timeoutSeconds should be derived from a measurement: check your probe endpoint’s p99 latency while the service is at peak load, then set the timeout well above it. A probe endpoint that is fast at 3 a.m. and slow at peak is precisely how a healthy service tears itself apart during its busiest hour.

What I’d do differently

The mistake I have made is copying a /healthz returns-200 handler from a tutorial into a real service and calling it done. It passes review, it passes in staging with everything healthy, and it lies the first time a dependency hiccups in production, sending traffic into a wall.

If I were writing probes from scratch, I would design them around the question “can this pod serve a request right now,” wire readiness to drain traffic on shutdown, keep liveness lenient enough that it only ever fixes truly stuck processes, and load-test the probe behavior under a dependency failure before trusting it. Probes are reliability code, and they deserve the same rigor as the request path they guard. For the broader platform context these probes run in, see Kubernetes Namespace Strategy for SaaS Platforms.

Sources

• Kubernetes, Configure Liveness, Readiness and Startup Probes: kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes
• Kubernetes, Pod Lifecycle (container probes): kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes