Kubernetes probes: health checks that do not cause outages

Reliable Kubernetes probes are three different control contracts, not three copies of one health endpoint. A `startupProbe` decides whether initialization has completed, a `readinessProbe` decides whether a Pod should receive Service traffic, and a `livenessProbe` decides whether restarting the container is more likely to recover it than leaving it running. Production reliability comes from keeping those questions separate, deriving timing from measured startup and recovery distributions, making probe handlers cheap and local, and connecting readiness to rollout and capacity policy. This matters for APIs, workers, gateways and stateful services where a false failure can amplify a small dependency incident into a fleet-wide restart storm.

The official Kubernetes probe guide documents the mechanics: readiness failure removes a Pod from Kubernetes Service traffic, liveness failure restarts the container, and a startup probe delays liveness and readiness until startup succeeds. Those are platform facts. The endpoint boundaries, timing method, dependency policy and rollout gates below are engineering guidance for turning those mechanics into a dependable production contract.

Start with the recovery action

Before writing an endpoint, ask what Kubernetes will do after failure. If the desired response is “wait; initialization is still progressing,” the signal belongs to startup. If it is “stop routing new requests but keep the process alive for recovery,” it belongs to readiness. If it is “this process cannot make progress and a clean restart is the safest local repair,” it belongs to liveness. A signal that has no safe automated action should be an alert or metric, not a liveness condition.

This action-first rule prevents a common category error: treating every dependency problem as evidence that the process is dead. A database outage, rate-limited upstream or unavailable cache may make some requests fail, but restarting every application replica does not repair the dependency. It destroys warm caches, creates reconnection bursts and removes otherwise useful capacity. Liveness should prove local forward progress, not the health of the entire internet.

Give each probe one narrow question

A startup endpoint should answer whether one-time initialization has reached the point where normal health evaluation is meaningful. Examples include loading required configuration, replaying a bounded journal, warming an essential in-process index or starting the request loop. It should not wait for optional background work that can continue after the application begins serving.

A readiness endpoint should answer whether this replica can accept new work now. It can include local admission signals such as queue saturation, an exhausted worker pool, a circuit breaker that makes every supported operation impossible, or a configuration generation that has not been applied. It should fail before overload becomes collapse, but only if removing the Pod from traffic leaves enough ready capacity elsewhere.

A liveness endpoint should answer whether the process can recover without restart. Useful checks are local: event-loop progress, a required internal worker heartbeat, an unrecoverable deadlock detector or corruption of in-memory state. Return success while the process is degraded but still capable of recovery. A restart is destructive state transition, so require strong evidence.

Do not put shared dependencies in liveness

Calling the primary database, an external payment API, DNS or a shared message broker from liveness couples every replica restart policy to the same dependency. During a shared outage, all Pods can fail together, restart together and reconnect together. This synchronizes load precisely when the dependency has the least capacity. The result is a cascading failure created by the repair mechanism.

Readiness requires a more nuanced decision. If the application truly cannot serve any correct request without a dependency, removing the replica from traffic may be appropriate. If it can serve cached reads, static routes, health-safe fallbacks or requests unrelated to the failed dependency, keep the Pod ready and expose degradation at the route or capability level. For mixed APIs, one global readiness bit is often too coarse; use application admission, circuit breakers and endpoint-specific error handling.

Never let the probe call the same overloaded worker pool it is meant to diagnose. Reserve a small execution path, avoid unbounded locks, and bound every internal operation. The handler should not write data, refresh tokens, perform migrations, warm caches or trigger recovery jobs. Health observation must not mutate business state.

Size startup from measurements, not hope

Kubernetes recommends setting the startup window using `failureThreshold * periodSeconds` so it covers the worst expected startup time. That is a budget, not a random delay. Measure cold starts by image version, node class, dataset size and configuration mode. Choose a percentile and safety margin that reflect the service objective, then alert when startup approaches the budget instead of silently increasing it forever.

For example, if legitimate cold starts are below 110 seconds at the chosen percentile and the team allows a 30-second margin, a 10-second period with `failureThreshold: 14` creates a 140-second window. Once startup succeeds, liveness can use a much shorter failure-detection budget. If startup never succeeds, Kubernetes restarts according to the Pod restart policy. Record the reason initialization is still incomplete so operators can distinguish slow progress from a permanent configuration error.

An illustrative configuration is:

startupProbe:
  httpGet: { path: /startupz, port: health }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 14
livenessProbe:
  httpGet: { path: /livez, port: health }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: health }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
  successThreshold: 2

These values are examples, not universal defaults. The right budget depends on scheduler delay, CPU limits, runtime pauses, request concurrency, dependency behavior and the application recovery model.

Understand the timing math

The Kubernetes Pod API documents a default `timeoutSeconds` of one second, `periodSeconds` of ten seconds, `failureThreshold` of three and `successThreshold` of one. Liveness and startup require `successThreshold: 1`; readiness can require multiple consecutive successes to reduce flapping after a failure. Explicitly set every timing field so an image, runtime or cluster change does not expose an accidental assumption.

The approximate detection time is not only `periodSeconds * failureThreshold`. Add the time until the next probe, each timeout, kubelet processing, termination grace, container stop and restart backoff. For readiness, also include EndpointSlice propagation and load-balancer behavior outside Kubernetes. Measure client-observed removal and re-entry rather than treating the YAML numbers as the full outage budget.

Set `timeoutSeconds` above a healthy handler’s high percentile but below the duration at which waiting has no value. A one-second default can be too strict for a process experiencing a short stop-the-world pause, while a very large timeout delays detection and consumes kubelet probe work. The best fix for a slow health endpoint is usually to make it cheaper, not to give it a network-scale timeout.

Prevent readiness flapping from becoming overload

Readiness is a traffic-control loop. If Pods declare themselves unready at the first sign of load, traffic concentrates on the remaining Pods, which then also become unready. Capacity disappears in steps and the rollout or autoscaler receives a misleading signal. Design admission with hysteresis: fail after sustained inability to accept work, and recover after multiple successful checks or a lower recovery threshold.

Keep enough headroom that removing one Pod does not overload the rest. Test readiness together with replica count, `maxUnavailable`, topology spread, autoscaling lag and disruption budgets. A probe that is safe with 30 replicas may be dangerous with two. Readiness should shed a bad replica, not implement global load shedding by accident.

For a queue consumer that does not receive Service traffic, readiness may still matter to controllers or operational dashboards, but it is not a substitute for pausing consumption. Use an application-level lease, broker flow control or graceful shutdown mechanism for work ownership. Match the signal to the actual traffic path.

Connect probes to Deployment progress

A Pod becoming ready once is not the same as a stable release. Kubernetes Deployments expose `minReadySeconds`, the period a new Pod must stay ready without a container crash before it is considered available. The default is zero. Set it to cover the early-failure window you care about so a briefly green process does not advance the rollout immediately.

Deployments also expose `progressDeadlineSeconds`, which defaults to 600 seconds and must be greater than `minReadySeconds`. Kubernetes reports `ProgressDeadlineExceeded` when progress stalls, but the controller only reports the condition; it does not automatically roll back. CI/CD or another higher-level controller must observe that condition and execute the release policy.

Build a rollout gate from stable availability, error rate, latency and restart evidence. A successful readiness response is necessary but not sufficient. Keep the previous ReplicaSet deployable, define who or what initiates rollback, and test the path before a risky release.

Use readiness gates only for external lifecycle facts

Kubernetes supports Pod `readinessGates` for additional conditions in Pod status. A Pod using them is ready only when all containers are ready and every specified condition is true; a missing custom condition defaults to false. This is useful when an operator or controller owns a fact that an in-container HTTP handler cannot know, such as external load-balancer registration or completion of a platform attachment.

Do not use readiness gates as a generic remote-call mechanism. The controller that writes the condition needs ownership, RBAC, retries, staleness handling and observability. Define who clears the condition and what happens if that controller is unavailable. Otherwise the gate becomes a new single point that can hold an entire rollout unready.

Choose HTTP, TCP, exec or gRPC deliberately

HTTP probes communicate an application-level state and are usually easiest to version and observe. TCP proves only that a socket accepts connections; a wedged server can still accept. Exec can inspect local state but adds process-creation overhead and is easy to make dependent on a shell or utility missing from a minimal image. Native gRPC probes use the standard health protocol but have specific port and timeout behavior that should be tested on the cluster version in use.

Keep the protocol aligned with what you need to know. A TCP liveness check may be enough for a simple proxy whose only useful state is the listener. A complex API should expose explicit health semantics. Do not make probe success depend on authentication middleware, tenant lookup or rate limits intended for user traffic. Protect the health port with network policy or node-local reachability rather than a fragile login flow.

Observe the control loop

Export handler latency and outcome by probe type, but avoid high-cardinality labels. Track Pod readiness transitions, restart count, last termination reason, startup duration, time to stable availability and rollout progress. Correlate failures with CPU throttling, memory pressure, runtime pauses, node events and dependency incidents. Alert on patterns: simultaneous liveness failures across replicas, repeated readiness oscillation, startup near its deadline and releases that exceed progress deadlines.

Log enough reason detail to diagnose the condition, but return only a minimal response to the kubelet. Never expose credentials, connection strings, stack traces, dependency topology or customer data from an unauthenticated health endpoint. Treat the internal diagnostics page and the probe contract as different interfaces.

Test failure injection before production: freeze the request loop, exhaust the worker pool, block the database, delay DNS, make the handler exceed timeout, corrupt required local state and terminate during a rollout. Verify the exact action each probe causes, the client-visible traffic behavior and whether the fleet recovers without synchronized restarts.

Decision guide

Use startup when the process has a legitimate initialization phase longer or more variable than the desired liveness detection time. Use readiness when the replica must temporarily leave new-request routing but should remain alive. Use liveness only when restart is a proven local recovery. Use a readiness gate when an external controller owns a required platform fact. Use metrics and alerts when humans or a higher-level system must decide. Use route-level degradation when only part of the product depends on the failed component.

Avoid liveness for databases or remote APIs, readiness that checks every dependency recursively, probe handlers behind the busy request pool, one shared endpoint with identical semantics, unexplained default timeouts, and automatic rollback assumptions. Do not copy probe timings from another service; derive them from this service’s measured behavior and capacity model.

Limits

Probes do not prove business correctness. A process can be live and ready while returning wrong data, violating authorization or dropping messages. They also do not replace synthetic transactions, SLO monitoring, tracing, dependency health, graceful shutdown or release verification. Kubernetes observes one replica from the node; only end-to-end evidence can show that the product works for users.

Aggressive probes trade faster local repair for a higher false-positive rate. Conservative probes reduce false restarts but extend exposure to a dead process. There is no universal interval. Choose the trade-off from error budgets, redundancy, recovery cost and measured distributions, then revisit it when traffic or runtime changes.

Executive checklist

Define one recovery action per signal. Keep liveness local and restart-worthy. Keep readiness capability-focused and capacity-aware. Size startup from cold-start evidence. Set timeouts, periods and thresholds explicitly. Pair readiness with replica headroom, rollout settings and observability. Use `minReadySeconds` for stability and act on `ProgressDeadlineExceeded` outside the Deployment controller. Protect health details, rehearse dependency and overload failures, and verify client-visible behavior. If the endpoint cannot explain why Kubernetes should take its next action, the probe contract is not ready.

Prepared by: Noor Yasser