Kubernetes graceful shutdown: drain traffic before processes exit

Zero-downtime shutdown in Kubernetes is a coordination problem, not a sleep command. A reliable design stops accepting new work, lets Service routing converge, drains in-flight requests, closes background consumers and exits before the Pod grace budget expires. The application must handle SIGTERM correctly; Kubernetes cannot finish business work on its behalf. A PodDisruptionBudget then limits simultaneous voluntary evictions, but it does not protect against node failure and does not control a Deployment rolling update. This matters for HTTP APIs, workers and stateful clients because a rollout that looks healthy at the controller level can still create reset connections, duplicated jobs or forced SIGKILL exits.

The Kubernetes Pod lifecycle documentation describes one shared termination clock: the kubelet runs a preStop hook when configured, then asks the runtime to send TERM, while the control plane updates Service endpoints. The default grace period is 30 seconds, and a container still running when the period expires is forcibly stopped. EndpointSlice records expose terminating, serving and ready conditions so routing implementations can distinguish an endpoint that is leaving from one that is available. Build your shutdown contract around those observable transitions rather than an assumed ordering between every load balancer, proxy and application process.

Treat termination as an application contract

Define shutdown as a state machine with explicit invariants:

1. The instance changes from accepting to draining exactly once. 2. New long-running work is rejected or redirected after draining starts. 3. Work already acknowledged is either completed, released with a lease, or made safe to retry. 4. The process exits successfully only after listeners, workers and durable writes reach a known state. 5. Every step has a deadline shorter than the remaining Kubernetes grace period.

The contract is workload-specific. An HTTP API may stop accepting connections and wait for active requests. A queue consumer must stop polling before it waits for handlers. A WebSocket service needs a reconnect instruction and a bounded close window. A leader must relinquish or expire its lease. Do not hide these different semantics behind one generic delay.

The shutdown path must be idempotent. Signals, hooks or library callbacks can overlap, and lifecycle hook delivery is intended to be at least once. A second shutdown request should observe the existing drain, not start another flush or acknowledge a job twice.

Understand the real termination timeline

When a Pod receives a deletion timestamp, several activities begin close together rather than as one perfectly serialized chain. The kubelet starts the local shutdown flow. If preStop exists, it runs before TERM is sent to the main process. At the same time, the control plane begins marking the Pod as terminating for workload and Service routing purposes.

The grace countdown begins before preStop. If a 60-second budget spends 15 seconds in the hook, the application does not receive a fresh 60 seconds after SIGTERM; it has roughly 45 seconds left. Kubernetes documents a small one-off two-second extension when preStop is still running as the period expires, but production logic should not depend on that emergency extension.

EndpointSlices retain terminating endpoints with ready set to false for compatibility. Most Service proxies ignore them for normal traffic, although an implementation may use an endpoint that is both serving and terminating if all endpoints are terminating. External load balancers, Ingress controllers, service meshes, DNS caches and keep-alive connections can converge at different speeds. Therefore, removing an endpoint from ordinary routing is necessary but not proof that no new packet can arrive.

Make the process own graceful shutdown

The application should register its signal handler before it reports readiness. A practical handler follows this order:

- flip an internal draining flag and make readiness fail; - stop accepting new connections or new queue messages; - tell long-lived clients to reconnect when the protocol supports it; - wait for in-flight work up to a bounded deadline; - release leases, flush buffered telemetry and close database clients; - exit with a code that distinguishes clean completion from timeout.

Process 1 inside the container must receive and forward the stop signal. Shell wrappers that do not use exec, detached child processes and custom supervisors often swallow or delay SIGTERM. Test the built image, not only the framework's local development server. If the container has native sidecars, Kubernetes delays stopping them until main containers exit and terminates sidecars in reverse declaration order; that is safer than encoding fragile ordering in long preStop scripts.

Readiness is a traffic admission signal, not a shutdown executor. It tells Kubernetes Services whether the Pod should receive ordinary traffic. Keep the readiness endpoint cheap and local enough to respond during dependency incidents. A shutdown handler can set it false immediately, but it must still close listeners and wait for work because existing connections do not disappear when readiness changes.

Use preStop as a short propagation buffer

preStop is useful when a process cannot observe deletion before SIGTERM or when routing components need a brief propagation window. Keep it short and deterministic. Its runtime is charged against terminationGracePeriodSeconds, and TERM is not sent until the hook completes. A 25-second sleep inside a 30-second budget can leave only a few seconds for the application to close correctly.

Prefer native application signal handling for business cleanup. A hook that calls an unauthenticated admin endpoint, depends on DNS, or runs a large shell program adds another failure path exactly when the system is already changing. If a hook saves state, make it idempotent and budget its worst-case latency. If its only job is to wait for routing convergence, record the chosen delay and validate it against observed endpoint and load-balancer propagation rather than copying an arbitrary value.

Calculate one explicit grace budget

Choose the Pod grace period from measured components:

grace >= preStop propagation buffer + longest accepted request or job slice + connection and client close time + durable flush or lease-release time + safety margin

Do not set it from the average request latency. Use a high percentile, a product deadline and the retry contract. If requests are allowed to run for two minutes, a 30-second grace period cannot make their shutdown graceful. Either shorten the accepted request deadline, support resumable work, transfer ownership, or allocate a larger budget.

Long grace periods have costs. Terminating Pods continue consuming CPU, memory and connections, and a Deployment can temporarily use more resources than replicas plus maxSurge because terminating Pods are not counted as available replicas. The answer is not always a larger number; it is often a bounded unit of work and a faster checkpoint.

Know what a PodDisruptionBudget protects

A PDB limits how many replicas of an application can be unavailable at once due to voluntary disruptions that use the Eviction API. kubectl drain respects it and retries rejected evictions. It is the right control for planned node maintenance, cluster upgrades and other cooperating eviction clients.

A PDB is not a shield against involuntary failure. Hardware loss, kernel failure and network partition can still remove capacity; those unavailable Pods count against the budget but were not prevented by it. Direct deletion can bypass the eviction policy. Deployment and StatefulSet rolling updates are configured by their own rollout strategy rather than being constrained by PDB, even though unavailable rollout Pods affect the budget observed by other evictions.

Choose the budget from service capacity:

| Workload | Useful starting point | Main risk | | --- | --- | --- | | Stateless API with four replicas | minAvailable: 3 or maxUnavailable: 1 | remaining replicas must carry peak load | | Quorum system with five members | minAvailable based on quorum plus repair margin | a healthy count may not equal safe membership | | Single replica | no PDB creates availability; minAvailable: 1 blocks voluntary eviction | maintenance can deadlock without planned downtime | | Broken Pods blocking a drain | consider unhealthyPodEvictionPolicy: AlwaysAllow | eviction must not wait forever for health that cannot return |

Check that the PDB selector matches exactly the intended controller. Inspect status.currentHealthy, desiredHealthy and disruptionsAllowed before maintenance. A policy object that selects nothing, the wrong version label or several unrelated workloads provides a false sense of protection.

Align Deployment capacity with draining

For rolling updates, maxUnavailable defines how much serving capacity may be removed and maxSurge defines temporary replacement capacity. minReadySeconds prevents a briefly-ready replacement from immediately counting as available. progressDeadlineSeconds reports a stalled rollout but does not automatically roll it back.

Capacity math must include draining replicas. Suppose an API has four desired replicas, maxUnavailable zero, maxSurge one and a 45-second grace period. The new Pod must become ready before an old Pod begins termination, but the old Pod may still consume resources for 45 seconds. Ensure the cluster has room for the new Pod plus old terminating work, connection pools and any surge in requests shifted to remaining replicas.

Use topology spread or anti-affinity so one node drain does not remove most replicas. Verify that HPA minimum replicas, disruption budget and rollout strategy describe compatible availability goals. A strict PDB cannot compensate for all replicas being placed on one node, and a perfect shutdown handler cannot serve traffic when the scheduler has no replacement capacity.

A production-oriented manifest

This example reserves a short propagation buffer but leaves most of the grace period for the application after SIGTERM:

~~~yaml apiVersion: apps/v1 kind: Deployment metadata: name: orders-api spec: replicas: 4 minReadySeconds: 10 progressDeadlineSeconds: 300 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 0 maxSurge: 1 template: spec: terminationGracePeriodSeconds: 45 containers: - name: api image: registry.example/orders-api:2026-10-18 ports: - name: http containerPort: 8080 readinessProbe: httpGet: path: /ready port: http periodSeconds: 3 timeoutSeconds: 1 failureThreshold: 2 lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 5"] --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: orders-api spec: minAvailable: 3 unhealthyPodEvictionPolicy: AlwaysAllow selector: matchLabels: app: orders-api ~~~

The image must actually contain the shell and sleep command for this exec hook. A distroless image often does not; use a supported Sleep hook on compatible clusters or implement the propagation phase in application code. The manifest does not make shutdown reliable by itself. The process still needs bounded SIGTERM handling, retry-safe work and enough capacity.

Observe shutdown as a first-class operation

Emit a shutdown_started event with Pod identity, reason when known and remaining deadline. Track in_flight_requests, active_jobs, graceful_shutdown_duration_seconds, forced_shutdown_total, rejected_during_drain_total and connection resets. Correlate them with deployment revision, PDB status, EndpointSlice conditions and node-drain events.

Alert on Pods that spend most of the grace period terminating, repeated exit code 137 with a termination context, drains blocked by disruptionsAllowed zero, elevated 5xx during rollout and replacements that never satisfy minReadySeconds. Exit code 137 alone is ambiguous because SIGKILL and memory pressure can both produce it; combine container state, reason, kubelet events and memory metrics.

Run failure drills before trusting the contract

Send requests whose durations span the grace window, start a rollout, and verify that acknowledged responses complete without resets. Drain one node under load and confirm that PDB blocks the second simultaneous eviction. Make one Pod unhealthy and verify that the chosen unhealthy eviction policy allows maintenance to progress. Exceed the cleanup deadline deliberately and confirm the timeout is visible rather than silent.

Also test the involuntary cases that PDB cannot prevent: stop a node, cut network access and exhaust replacement capacity. For workers, kill the process after it receives a message but before acknowledgement and prove the job is retried without duplicate side effects. For WebSockets, observe reconnect distribution so thousands of clients do not create a synchronized reconnect storm.

Anti-patterns and limits

Sleeping in preStop without measuring propagation is cargo cult reliability. Marking readiness false without handling existing connections abandons work. Relying on PDB during Deployment rollout misunderstands the controller boundary. Using force deletion during routine maintenance bypasses the grace contract. Giving every workload a very long grace period can stall operations and overcommit nodes.

Kubernetes graceful termination cannot guarantee external client behavior, exactly-once processing or database transaction recovery. Use idempotency keys, leases, checkpoints and durable queues for business correctness. Use application-level request deadlines and retry budgets so clients know when recovery is safe. For singletons that cannot tolerate overlap or absence, design explicit leadership and fencing rather than assuming Pod names provide exclusivity.

Production checklist

Handle SIGTERM in process 1 and test the built image. Stop new work before waiting for current work. Make cleanup idempotent and deadline-aware. Keep preStop short and include it in the one grace budget. Measure routing convergence and request duration. Configure Deployment availability and surge capacity independently from PDB. Use the Eviction API for maintenance and inspect PDB status before draining. Spread replicas across failure domains. Test graceful and involuntary failures under load. A shutdown is successful only when users see no unexplained errors and acknowledged work remains correct.

Prepared by: Noor Yasser