Kubernetes TLS rotation: renew certificates without stale Pods

Kubernetes TLS rotation is safe only when issuance, Secret propagation and certificate reload are treated as three separate stages. cert-manager can renew the certificate and replace the Secret correctly, but zero downtime still depends on the TLS consumer detecting the new files or restarting gracefully. Mount the whole Secret volume rather than a subPath, make reload behavior explicit, verify the certificate served from outside the cluster and alert on both expiry and reload lag. This matters to platform teams running Ingress, gateways or application-level TLS.

This guide is based on the official cert-manager, Kubernetes and ingress-nginx documentation verified on 21 October 2026. The manifests and timing budgets are examples, not universal defaults; confirm issuer limits, controller behavior and the certificate lifetime used by your environment.

Separate issuance, delivery and activation

A green Certificate condition proves that cert-manager obtained material and wrote the target Secret. It does not prove that a load balancer, Ingress controller or application is serving that material. The production path has three contracts:

1. issuance: the issuer signs a certificate before the renewal deadline;
2. delivery: Kubernetes projects the updated Secret to the consumer or the controller observes the API update;
3. activation: the TLS process reloads the key pair without dropping useful traffic.

Measure each boundary independently. Record the Certificate Ready condition, Not After timestamp, Secret resourceVersion, the certificate serial number or fingerprint observed by the consumer, and the serial served from an external probe. A single “certificate valid” dashboard hides the most common stale-consumer failure: the Secret is current while clients still receive the previous certificate.

If TLS terminates at a managed cloud load balancer, its certificate service may own stages two and three. Do not add cert-manager merely to mirror a control plane you do not use. If TLS terminates in ingress-nginx or an application Pod, you own the reload contract and must test it.

Declare the rotation policy instead of relying on memory

cert-manager changed the default private-key rotation policy in v1.18: versions at or above v1.18 default to Always, while older releases default to Never. Declare the policy explicitly so a chart downgrade, cluster migration or mixed environment cannot silently change the security contract. With Always, cert-manager generates a new private key during reissuance and waits until the new certificate is signed before replacing tls.key in the Secret.

Use renewBefore or renewBeforePercentage to leave enough time for issuer retries and operational repair. Do not set a window by copying another cluster: the useful margin depends on certificate duration, issuer rate limits, DNS or HTTP challenge latency, change freezes and the time humans need to respond.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: checkout-api
  namespace: edge
spec:
  secretName: checkout-api-tls
  dnsNames:
    - api.example.com
  issuerRef:
    kind: ClusterIssuer
    name: production-acme
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always
  renewBefore: 360h

Treat the Certificate resource as desired state and the Secret as generated output. Do not patch tls.crt or tls.key manually under the same secretName. Manual edits can create a key/certificate mismatch and will be reconciled by the controller. For an intentional early rotation, cert-manager recommends cmctl renew rather than deleting the Secret.

Choose a reload contract for the real TLS consumer

Kubernetes updates Secret-backed volumes eventually, not synchronously. The kubelet watches or polls according to its configured change-detection strategy, and projection can take a sync period plus propagation delay. A Secret mounted with subPath never receives automated updates. A Secret consumed through environment variables also remains stale until the container restarts.

That creates three valid consumer patterns:

• controller watches the Kubernetes API: common for Ingress and gateway controllers; verify that the specific controller observes Secret updates and reloads safely;
• application watches the mounted directory: read the new key pair together, validate that they match, then atomically swap the in-memory TLS configuration;
• application loads once at startup: trigger a controlled Deployment rollout when the Secret changes and preserve capacity with readiness, surge and draining settings.

Never assume that changing a mounted file changes a process. Many servers load certificates once. Conversely, restarting every workload on every Secret change creates avoidable churn when a controller can reload in place. The decision should be a documented capability of the TLS terminator, not a guess in a Helm chart.

For file watchers, watch the directory rather than one inode. Kubernetes projects volume updates using link changes, so an implementation that watches only the original file can miss replacement. Read tls.crt and tls.key after the update settles, parse both, confirm that their public keys match, then expose the new certificate. Keep the last known-good pair if parsing or matching fails.

Design a zero-downtime activation path

The safe sequence is prepare, observe, activate and verify. cert-manager prepares the signed pair before overwriting the Secret when rotationPolicy is Always. Kubernetes or the controller then observes the change. The consumer activates only a complete, matching pair. Finally, an external probe confirms what clients receive.

For a startup-only application, a restart controller can annotate the Pod template when the Secret changes. That mechanism is operationally convenient but is not a substitute for Deployment safety. Require at least two ready replicas where availability needs it, use maxUnavailable: 0 and a positive maxSurge when capacity allows, and keep readiness false until the process has loaded the new certificate. Reuse the Kubernetes graceful shutdown pattern so old Pods drain rather than terminate active connections.

Certificate Ready with revision N
  -> Secret resourceVersion changes
  -> consumer observes a complete key pair
  -> in-memory reload or controlled rollout
  -> readiness remains healthy
  -> external TLS probe serves revision N
  -> old certificate remains accepted only during bounded propagation

For ingress-nginx, the official architecture lists a Secret update as a synchronization event requiring configuration reload. That is evidence for that controller, not a promise for every Ingress implementation. Pin the controller version and run a rotation test after upgrades.

Verify the certificate that users actually receive

Do not finish a rotation runbook at kubectl describe certificate. Test from outside each network path: public Internet, private edge, regional load balancer and service mesh if they terminate TLS independently. Compare subject alternative names, issuer, Not Before, Not After and fingerprint with the intended revision.

Probe all hostnames, not only the apex. A wildcard and an explicit hostname may route to different listeners. Test SNI explicitly because the default certificate can appear healthy while the named virtual host serves stale material. If multiple regions or anycast edges cache independently, sample each region until the convergence objective is met.

Track these timestamps: renewal started, signed certificate became Ready, Secret changed, each consumer acknowledged reload, and the external probe first observed the new fingerprint. Their differences expose whether the bottleneck is the issuer, Kubernetes delivery or application activation.

Alert before expiry with enough margin to repair. Also alert when Certificate Ready is false, when a challenge or order remains pending, when the Secret revision advances but the served fingerprint does not, and when different replicas serve different serials beyond the expected overlap. Expiry monitoring alone cannot detect a newly issued certificate stuck behind a stale process.

Rehearse normal renewal and emergency key replacement

Automatic renewal tests the ordinary path, but a compromised private key needs an urgent rotation. Rehearse cmctl renew in a non-production namespace, verify that Always produces a new private key, confirm the consumer reloads, and measure external convergence. Then repeat in production under a controlled change window before an incident requires it.

Do not delete the Secret to force rotation. cert-manager explicitly recommends renewal through cmctl; deletion briefly removes the object other components depend on and makes recovery timing less predictable. Keep issuer rate limits in the runbook so a fleet-wide emergency does not create a renewal storm.

Disaster recovery has a different order. cert-manager documents restoring Secrets before Certificate resources to avoid unnecessary reissuance; restore Secrets before annotated Ingress resources when using ingress-shim. Exclude ephemeral Order, Challenge and usually CertificateRequest resources from backups according to the cert-manager guidance. Test this order on your own cluster rather than treating a backup archive as proof of recoverability.

Best practices, anti-patterns and limits

Use least-privilege RBAC so only the TLS controller and required workloads can read the Secret. Keep the Secret in the same namespace as the consumer when the API requires it, and avoid copying private keys across namespaces unless a reviewed controller owns that distribution. Kubernetes Secrets are not encrypted merely because they are Secrets; configure encryption at rest and protect etcd backups.

Useful practices are explicit rotationPolicy, bounded renewal margin, whole-volume mounts, atomic pair validation, an external served-certificate probe, staged rollout tests and a documented emergency command. Useful evidence includes resource versions and fingerprints, not private-key contents.

Anti-patterns include subPath mounts for rotating material, environment variables for live TLS keys, deleting Secrets to renew, restarting all workloads without readiness and surge capacity, trusting only the control-plane Ready condition, and logging or exporting private keys for debugging. Do not use temporary certificates as a normal availability strategy; they are a bootstrap behavior for specific flows, not proof that clients will trust the endpoint.

This design cannot remove every interruption. Long-lived TLS connections continue with the certificate negotiated when the connection started, some load balancers converge asynchronously, and a consumer may not support graceful reload. Define the overlap and convergence objective accordingly. When a managed certificate service already terminates TLS and provides rotation guarantees, prefer its native lifecycle unless Kubernetes must own the key material.

The production contract is concise: cert-manager issues a complete pair, Kubernetes delivers it without subPath, the real terminator reloads or rolls safely, and external probes prove what users receive. When those four statements are observable and rehearsed, automatic renewal becomes an operational capability rather than a hopeful YAML field.

Prepared by: Noor Yasser