Kubernetes CronJobs without duplicate or missing work

A reliable Kubernetes CronJob is not a guarantee that business work happens exactly once. Kubernetes documents that a CronJob creates Jobs approximately once per scheduled time and that, in some circumstances, two Jobs can be created or no Job can be created. Production correctness therefore comes from a layered contract: give every scheduled occurrence a stable identity, make side effects idempotent, choose an explicit concurrency policy, bound how late a run may start and how long it may execute, classify retryable failures, observe expected versus completed occurrences, and reconcile gaps from an external source of truth. This is useful for billing, reports, renewals, inventory synchronization, backups and notifications where a duplicate or missing run has real cost.

The official Kubernetes CronJob documentation states the core limitation directly: Job creation is approximate, duplicate creation is possible, omission is possible, and Jobs should be idempotent. That is a documented platform fact. The database patterns, service boundaries and alerting model below are engineering guidance for turning that scheduler behavior into a dependable product workflow.

Treat CronJob as a trigger, not the system of record

A CronJob answers when Kubernetes should attempt to create a Job. It does not know whether an invoice was already issued, an order was already synchronized or a customer already received a notification. Those facts belong in the application database or another authoritative system. Let the scheduler propose an occurrence; let the application decide whether the corresponding business effect is new, already complete, currently leased, failed and retryable, or permanently rejected.

The minimal durable record is an occurrence ledger. Its identity should combine workflow name and logical schedule time, not Pod name or Job UID. A unique constraint on workflow_key plus scheduled_for makes competing Jobs converge on the same row. Store status, attempt count, lease owner and expiry, input version, output reference, last error, started_at and completed_at. If the insert loses the uniqueness race, the second worker should read the existing state and exit or help resume it according to policy.

Kubernetes v1.32 and later annotate CronJob-created Jobs with batch.kubernetes.io/cronjob-scheduled-timestamp, formatted as RFC3339. Use that timestamp when available as the occurrence time. For compatibility or manual replays, also pass an explicit occurrence value into the Job and validate it. Never derive identity from the Pod start time: a delayed or retried Pod would become a different business occurrence.

Decide what duplicate safety means

Idempotency is not merely “check then write.” Two workers can both check before either writes. Put the guard and the state change in the same atomic boundary. For a database mutation, claim the occurrence through a unique insert or compare-and-swap, update business rows in a transaction, and commit an outbox event in that transaction. A separate dispatcher can deliver email, webhook or broker messages with its own delivery key.

External providers need their own idempotency boundary. If a payment or email API accepts an idempotency key, derive it from the occurrence and business entity and persist the provider response. If it does not, write a durable intent before calling, record the remote reference immediately after, and reconcile ambiguous timeouts instead of blindly retrying. A network timeout means “outcome unknown,” not “nothing happened.”

Some work is naturally repeatable: recomputing a cache from immutable input or replacing an object at a deterministic key. Some is not: charging a card, incrementing a balance, appending a ledger entry or sending a user-visible message. Classify every step and isolate irreversible effects behind the strongest available idempotency control.

Choose concurrencyPolicy from business semantics

The CronJob spec supports Allow, Forbid and Replace. Kubernetes documents Allow as the default. Allow lets schedules overlap and is appropriate only when occurrences are independent and downstream capacity is bounded elsewhere. It does not remove the need for idempotency because duplicate Jobs for one occurrence can still exist.

Forbid skips a new run while the prior Job from that CronJob is still running. It fits a workflow that must not overlap, such as rebuilding one shared snapshot, but it can silently turn slow execution into missed occurrences. Kubernetes counts an occurrence blocked by Forbid as missed. Monitor runtime against interval and decide whether the skipped business window should be reconciled later.

Replace stops the currently running Job when a new schedule arrives and starts the new one. Use it only when stale work has no value and the application can tolerate interruption, such as refreshing a disposable projection. It is unsafe for a multi-step workflow that can leave partial external effects unless each step is resumable and compensatable.

Concurrency policy is scoped to Jobs created by the same CronJob. The official docs note that separate CronJob objects may still run concurrently. If two schedules can touch the same account, inventory row or export target, enforce exclusion or serialization in the application layer rather than assuming Kubernetes coordinates them.

Separate start lateness from execution duration

The fields startingDeadlineSeconds and activeDeadlineSeconds solve different problems. The CronJob-level starting deadline limits how late the controller may create a Job after its scheduled time. When that window is exceeded, Kubernetes skips the occurrence; future schedules continue. The Job-level active deadline limits total Job duration across Pod retries. When it expires, running Pods are terminated and the Job fails with DeadlineExceeded.

Set the start deadline from business freshness. A morning exchange-rate import may still be useful 20 minutes late; an every-minute reconciliation may not be useful hours later. Do not set it below ten seconds: the CronJob controller checks on a ten-second cadence and Kubernetes warns that such a Job might not be scheduled. A very large or absent deadline can produce undesirable catch-up behavior after an outage.

Set activeDeadlineSeconds from the longest acceptable execution, including backoff and recovery, not from the typical P50 runtime. Kubernetes states that the active deadline takes precedence over backoffLimit. The timeout should leave enough room for the worker to stop safely, release leases and record the outcome before the Pod is terminated.

The two deadlines form a service contract: start no later than the freshness budget and finish no later than the processing budget. A deadline failure must enter reconciliation; it should not disappear as a generic Pod error.

Make schedule and time zone explicit

Set spec.timeZone rather than relying on the kube-controller-manager local zone. Kubernetes documents timeZone as stable since v1.27 and rejects CRON_TZ or TZ embedded in the schedule. Use Etc/UTC when the workflow is interval-based or global. Use an IANA business zone only when the promise is genuinely local, such as “09:00 Cairo time.”

Daylight-saving transitions make local schedules ambiguous or nonexistent in some regions. Define the product rule: run once at the next valid instant, skip, or represent both offsets as separate occurrences. The occurrence ledger should store an absolute timestamp and, when relevant, the intended local date and zone. Do not let a container image timezone silently reinterpret the controller’s schedule.

Changing a CronJob updates only future Jobs. Existing Jobs and Pods continue with the old template. During a rollout, include configuration version in the occurrence record and logs so operators can distinguish old and new behavior.

Classify retries instead of retrying everything

The Kubernetes Job documentation says backoffLimit defaults to six and failed Pods are recreated with exponential backoff, capped at six minutes. A retry is useful for a transient network failure or evicted node; it wastes time and can repeat damage for invalid credentials, malformed input or a violated business rule.

Use podFailurePolicy to classify known exit codes and Pod conditions. The official failure-policy task demonstrates actions such as FailJob for a terminal application code, Ignore for a disruption that should not consume the retry budget, and Count for ordinary failures. Design the container entrypoint to emit deliberate exit codes and structured error reasons. Avoid a shell wrapper that converts every failure into exit code 1 and destroys the distinction.

Keep restartPolicy at Never when you want each attempt visible as a separate Pod with preserved logs. OnFailure may restart a container inside the same Pod and complicate attempt accounting. Whichever you choose, application idempotency must survive process restart between the remote side effect and the local success record.

A production architecture

The control plane evaluates schedule, time zone, suspension, starting deadline and concurrency policy, then creates a Job. The Pod reads the scheduled occurrence, calls a coordinator to claim the durable ledger row, executes bounded business steps, commits state plus outbox records, and marks the occurrence complete. A dispatcher delivers external events. A separate reconciler compares authoritative business windows with completed ledger rows and creates explicit replay requests for missing or ambiguous work.

For a daily settlement at 02:00 UTC, a practical specification might use timeZone Etc/UTC, concurrencyPolicy Forbid, a start deadline of 1,800 seconds, an active deadline of 2,400 seconds, backoffLimit 3, and a TTL long enough for operational investigation. These values are illustrative, not universal. Derive them from freshness, worst-case duration, provider limits and incident-response needs.

Do not let the CronJob Pod own an unbounded scan. Partition work into deterministic pages or entities, checkpoint progress, and make each item independently idempotent. If volume can exceed one interval, the CronJob should enqueue bounded tasks to a durable queue and exit, rather than overlap a growing monolith.

Observe expected, created, started and completed

A green Pod count is not a business SLO. Track four timestamps: expected schedule, Job creation, first worker start and business completion. From them derive schedule lag, queue or scheduling lag, execution duration and end-to-end freshness. Tag metrics with workflow and result class, but avoid high-cardinality occurrence IDs in metric labels; keep those in logs and traces.

Alert when no successful occurrence exists inside the freshness window, when duplicate claim conflicts spike, when Forbid causes skipped schedules, when runtime approaches the interval, when active deadlines fire, when terminal failures repeat or when reconciliation finds gaps. Controller logs mentioning too many missed start times are useful, but business reconciliation is stronger because it asks whether required work exists, not whether the controller emitted an object.

For Kubernetes versions that provide the scheduled-timestamp annotation, record it alongside Job UID and the application occurrence key. Correlate Pod logs, traces, database ledger and external provider reference. An operator should be able to answer: which business window was intended, which code and configuration ran, what effects committed, and whether a safe replay is available?

Retention and cleanup

CronJob history limits control how many successful and failed Jobs remain associated with the schedule. The TTL-after-finished controller can remove a completed or failed Job after spec.ttlSecondsAfterFinished, including dependent objects. Kubernetes notes that the TTL timer starts when the Job reaches Complete or Failed and that time skew can affect cleanup timing.

Object cleanup is not audit retention. Export logs and retain the application occurrence ledger according to business and compliance needs before Kubernetes deletes Jobs and Pods. Keep failures longer than successes if that improves investigation, but avoid unlimited object accumulation in the API server. Verify that your logging pipeline flushes before Pod removal.

Decision guide

Use Allow when each occurrence is independent, overlap is acceptable and application-level admission protects dependencies. Use Forbid when only one execution may operate at a time and a skipped schedule is explicitly detected and reconciled. Use Replace when only the newest refresh matters and cancellation cannot corrupt external state. Use a durable queue or workflow engine instead of CronJob when work needs per-item retries, long-running orchestration, human approval, event-driven execution or strong delivery guarantees across many steps.

Use Kubernetes Job directly for a one-off bounded task. Use an external managed scheduler when the trigger must survive cluster loss or coordinate multiple clusters. Use a database scheduler only when database ownership and load are intentional. CronJob is a strong cluster-native trigger; it is not automatically the best owner of the business calendar.

Best practices and anti-patterns

Pin an explicit image digest, run with a dedicated least-privilege ServiceAccount, set resource requests and limits, make shutdown handling record an interruptible state, and validate the occurrence before work. Test duplicate Job creation, delayed start, controller downtime, Pod eviction, provider timeout after success, lost database connection after commit, daylight-saving transition and manual replay.

Avoid using Pod name as an idempotency key, trusting Forbid as duplicate prevention, leaving timeZone implicit, treating timeout as failure-before-effect, retrying every exit code, deleting Jobs before logs are durable, or alerting only on Pod failure. Never update a “last successful run” timestamp before the business transaction commits.

Limits

Kubernetes can create and supervise Jobs, but it cannot prove semantic success inside an external payment, ERP, email or storage system. A Complete Job means the containers met the Job success criteria; it does not prove every intended record was processed. Conversely, a failed Pod may have completed an external effect before losing its acknowledgement.

Exactly-once execution is not a realistic scheduler promise across distributed systems. Aim for at-least-once triggering plus exactly-once-or-idempotent business effects, observable ambiguity and deterministic reconciliation. That design survives both duplicate and missing scheduler events.

Executive checklist

Define the logical occurrence key and unique constraint. Choose timeZone, concurrencyPolicy, starting deadline, active deadline and retry policy from business semantics. Make every irreversible effect idempotent or reconcilable. Persist an occurrence ledger and configuration version. Observe expected-to-completed freshness, not only Pods. Retain evidence before TTL cleanup. Rehearse duplicate, omission, delay, interruption and ambiguous timeout. If the workflow cannot be safely replayed, it is not ready for a production CronJob.

Prepared by: Noor Yasser