Durable Jobs

A job queue is easy until a client retries, a worker disappears halfway through a job, or a deploy goes wrong. Durable Jobs is a small service that handles each of those cases explicitly, plus a lab that breaks it on purpose and records how long recovery takes.

How it works

• Durable, idempotent submission. A job is committed to PostgreSQL before the API acknowledges it. Submissions carry an idempotency key: a retry returns the original job, and a retry with different input is rejected.
• Claims and leases. Workers claim jobs with SKIP LOCKED and commit the claim before doing any work. Claims are leases, so a job held by a worker that vanished becomes claimable again once the lease expires.
• Fenced completion. Each claim carries a fencing token. A stale worker that wakes up after its lease moved on can’t overwrite the newer result.
• Bounded retries. Attempts back off and are capped, which separates temporary failures from jobs that will never succeed.
• Observability. JSON logs, per-attempt history, and database-backed Prometheus metrics for queue depth and latency, with Grafana dashboards and alerts.

Reliability lab

Terraform provisions a private EKS environment: internal HTTPS load balancing, private DNS, workload identities, and encrypted database storage. Six failure drills cover worker interruption, a database restart, slow writes, a failed rollout, invalid jobs, and retry exhaustion. A separate load test measures normal traffic.

These are synthetic results from a short single-node lab (one t3.medium node, two API replicas, one worker, one PostgreSQL pod) on September 28–29, 2026. They aren’t production capacity or availability claims, and the pod restart isn’t a backup-restore test.

Hosted CI runs container, race, integration, and smoke checks, offline infrastructure checks, and a disposable Kubernetes recovery demo.