Background work that survives failure

---
name: jobs-and-webhooks
description: "Applies “Background work that survives failure” as a repeatable product-quality gate. Use during backend work when idempotent jobs, retries with backoff, dead letters someone actually reads."
---

# Background work that survives failure

Apply this skill during **Backend** work. It supports Node / API, Python, Any stack.

Idempotent jobs, retries with backoff, dead letters someone actually reads.

## Instructions

Design every background job to be idempotent: keyed by a natural identifier so running it twice produces the same result. Retry with exponential backoff and a maximum attempt count, then route to a dead-letter queue that a human is notified about — silent exhaustion is data loss on a delay. Webhook endpoints acknowledge fast (validate, persist, 200) and do real work asynchronously. Protect scheduled work against overlap with locks or single-flight queues, and assume the scheduler will someday fire twice or not at all. Long-running work never blocks a user-facing request: enqueue, return, and notify on completion. For every async path, answer in writing: what happens if this fails halfway?

## Detect the problem

- Jobs that assume they run exactly once
- Webhook work done inline in the request
- A dead-letter queue nobody reads

## Hold the gate

Do not declare this phase complete until every item passes:

- [ ] Every job is idempotent — running it twice produces the same world
- [ ] Retries use backoff with a max attempt count; exhausted jobs land in a dead-letter path
- [ ] Webhook handlers ack fast and process async; slow work never blocks a request
- [ ] Scheduled work has overlap protection (lock/queue) — two runs can't collide
- [ ] A failed job is visible to a human within minutes, not discovered by a user

## Verify the result

Kill a job halfway and rerun it — the world must end in the same correct state.