Prompt
How do I build a worker system for long-running jobs?
Latest observation
A solid worker system for long-running jobs usually has 5 parts:
- API / producer — accepts a request and creates a job
- Job store / queue — holds jobs durably until workers can pick them up
- Workers — execute the long-running work asynchronously
- State tracking — lets clients query progress/status/result
- Retry / failure handling — handles crashes, timeouts, and transient errors
Recommended architecture
1) Put jobs in a durable queue
When a user starts a job, don’t do the work in the request thread. Instead:
- create a job record in a DB
- enqueue a job ID into a queue
- immediately return
202 Acceptedwith the job ID
Common queue choices:
- Redis-based: BullMQ, RQ, Sidekiq, Celery
- Database-backed: Postgres job table + polling or
FOR UPDATE SKIP LOCKED - Cloud queues: SQS, RabbitMQ, Pub/Sub, Azure Service Bus
For long-running jobs, a queue plus DB is usually enough.
2) Store job state in a database
Track:
job_idstatus: queued, running, succeeded, failed, canceledprogress: 0–100 or step-basedattempt_countstarted_at,finished_aterror_messageresult_locationif output is large
This lets your API answer:
- “Is my job done?”
- “Did it fail?”
- “How far along is it?”
3) Workers should be idempotent
Workers may run a job more than once if:
- a worker crashes mid-job
- the queue redelivers
- a timeout occurs
So design each job so rerunning it doesn’t corrupt data. Typical techniques:
- check if output already exists before writing
- use idempotency keys
- write to temp files then atomically rename
- use unique constraints in DB
4) Use acknowledgements and retries
A worker should:
- claim a job
- mark it
running - process it
- mark it
succeededorfailed - acknowledge completion to the queue
If the worker crashes before ack:
- the queue should make the job visible again
Add retry logic for transient failures:
- exponential backoff
- max attempts
- dead-letter queue or failed jobs table
5) Make long-running work resumable or checkpointed
If jobs can take minutes or hours, don’t rely on one in-memory process.
Good patterns:
- split the job into smaller steps
- save checkpoints after each step
- persist intermediate progress
- allow a job to resume from the last checkpoint
A simple implementation pattern
API flow
- Client calls
POST /jobs - Server creates DB record with
queued - Server sends
job_idto queue - Server returns:
{ "job_id": "abc123", "status": "queued" }
Worker flow
- Worker receives
job_id - Loads job from DB
- Sets status to
running - Executes task
- Saves progress/results
- Sets status to
succeededorfailed
Status endpoint
Client calls:
GET /jobs/abc123
Response:
{
"job_id": "abc123",
"status": "running",
"progress": 42
}
Important operational concerns
Concurrency control
If multiple workers can pick up the same job:
- use row locking (
SELECT ... FOR UPDATE SKIP LOCKED) - or queue semantics that guarantee single consumer delivery
- or a lease/visibility timeout system
Timeouts
Set timeouts for:
- job execution
- external API calls
- lock leases
- retries
But don’t set the overall job timeout so low that normal jobs fail.
Observability
Add:
- structured logs with
job_id - metrics: queue length, job duration, success/failure rates
- tracing if jobs call other services
Cancellation
Support canceling jobs:
- API sets status
canceled - worker checks periodically and exits gracefully
- if worker is in a blocking operation, use a timeout or cooperative cancellation token
Technology choices
If you want the simplest practical setup
- Postgres for job table
- Redis or Postgres queue
- Background worker process
- API server separate from workers
If you’re in a popular ecosystem
- Python: Celery + Redis/RabbitMQ
- Node.js: BullMQ + Redis
- Ruby: Sidekiq + Redis
- Go: custom worker + Postgres/SQS
- Java: Spring + RabbitMQ/SQS
Minimal design example
DB table
jobs(
id,
type,
status,
progress,
payload,
result,
error,
attempts,
created_at,
updated_at
)
Worker pseudo-code
while True:
job = claim_next_job()
if not job:
sleep(1)
continue
try:
mark_running(job.id)
result = run_job(job.payload)
save_result(job.id, result)
mark_succeeded(job.id)
except Exception as e:
record_failure(job.id, str(e))
retry_or_fail(job.id)
Best practices summary
- Return immediately from the API
- Persist job state in a DB
- Use a durable queue
- Make jobs idempotent
- Add retries and dead-letter handling
- Support progress tracking and cancellation
- Keep workers stateless
- Monitor everything
If you want, I can also give you:
- a Postgres-only implementation,
- a Redis + worker implementation, or
- a code example in Python/Node/Go.