Prompt
How do I build a worker system that can scale independently?
Latest observation
To build a worker system that can scale independently, design it so workers are decoupled from the web/API layer and can be added or removed without affecting request handling.
Core idea
Split your system into:
- Producers — your app/API that creates work
- Queue — stores jobs durably
- Workers — consume jobs asynchronously
- Result/state store — tracks status, retries, results
This lets you scale workers based on backlog, CPU, memory, or latency without scaling the front-end app.
A practical architecture
1) Put jobs on a queue
When a user triggers work, don’t do the work inline. Instead:
- API validates request
- API writes a job record to DB or queue
- API returns a job ID immediately
Use a queue like:
- AWS SQS
- RabbitMQ
- Redis Streams / BullMQ
- Kafka
- Google Pub/Sub
For most app workflows, a durable queue like SQS or RabbitMQ is a good starting point.
2) Make workers stateless
Workers should not rely on local disk or in-memory state for correctness.
Each worker:
- pulls a job
- processes it
- updates job state
- acknowledges completion
- can be restarted anytime
If a worker dies mid-job, the queue should redeliver the job after visibility timeout / lease expiry.
3) Store job state separately
Have a database table or document store for:
job_id- status:
queued | running | succeeded | failed - attempts
- timestamps
- error message
- output/result pointer
This gives you:
- observability
- retries
- user-facing progress tracking
- idempotency support
4) Use idempotent job handlers
Because queues often deliver “at least once,” a job may run more than once.
Your worker logic should be safe if repeated:
- use idempotency keys
- check if result already exists
- avoid double-charging, double-sending, double-writing
Example:
- before processing, check whether
job_idis already marked completed - if so, skip
5) Scale workers independently
Run workers as a separate service, container, or deployment from the API.
Examples:
- Kubernetes: separate Deployment for workers, scale replicas independently
- ECS: separate service for workers
- Docker Compose locally: separate worker container
- Serverless: queue-triggered functions
Autoscale workers based on:
- queue depth
- oldest message age
- CPU utilization
- processing latency
- custom business metrics
A common rule: scale to queue backlog, not just CPU.
A simple request flow
- User submits a job to API
- API writes job to DB and queue
- API returns
job_id - Worker fetches job
- Worker processes it
- Worker writes result/status
- Client polls
/jobs/{job_id}or gets notified via webhook/WebSocket
Key design requirements
Reliability
- acknowledgements after successful processing
- retries with backoff
- dead-letter queue for poison messages
- visibility timeout longer than typical job duration
Throughput
- multiple worker instances
- concurrent processing per worker if jobs are I/O-bound
- batch fetching where supported
Fault tolerance
- job timeouts
- retry limits
- dead-letter handling
- idempotent writes
- graceful shutdown: stop taking new jobs, finish current ones, ack properly
Observability
Track:
- queue length
- job age
- success/failure rate
- retry count
- average processing time
- worker saturation
Example scaling strategy
If you’re using Kubernetes:
- API deployment: fixed or separately scaled based on request load
- Worker deployment: autoscale from 2 to 50 replicas based on queue depth
- HPA/KEDA can scale workers from queue metrics
KEDA is especially useful because it can scale based on queue length directly.
Common pitfalls
- Doing heavy work inside web requests
- Using a queue but keeping job state only in memory
- Not handling duplicate delivery
- No dead-letter queue
- Long-running jobs without heartbeats/leases
- Scaling workers but not protecting downstream dependencies like databases or third-party APIs
If you want a clean default stack
A solid general-purpose setup is:
- API: any framework
- Queue: SQS or RabbitMQ
- Workers: separate containerized service
- State: Postgres
- Autoscaling: KEDA or cloud-native autoscaling
- Monitoring: Prometheus/Grafana or cloud metrics
If you want, I can also give you:
- a reference architecture diagram,
- a sample implementation in Python/Node/Go, or
- a Kubernetes/KEDA setup for autoscaling workers.