Prompt
How do I move slow work off my web requests with a message queue?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
A message queue helps you turn a slow, user-facing web request into two parts:
- Quick web response: accept the request, validate it, and enqueue work.
- Background processing: a worker consumes the queued job and does the slow task asynchronously.
Basic flow
-
User submits a request
- Example: “Generate PDF”, “Send email”, “Resize image”, “Run report”.
-
Web app enqueues a job
- The request handler adds a message to a queue.
- It returns immediately, often with:
202 Accepted- a job/task ID
- a way to check status later
-
Worker processes the job
- A separate service reads messages from the queue.
- It performs the slow operation.
- It stores results or updates job status in a database/cache.
-
Client checks status or gets notified
- Poll an endpoint like
/jobs/{id} - Or use webhooks / email / websocket updates
- Poll an endpoint like
Why this helps
- Lower request latency
- Better user experience
- Fewer timeouts
- Scales independently: web servers and workers can scale separately
- More resilience: the queue buffers bursts of traffic
Common architecture
Web server
- Validate input
- Create job record in DB
- Push job message to queue
- Return response
Queue
- RabbitMQ, SQS, Azure Service Bus, Google Pub/Sub, Kafka, Redis queue, etc.
Worker
- Pull message
- Do work
- Update job status/result
- Ack message
Important design patterns
1. Make jobs idempotent
Workers may receive the same message more than once. Your job should be safe to run twice.
Example:
- If sending an email, record that a notification with
job_id=123was already sent. - If generating a report, write output to a deterministic location and overwrite safely.
2. Store job state separately
Keep a jobs table or status store with:
job_idstatus: queued, running, succeeded, failedcreated_at,updated_atresult_locationorerror_message
3. Use retries carefully
- Retry transient errors automatically
- Set a max retry count
- Send failures to a dead-letter queue after repeated failures
4. Keep the payload small
Pass references, not large blobs:
- Good:
{ user_id: 42, report_id: 991 } - Avoid: entire PDF/image/file contents
5. Acknowledge only after success
Workers should acknowledge a message after the job completes successfully. If the worker crashes first, the message can be re-delivered.
Example request lifecycle
API request
POST /reports
{
"type": "monthly-sales",
"month": "2026-01"
}
Web response
{
"job_id": "abc123",
"status": "queued"
}
HTTP status: 202 Accepted
Worker does:
- Read job
abc123 - Generate report
- Save to object storage
- Mark job as
succeeded
Status endpoint
GET /jobs/abc123
{
"job_id": "abc123",
"status": "succeeded",
"result_url": "https://..."
}
Example pseudocode
Web handler
def create_report(request):
job = db.insert_job(status="queued", type="report")
queue.publish({
"job_id": job.id,
"type": "report",
"month": request.json["month"]
})
return {"job_id": job.id, "status": "queued"}, 202
Worker
def worker_loop():
while True:
msg = queue.consume()
try:
db.update_job(msg["job_id"], status="running")
generate_report(msg["month"])
db.update_job(msg["job_id"], status="succeeded")
queue.ack(msg)
except Exception as e:
db.update_job(msg["job_id"], status="failed", error=str(e))
queue.nack(msg) # or retry / dead-letter
When not to use a queue
A queue is useful when the task is:
- slow
- retryable
- independent of the immediate HTTP response
Don’t use it when:
- the user must get the result immediately
- the work is tiny and simpler to do inline
- the task needs a strict synchronous transaction with the request
A practical starting architecture
If you’re starting from a simple web app:
- Web app: handles requests and creates job records
- Queue: SQS/RabbitMQ/Redis
- Worker service: separate process/container
- DB: tracks job status
- Object storage: stores large outputs
- Status endpoint: lets the client poll progress
Things to watch out for
- Duplicate job execution
- Lost messages if publishing and DB writes aren’t coordinated
- Long-running workers timing out
- Queue backlog during traffic spikes
- Not enough observability: logs, metrics, tracing
If you want, I can also show:
- a concrete example in Node.js, Python, or Go, or
- a recommended setup for SQS + worker + database status table.