Prompt
Need a simple queue for slow tasks off the request path
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
A simple pattern for “slow tasks off the request path” is:
- Accept the request
- Persist a task record
- Return immediately
- Have a worker process the queue in the background
Minimal architecture
- API / web app: enqueues work
- Queue storage: Redis, database table, SQS, RabbitMQ, etc.
- Worker process: polls queue and runs the slow jobs
Simplest implementation options
Option A: Use your existing database
Best if you want the least moving parts.
Create a table like:
CREATE TABLE job_queue (
id BIGSERIAL PRIMARY KEY,
job_type TEXT NOT NULL,
payload JSONB NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
attempts INT NOT NULL DEFAULT 0,
run_at TIMESTAMP NOT NULL DEFAULT NOW(),
locked_at TIMESTAMP NULL,
finished_at TIMESTAMP NULL,
last_error TEXT NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
Worker loop:
- select next
pendingjob - mark it
processing - execute it
- mark
doneorfailed - retry failed jobs with backoff
Useful query pattern:
SELECT *
FROM job_queue
WHERE status = 'pending' AND run_at <= NOW()
ORDER BY created_at
LIMIT 1
FOR UPDATE SKIP LOCKED;
This avoids two workers grabbing the same job.
Option B: Use Redis
Good if you need speed and can accept slightly more infrastructure.
- push job IDs/payloads into a Redis list or stream
- worker blocks waiting for jobs
- acknowledge when done
Option C: Managed queue
If you’re in cloud infra, easiest operationally:
- AWS SQS
- Google Pub/Sub
- Azure Queue Storage
- RabbitMQ if you already run it
Basic worker behavior
- Idempotent jobs: safe to run more than once
- Retries: 3–5 attempts with exponential backoff
- Timeouts: prevent stuck jobs
- Dead-letter queue: capture permanently failing jobs
- Visibility: log job IDs and errors
Tiny example flow
Request handler
def create_order(request):
order = save_order(request.json)
enqueue_job("send_email", {"order_id": order.id})
return {"ok": True, "order_id": order.id}, 202
Worker
while True:
job = fetch_next_job()
if not job:
sleep(1)
continue
try:
if job.type == "send_email":
send_email(job.payload["order_id"])
mark_done(job.id)
except Exception as e:
mark_failed_or_retry(job.id, str(e))
Recommended “simple but solid” setup
If you want minimal complexity:
- Postgres queue table
- one worker process
FOR UPDATE SKIP LOCKED- retry + dead-letter table
- idempotent task design
If you want, I can give you:
- a ready-to-use SQL schema
- a Python/Node/Go worker example
- or a production-safe design for your stack