Prompt

How do I build a worker system for long-running jobs?

Technology / Cloud Infrastructure2 observationsLast seen Nov 8, 2026

Latest observation

Nov 8, 2026OpenAI APIWeb search: off

A solid worker system for long-running jobs usually has 5 parts:

  1. API / producer — accepts a request and creates a job
  2. Job store / queue — holds jobs durably until workers can pick them up
  3. Workers — execute the long-running work asynchronously
  4. State tracking — lets clients query progress/status/result
  5. Retry / failure handling — handles crashes, timeouts, and transient errors

Recommended architecture

1) Put jobs in a durable queue

When a user starts a job, don’t do the work in the request thread. Instead:

  • create a job record in a DB
  • enqueue a job ID into a queue
  • immediately return 202 Accepted with the job ID

Common queue choices:

  • Redis-based: BullMQ, RQ, Sidekiq, Celery
  • Database-backed: Postgres job table + polling or FOR UPDATE SKIP LOCKED
  • Cloud queues: SQS, RabbitMQ, Pub/Sub, Azure Service Bus

For long-running jobs, a queue plus DB is usually enough.

2) Store job state in a database

Track:

  • job_id
  • status: queued, running, succeeded, failed, canceled
  • progress: 0–100 or step-based
  • attempt_count
  • started_at, finished_at
  • error_message
  • result_location if output is large

This lets your API answer:

  • “Is my job done?”
  • “Did it fail?”
  • “How far along is it?”

3) Workers should be idempotent

Workers may run a job more than once if:

  • a worker crashes mid-job
  • the queue redelivers
  • a timeout occurs

So design each job so rerunning it doesn’t corrupt data. Typical techniques:

  • check if output already exists before writing
  • use idempotency keys
  • write to temp files then atomically rename
  • use unique constraints in DB

4) Use acknowledgements and retries

A worker should:

  • claim a job
  • mark it running
  • process it
  • mark it succeeded or failed
  • acknowledge completion to the queue

If the worker crashes before ack:

  • the queue should make the job visible again

Add retry logic for transient failures:

  • exponential backoff
  • max attempts
  • dead-letter queue or failed jobs table

5) Make long-running work resumable or checkpointed

If jobs can take minutes or hours, don’t rely on one in-memory process.

Good patterns:

  • split the job into smaller steps
  • save checkpoints after each step
  • persist intermediate progress
  • allow a job to resume from the last checkpoint

A simple implementation pattern

API flow

  1. Client calls POST /jobs
  2. Server creates DB record with queued
  3. Server sends job_id to queue
  4. Server returns:
    { "job_id": "abc123", "status": "queued" }
    

Worker flow

  1. Worker receives job_id
  2. Loads job from DB
  3. Sets status to running
  4. Executes task
  5. Saves progress/results
  6. Sets status to succeeded or failed

Status endpoint

Client calls:

  • GET /jobs/abc123

Response:

{
  "job_id": "abc123",
  "status": "running",
  "progress": 42
}

Important operational concerns

Concurrency control

If multiple workers can pick up the same job:

  • use row locking (SELECT ... FOR UPDATE SKIP LOCKED)
  • or queue semantics that guarantee single consumer delivery
  • or a lease/visibility timeout system

Timeouts

Set timeouts for:

  • job execution
  • external API calls
  • lock leases
  • retries

But don’t set the overall job timeout so low that normal jobs fail.

Observability

Add:

  • structured logs with job_id
  • metrics: queue length, job duration, success/failure rates
  • tracing if jobs call other services

Cancellation

Support canceling jobs:

  • API sets status canceled
  • worker checks periodically and exits gracefully
  • if worker is in a blocking operation, use a timeout or cooperative cancellation token

Technology choices

If you want the simplest practical setup

  • Postgres for job table
  • Redis or Postgres queue
  • Background worker process
  • API server separate from workers

If you’re in a popular ecosystem

  • Python: Celery + Redis/RabbitMQ
  • Node.js: BullMQ + Redis
  • Ruby: Sidekiq + Redis
  • Go: custom worker + Postgres/SQS
  • Java: Spring + RabbitMQ/SQS

Minimal design example

DB table

jobs(
  id,
  type,
  status,
  progress,
  payload,
  result,
  error,
  attempts,
  created_at,
  updated_at
)

Worker pseudo-code

while True:
    job = claim_next_job()
    if not job:
        sleep(1)
        continue

    try:
        mark_running(job.id)
        result = run_job(job.payload)
        save_result(job.id, result)
        mark_succeeded(job.id)
    except Exception as e:
        record_failure(job.id, str(e))
        retry_or_fail(job.id)

Best practices summary

  • Return immediately from the API
  • Persist job state in a DB
  • Use a durable queue
  • Make jobs idempotent
  • Add retries and dead-letter handling
  • Support progress tracking and cancellation
  • Keep workers stateless
  • Monitor everything

If you want, I can also give you:

  1. a Postgres-only implementation,
  2. a Redis + worker implementation, or
  3. a code example in Python/Node/Go.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.