Prompt
I'm building a web app that needs background job processing, what should I use?
Latest observation
Oct 24, 2026
OpenAI APIWeb search: off
It depends on your stack and how much reliability/scale you need, but here’s the practical rule:
If you want the simplest good option
Use a queue + worker setup with a managed backend if possible.
Common choices by stack
- Node.js / TypeScript:
- BullMQ + Redis
- Good for: retries, delayed jobs, rate limiting, dashboards
- Python:
- Celery + Redis or RabbitMQ
- Good for: mature, feature-rich, widely used
- Ruby:
- Sidekiq + Redis
- Go:
- Asynq + Redis
- Java / JVM:
- RabbitMQ, Kafka, or Spring-based job infrastructure
- .NET:
- Hangfire or Quartz.NET
If you’re already on a cloud platform
Consider managed services:
- AWS: SQS + ECS/Lambda workers, or EventBridge for scheduling
- GCP: Cloud Tasks, Pub/Sub, Cloud Run jobs
- Azure: Service Bus, Azure Functions, WebJobs
Managed queues reduce ops burden and are a good default for production.
What to pick based on job type
- Short, reliable async work (emails, webhooks, image processing):
Use a queue worker - Scheduled jobs (cron-like):
Use a scheduler plus queue/worker - High-volume event streaming:
Use Kafka/PubSub - Simple app with low scale:
A queue-based worker is still best; avoid DIY in-app background threads
Things to avoid
- Running background jobs inside the web server process if you care about reliability
- Using only
setTimeout/setIntervalfor important jobs - Building your own retry/locking system unless you really need to
A solid default architecture
- Web app receives request
- App enqueues job in Redis/queue
- Worker process picks it up
- Worker retries on failure with backoff
- Store job status in DB if users need visibility
My default recommendation
- Node: BullMQ + Redis
- Python: Celery + Redis/RabbitMQ
- Need managed/cloud-native: SQS/Cloud Tasks/Service Bus
If you tell me:
- your backend language,
- expected job volume,
- whether you need cron/scheduling,
- and what cloud you use,
I can recommend a specific stack and architecture.