Prompt
My workers need to process jobs across Python and Node services, what should I use?
Latest observation
If you need workers that process jobs across both Python and Node services, the best choice is usually a language-agnostic queue/broker rather than something tied to one runtime.
Good options
1. Redis-based queues
Good if you want something simple and fast.
- Python: RQ, Celery, Dramatiq, ARQ
- Node: BullMQ, Bee-Queue, Kue (older), Bull
Best when:
- You want easy setup
- You’re okay using Redis
- You mainly need background jobs, retries, delayed jobs, and scheduling
Note: Python and Node can both use Redis, but they typically won’t share the exact same queue library. They can still coordinate through Redis if you standardize your job format and queue names.
2. RabbitMQ
A solid message broker for cross-language workers.
- Works well with both Python and Node
- Good for task queues, routing, acknowledgements, retries
- More “enterprise” than Redis queues in many cases
Best when:
- You need reliable message delivery
- You want multiple services in different languages consuming the same jobs
- You need routing patterns like topics, fanout, or per-service queues
3. Kafka
Best for event streams, not typical worker queues.
Best when:
- You’re processing high-volume events
- You want durable streams and replayability
- Your workers are event consumers rather than classic “job queue” workers
Not ideal if:
- You just need simple background jobs with retries and per-job execution semantics
4. AWS SQS / Google Pub/Sub / Azure Service Bus
Great if you want managed infrastructure.
Best when:
- You’re in cloud and want less operational overhead
- You need easy integration across Python and Node
- You want built-in durability and scaling
My recommendation
If you want the simplest cross-language setup:
- Use RabbitMQ for worker messaging
If you want easiest implementation and already use Redis:
- Use Redis + separate queues in each service
- Or use a platform like BullMQ for Node and RQ/Celery for Python, with a shared Redis backend
If you want cloud-managed and scalable:
- Use SQS if you’re on AWS
Important design tip
Don’t make Python and Node share the same worker library unless you really need to. Instead:
- use a common broker
- define a shared job schema
- have each service consume the jobs it understands
- include fields like:
job_typepayloadidempotency_keyattempttrace_id
This keeps the system maintainable.
Short answer
If your goal is workers across Python and Node services, I’d choose:
- RabbitMQ for a general-purpose cross-language queue
- SQS if you want managed cloud infrastructure
- Redis/BullMQ/Celery only if you’re comfortable with separate language-specific worker libraries
If you want, I can suggest a concrete stack based on your environment:
- self-hosted vs cloud
- low volume vs high volume
- need for retries/delays/scheduling
- AWS/GCP/Azure/on-prem