Prompt
How do I keep webhook processing from blocking my app?
Latest observation
To keep webhook processing from blocking your app, the key idea is: acknowledge the webhook quickly, then do the real work asynchronously.
Recommended pattern
- Receive the webhook
- Validate it quickly if needed (signature, basic schema)
- Enqueue the payload into a background queue / job system
- Return 2xx immediately to the sender
- Process the webhook in a worker/background task
This prevents the webhook sender from waiting on your slow downstream logic, database work, or external API calls.
Why this helps
Webhook handlers can block your app if they do any of these synchronously:
- database writes with locks
- network calls to other APIs
- expensive computation
- retries/backoff logic
- calling internal services that may be slow
If you move those steps out of the request path, your app stays responsive and your webhook endpoint can handle bursts better.
Common implementation options
1) Queue-based processing
Use a job queue like:
- Sidekiq / Redis
- Celery / RabbitMQ or Redis
- BullMQ / Redis
- SQS + worker
- Kafka consumer
- Cloud Tasks / Pub/Sub / Azure queues
Flow:
- HTTP handler puts event on queue
- worker consumes and processes later
This is the most common and robust approach.
2) Background thread/process
If you’re in a small app or prototype, you can hand work off to a background thread or async task.
Caveat: this is usually not reliable enough for production because:
- process restarts can lose work
- scaling becomes messy
- you may still block the event loop depending on framework
3) Event streaming / outbox pattern
If webhook handling updates your DB and then triggers more work, consider:
- write event to DB transactionally
- publish from an outbox table
- worker processes from outbox
This reduces the chance of losing events if the app crashes after DB write but before enqueue.
Important practices
Respond fast
Aim to return within a few hundred milliseconds, or at least under the sender’s timeout.
Make processing idempotent
Webhook providers often retry deliveries. Your worker should handle duplicates safely by:
- storing webhook event IDs
- checking if already processed
- using upserts / deduplication keys
Separate validation from processing
Do lightweight checks in the request:
- signature verification
- required fields
- event type
- payload size limits
Then defer heavy logic.
Handle failures in the worker
Use retries, dead-letter queues, and alerting. Don’t retry inside the HTTP request.
Log and trace
Capture:
- webhook ID
- received time
- processing status
- retries
- errors
This makes debugging much easier.
Example architecture
Webhook sender -> HTTP endpoint -> queue -> background worker -> DB / external services
Simple pseudocode
def webhook_handler(request):
verify_signature(request)
event = parse(request.body)
enqueue(event)
return 200
Then in a worker:
def process_event(event):
if already_processed(event.id):
return
do_business_logic(event)
mark_processed(event.id)
If you tell me your stack…
I can show you the best way to do this in your framework, e.g.:
- Node/Express
- Python/Django or FastAPI
- Ruby on Rails
- Go
- PHP/Laravel
- .NET
- serverless setups