How do I troubleshoot failed webhook deliveries when the provider says it was sent but my app never got it?

When a provider says “webhook sent” but your app never sees it, the key is to trace the request end-to-end and separate provider-side success from your server actually receiving and accepting it.

1) Confirm what “sent” means

Providers often mean one of these:

• They attempted delivery
• They got a 2xx response
• They queued it for retry
• Their internal logs show it left their system

Look for:

• HTTP status code returned to the provider
• Delivery timestamp
• Retry count
• Request ID / event ID
• Response body the provider received

If they got a 2xx, the problem is usually in your app’s routing, logging, or async processing. If not, it’s often network/TLS/auth/firewall related.

2) Check your webhook endpoint directly

From outside your infrastructure, test the exact endpoint the provider uses:

curl -i -X POST https://yourdomain.com/webhook \
  -H 'Content-Type: application/json' \
  -d '{"test":"ok"}'

Verify:

• DNS resolves correctly
• TLS certificate is valid
• Endpoint is public and reachable
• The path is correct
• It accepts the method and content type
• It responds quickly with a 2xx

Also test from multiple networks if possible.

3) Look at server logs at the edge

Check logs in this order:

• CDN / WAF / reverse proxy (Cloudflare, AWS ALB, Nginx, Apache)
• Load balancer logs
• App server logs
• Application-level webhook logs

Common issues:

• Request blocked by WAF/rate limiting
• Wrong Host header / virtual host routing
• Proxy forwarding misconfigured
• App route not mounted in production
• Request body too large
• Timeout before your app responded

4) Make sure you’re actually logging the inbound request

Sometimes the webhook arrives, but your code rejects it before you notice.

Log:

• Timestamp
• Request method and path
• Headers relevant to the provider
• Request body size
• Event ID / delivery ID
• Response status sent back

If possible, log the raw request before validation so you can distinguish:

• never arrived
• arrived but failed signature verification
• arrived but failed JSON parsing
• arrived but app errored

5) Validate signature and timestamp logic

A very common failure mode is “received but rejected.”

Check:

• Shared secret matches provider config
• You’re verifying the exact raw body, not a reserialized JSON object
• Timestamp tolerance isn’t too strict
• Header names are correct
• You’re not comparing against the wrong encoding

If signature verification fails, return a clear log entry and a distinct 4xx reason.

6) Check for redirects

Many webhook providers do not follow redirects reliably.

Avoid:

• HTTP → HTTPS redirects
• non-www → www redirects
• trailing slash redirects
• locale redirects

Webhook URL should be the final destination, returning 200/204 directly.

7) Inspect firewalls, allowlists, and IP restrictions

If your app is behind:

• firewall rules
• security groups
• IP allowlists
• zero-trust access controls

Make sure the provider’s IP ranges are allowed. Providers often rotate or publish ranges, so confirm they’re current.

Also check whether your infrastructure blocks:

• unknown user agents
• missing headers
• POST requests from certain regions

8) Confirm DNS and load balancer routing

A surprisingly common issue is stale or incorrect DNS.

Check:

• The webhook hostname resolves to the expected IP
• There’s no old A/AAAA record
• IPv6 isn’t broken if the provider prefers it
• Load balancer target health is good
• Multiple environments aren’t sharing the same URL

9) Check response time and timeouts

Even if the request reaches you, the provider may time out before your app finishes.

Best practice:

• Immediately return 200 OK or 204 No Content
• Process asynchronously in a queue/background job
• Avoid long database calls or external API calls inside the webhook handler

If your handler takes too long, the provider may retry or mark it failed.

10) Compare provider event IDs with your logs

Use the provider’s delivery ID or event ID to search:

• proxy logs
• app logs
• distributed tracing
• APM logs

If you have no trace of the event anywhere, it likely never reached your infrastructure.

11) Test with a webhook inspector

Use a tool like:

• RequestBin / webhook.site
• Pipedream
• ngrok / Cloudflare Tunnel

Point the provider at the inspector:

• If it arrives there, the provider is fine and your app/infrastructure is the issue
• If it doesn’t, the problem is on the provider or network path

12) Reproduce from the provider’s source network if possible

Some providers offer:

• test webhooks
• replay delivery
• resend functionality
• delivery logs with response headers

Use retries/replay to see whether failures are consistent or intermittent.

13) Watch for environment mismatches

Make sure:

• production vs staging webhook URL is correct
• secret belongs to the right environment
• deployment hasn’t overwritten the route
• blue/green or canary deployment isn’t sending traffic elsewhere

14) Add a dead-simple health endpoint and a dead-simple webhook endpoint

Sometimes the issue is app complexity. Create:

• /healthz for reachability
• /webhook/provider that does minimal work and logs aggressively

This helps isolate whether the framework, middleware, auth layer, or router is dropping the request.

15) Ask the provider for the exact delivery details

Request:

• timestamp
• destination URL
• source IP
• request headers
• status code returned
• response body
• retry history

With that, you can usually pinpoint the issue quickly.

---

Fast triage checklist

If you want the shortest path:

1. Check provider delivery logs for HTTP status.
2. Hit the endpoint with curl.
3. Inspect CDN/WAF/LB logs.
4. Verify signature logic using the raw body.
5. Ensure endpoint returns 2xx quickly.
6. Test via webhook.site or ngrok.
7. Confirm DNS, TLS, and firewall rules.

---

Best practice for webhook handlers

• Accept and acknowledge fast
• Log raw inbound metadata
• Verify signature using raw body
• Process asynchronously
• Make handlers idempotent
• Keep a replay/resend path

If you want, I can also give you a step-by-step debugging flowchart or a checklist tailored to a specific stack like AWS, Nginx, Cloudflare, Express, FastAPI, or Rails.