What’s the best way to build event-driven integrations without polling?

The best way is to use webhooks or message-driven event streams instead of polling, depending on how much control you have over the systems involved.

Good options, from simplest to most robust

1. Webhooks

Have the source system call your integration when something happens.

• Best for: SaaS apps, payments, CRM, ticketing, CI/CD events
• Pros: Near real-time, efficient, simple to implement
• Cons: You need a public endpoint, retries/idempotency handling, and security verification

Key practices:

• Verify signatures/HMAC
• Make handlers idempotent
• Respond quickly and process asynchronously
• Retry safely with deduplication

2. Event streams / message brokers

Publish events to a broker like Kafka, RabbitMQ, SNS/SQS, Pub/Sub, or EventBridge.

• Best for: Internal systems, high volume, decoupled architectures
• Pros: Durable, scalable, replayable, good fan-out
• Cons: More infrastructure and operational complexity

Key practices:

• Define a stable event schema
• Use correlation IDs and versioning
• Design consumers to be idempotent
• Separate transport from business logic

3. Change-data capture (CDC)

Emit events from database changes, often via Debezium or similar tooling.

• Best for: Legacy apps or when you can’t modify the application layer much
• Pros: Captures changes reliably at the data layer
• Cons: Harder to map DB changes to meaningful business events

4. Server-sent events / WebSockets

Useful when you need real-time push to clients, not just backend integrations.

• Best for: UI/live updates
• Pros: Low latency
• Cons: Not ideal for durable integration workflows

---

Recommended pattern for most integrations

A strong default architecture is:

1. Source emits an event (webhook or broker)
2. Ingestion layer validates and normalizes
3. Queue/bus buffers the event
4. Workers process asynchronously
5. Store delivery status and dedupe keys
6. Retry with backoff
7. Expose dead-letter handling for failures

This avoids polling while keeping the system resilient.

---

If you’re choosing one approach

• External SaaS integration: webhooks
• Microservices/internal systems: event bus or message broker
• Database-centric legacy system: CDC
• Live frontend updates: WebSockets/SSE

---

Common pitfalls to avoid

• Relying on “exactly once” delivery assumptions
• Doing heavy work directly in webhook handlers
• No idempotency keys
• No retry strategy
• No event versioning
• Tight coupling between producer and consumer

If you want, I can also give you:

1. a reference architecture diagram,
2. a comparison of webhook vs Kafka vs queue, or
3. a step-by-step implementation pattern for a specific stack.