How do I build a real-time integration between two systems

Building a real-time integration between two systems usually comes down to choosing the right communication pattern, defining the contract, and making it reliable and observable.

1. Clarify what “real-time” means

Before designing anything, define:

• Latency target: milliseconds, seconds, or near-real-time?
• Direction: one-way sync or two-way?
• Trigger type: event-driven, polling, or API call on demand?
• Volume: a few events/minute or thousands/sec?
• Consistency needs: can data be eventually consistent, or must it be immediate?

2. Choose an integration pattern

A. Event-driven integration

Best when one system publishes changes and the other reacts.

Typical flow:

1. System A emits an event like OrderCreated
2. A message broker or event bus carries it
3. System B consumes it and updates its state

Tools/patterns:

• Kafka
• RabbitMQ
• AWS EventBridge / SNS + SQS
• Azure Service Bus / Event Grid
• Google Pub/Sub

Pros:

• Fast
• Decoupled
• Scales well

Cons:

• More moving parts
• Requires careful handling of duplicates and ordering

B. API-driven synchronous integration

Best when System B needs an immediate response from System A.

Typical flow:

1. System B calls System A’s API
2. System A returns data or confirmation immediately

Pros:

• Simple
• Easy to reason about

Cons:

• Tighter coupling
• Failures/latency in one system affect the other
• Less resilient at scale

C. Hybrid approach

Very common:

• Use APIs for queries or immediate actions
• Use events for propagation of changes

This is often the best real-world design.

3. Define the data contract

Agree on:

• Event names and payload structure
• Field types and required fields
• Versioning strategy
• Error codes / retry behavior

Example event:

{
  "eventType": "OrderCreated",
  "eventId": "c7b0f2e1-1d2d-4f68-9d1b-3f6a9b0c9a11",
  "timestamp": "2026-10-09T12:00:00Z",
  "sourceSystem": "SystemA",
  "data": {
    "orderId": "12345",
    "customerId": "987",
    "amount": 49.99
  }
}

4. Make it reliable

Real-time integrations fail unless you design for failure.

Key practices:

• Idempotency: processing the same message twice should not break anything
• Retries with backoff: transient failures should retry automatically
• Dead-letter queue (DLQ): poison messages should be isolated
• Acknowledgements: only mark messages handled after successful processing
• Ordering strategy: if order matters, partition by key or process sequentially
• Timeouts and circuit breakers for API calls

5. Handle data consistency

Common approaches:

• Eventual consistency: accept small delays
• Outbox pattern: write business data and integration event in one transaction, then publish asynchronously
• Saga pattern: coordinate multi-step workflows across systems

6. Secure the integration

Use:

• Authentication: OAuth2, mTLS, API keys, IAM roles
• Authorization: least privilege
• Encryption in transit: TLS
• Encryption at rest for brokers/queues
• Secret management: Vault, AWS Secrets Manager, Azure Key Vault

7. Add observability

You need to know what happened to each message:

• Correlation IDs
• Structured logs
• Metrics: throughput, latency, retries, failures
• Tracing: distributed tracing if APIs/services are involved
• Alerting: queue backlog, error spikes, DLQ growth

8. Test thoroughly

Test:

• Happy path
• Duplicate events
• Out-of-order delivery
• Partial outages
• Timeouts
• Schema changes
• High load

9. Example architecture

A common real-time setup:

• System A writes business data
• An outbox table stores an event
• A publisher sends events to Kafka
• System B consumes the event
• System B updates its own database
• Failures go to a DLQ
• Monitoring tracks lag and errors

10. Practical rule of thumb

• If you need immediate response: use API calls
• If you need scalable real-time sync: use events/messages
• If you need both: use a hybrid design

If you want, I can sketch a concrete architecture for your exact case, for example:

• Salesforce ↔ ERP
• Web app ↔ backend service
• Database ↔ database
• On-prem ↔ cloud