Prompt
How can I support webhooks, scheduled sync, and manual re-syncs in one integration system?
Latest observation
A good way to support webhooks, scheduled sync, and manual re-syncs in one integration system is to treat them all as different triggers for the same sync pipeline.
Core idea
Build a single ingestion/sync flow with:
-
Trigger layer
- Webhook received
- Scheduled job runs
- User clicks “Sync now”
-
Job normalization
- Convert every trigger into a standard sync request:
integration_idaccount_idsync_type(webhook,scheduled,manual)cursor/sincepriorityrequested_byreason
- Convert every trigger into a standard sync request:
-
Queue + worker
- Put all sync requests onto a queue
- Workers process them through the same sync engine
-
Sync engine
- Fetch remote data
- Apply deduplication
- Upsert into your DB
- Update cursor/state
- Emit downstream events
Recommended architecture
1) Separate trigger handling from sync execution
Each trigger should be lightweight:
-
Webhook handler
- Validate signature
- Parse payload
- Enqueue a sync job immediately
- Return 200 quickly
-
Scheduler
- Periodically enqueues jobs for accounts due for polling
- Can do incremental syncs based on cursor
-
Manual resync
- User action enqueues a job with higher priority
- Optionally allows a “full resync” or “since timestamp”
This keeps your system consistent and avoids three different sync implementations.
2) Use a unified sync job schema
Example:
{
"job_id": "abc123",
"integration": "stripe",
"account_id": "acct_1",
"trigger": "webhook",
"sync_mode": "incremental",
"priority": "high",
"cursor": "2026-10-24T10:00:00Z",
"force_full": false,
"requested_by": "system",
"metadata": {
"event_id": "evt_789"
}
}
For manual re-sync:
{
"integration": "stripe",
"account_id": "acct_1",
"trigger": "manual",
"sync_mode": "full",
"priority": "urgent",
"force_full": true,
"requested_by": "user_42"
}
For scheduled sync:
{
"integration": "stripe",
"account_id": "acct_1",
"trigger": "scheduled",
"sync_mode": "incremental",
"priority": "normal"
}
3) Make sync idempotent
This is critical because webhooks, scheduled jobs, and manual retries can overlap.
Use:
- Event deduplication
- Store webhook event IDs and ignore duplicates
- Entity upserts
- Use remote object IDs as unique keys
- Cursor/state versioning
- Only advance cursors after successful processing
- Job deduplication
- Prevent two identical jobs from running at the same time
Example:
- Webhook arrives for order
123 - Scheduler also polls order
123 - Manual resync runs at same time
All three should safely converge to one correct record state.
4) Use per-account sync state
Store sync state per integration account:
last_successful_sync_atlast_cursorlast_webhook_atsync_in_progresssync_versionbackoff_untillast_error
This lets your system decide:
- What to fetch
- Whether a scheduled sync is needed
- Whether a webhook should trigger incremental fetch
- Whether a manual resync should bypass normal throttling
5) Decide how each trigger behaves
Webhooks
Best for near-real-time updates.
Typical behavior:
- Webhook arrives with minimal payload
- Enqueue incremental sync for the affected account/entity
- Use the webhook as a signal, not as the full source of truth
Why:
- Webhooks can be partial, out of order, duplicated, or missing fields
Scheduled sync
Best for reconciliation and backstop.
Typical behavior:
- Poll remote API every N minutes/hours
- Catch missed webhooks
- Reconcile state
- Heal inconsistencies
Manual re-sync
Best for user-initiated repair or onboarding.
Typical behavior:
- Can re-sync one entity, one account, or all data
- Usually higher priority
- Can support:
- incremental retry
- full backfill
- date-range replay
6) Handle priority and concurrency
Use job priority levels:
urgentfor manual synchighfor webhook-driven syncnormalfor scheduled sync
Also limit concurrency:
- per integration
- per account
- optionally per tenant
This prevents one noisy account from overwhelming the system.
7) Build a sync orchestrator
A small orchestrator can determine the right action:
- If webhook event refers to a specific object:
- sync just that object or its parent resource
- If scheduled and cursor exists:
- fetch since cursor
- If manual full resync:
- queue backfill chunks
This keeps trigger logic simple while allowing integration-specific behavior.
8) Track sync outcomes
Record:
- job status: queued / running / succeeded / failed / partial
- items fetched
- items updated
- cursor advanced
- retry count
- error details
- source trigger
This helps with debugging and auditing.
9) Practical pattern to follow
A solid pattern is:
- All triggers enqueue jobs
- All jobs go through one worker pipeline
- All writes are idempotent
- All state updates happen after successful processing
- Scheduled sync acts as reconciliation
- Manual sync can override normal timing
Simple mental model
Think of it like this:
- Webhook = “Something changed”
- Scheduler = “Make sure nothing was missed”
- Manual resync = “Fix this now”
But all three should end up as the same operation:
“Run sync for this account using the best available cursor and rules.”
If you want a concrete implementation approach
A typical stack might be:
- API service for webhook + manual requests
- Scheduler cron / job runner
- Queue like SQS/RabbitMQ/Kafka
- Worker service for sync execution
- DB tables for:
- sync_jobs
- sync_state
- webhook_events
- integration_entities
If you want, I can also sketch:
- a database schema for this,
- a job state machine, or
- a reference implementation in Node/Python.