I'm building a marketplace with video uploads and need automatic transcode jobs plus webhooks

A solid pattern for this is:

Recommended flow

1. User uploads video

   • Upload directly to object storage (S3, GCS, etc.) using a signed URL.
   • Store a videos record in your DB with status like uploaded.

2. Create a transcode job

   • When upload completes, enqueue a job in a queue or message bus.
   • Job contains:
     • video_id
     • source file URL / object key
     • output profiles (1080p, 720p, thumbnail, etc.)

3. Worker processes transcoding

   • A background worker consumes the job.
   • It invokes your transcoding service/engine:
     • AWS MediaConvert
     • Mux
     • Cloudflare Stream
     • FFmpeg worker on Kubernetes/EC2
   • Update DB status as it progresses: processing, ready, failed.

4. Webhook notifications

   • When processing state changes, emit a webhook event to your app or third-party consumers.
   • Webhook events might include:
     • video.uploaded
     • video.transcode.started
     • video.transcode.progress
     • video.transcode.completed
     • video.transcode.failed

---

Core components

1. Upload service

• Handles auth and signed upload URLs.
• Returns upload URL + object key.
• Confirms completion via client callback or storage event.

2. Job queue

• Use:
  • SQS + worker
  • Redis queue (BullMQ/Celery/RQ)
  • Kafka/RabbitMQ if you need scale/eventing
• Makes transcode processing async and retryable.

3. Transcode worker

• Receives job
• Downloads input or streams from storage
• Produces outputs
• Writes metadata back to DB
• Emits events/webhooks

4. Webhook system

• Store subscriber endpoints in DB.
• Sign payloads with HMAC.
• Retry delivery with exponential backoff.
• Store delivery logs and last attempt status.

---

Suggested database tables

videos

• id
• owner_id
• source_key
• status
• duration
• metadata
• created_at
• updated_at

transcode_jobs

• id
• video_id
• status
• provider_job_id
• input_key
• output_manifest_key
• error_message
• created_at
• updated_at

webhook_subscriptions

• id
• owner_id
• url
• secret
• event_types
• enabled
• created_at

webhook_deliveries

• id
• subscription_id
• event_type
• payload
• status
• attempt_count
• last_error
• next_retry_at
• created_at

---

Webhook payload example

{
  "event": "video.transcode.completed",
  "id": "evt_12345",
  "created_at": "2026-10-01T12:34:56Z",
  "data": {
    "video_id": "vid_789",
    "status": "ready",
    "renditions": [
      { "quality": "1080p", "url": "https://cdn.example.com/..." },
      { "quality": "720p", "url": "https://cdn.example.com/..." }
    ]
  }
}

Signature header

• X-Webhook-Signature: t=timestamp,v1=hmac_sha256(...)

---

Best practices

• Idempotency: make job processing and webhook delivery safe to retry.
• Retries: retry transient failures, dead-letter permanent ones.
• Progress updates: if you need them, emit events from worker/provider callbacks.
• Security:
  • signed uploads
  • webhook signing
  • allowlist admin-only endpoints if internal
• Observability:
  • job logs
  • webhook delivery logs
  • metrics for processing time/failure rate

---

If you want a simple stack

A practical setup for many marketplaces:

• Frontend: direct-to-S3 upload
• API: Node/Go/Python
• Queue: SQS or Redis/BullMQ
• Transcoding: AWS MediaConvert or Mux
• Webhooks: your own delivery service with retries/signing

---

If you want, I can also sketch:

1. a full architecture diagram,
2. API endpoints for upload/transcode/webhooks, or
3. a sample implementation in Node.js/Python.