Prompt
How do I set up an API for batch transcoding a video library?
Latest observation
A good way to set this up is to design it as a job-based API rather than a single “convert this file now” endpoint. Batch transcoding usually needs: uploads, job queueing, progress tracking, retries, storage management, and callbacks when complete.
1) Define the workflow
Typical flow:
- Client uploads or registers one or more source videos.
- API creates a transcoding job for each file or a batch.
- Workers pick jobs from a queue and run ffmpeg or a managed transcoding service.
- API exposes job status/progress.
- Output files are stored in object storage and made available for download/streaming.
- Optional webhook notifies the client when a job finishes or fails.
2) Recommended architecture
Core components
- API service: handles auth, job creation, status queries
- Queue: SQS, RabbitMQ, Redis queue, Kafka, etc.
- Worker pool: transcoding workers with ffmpeg
- Database: stores jobs, assets, presets, status, logs
- Object storage: S3, GCS, Azure Blob, or similar
- CDN: for serving finished outputs
Simple stack example
- Backend: Node.js / Python / Go
- Queue: Redis + BullMQ / Celery / Sidekiq
- DB: Postgres
- Storage: S3
- Worker: Docker container with ffmpeg
3) Design the API endpoints
Create a batch
POST /v1/batches
Request:
{
"name": "weekly-library-transcode",
"source_ids": ["vid_001", "vid_002", "vid_003"],
"preset": "h264_1080p",
"webhook_url": "https://client.example.com/transcode-events"
}
Response:
{
"batch_id": "batch_123",
"status": "queued",
"job_count": 3
}
Create a single job
POST /v1/transcode-jobs
{
"source_url": "s3://mybucket/raw/video.mp4",
"output_formats": ["mp4", "hls"],
"preset": "h264_1080p",
"priority": 5
}
Response:
{
"job_id": "job_456",
"status": "queued"
}
Check job status
GET /v1/transcode-jobs/{job_id}
Response:
{
"job_id": "job_456",
"status": "processing",
"progress": 42,
"started_at": "2026-11-01T12:00:00Z",
"input": {
"duration_seconds": 3600
},
"output": null,
"error": null
}
List jobs in a batch
GET /v1/batches/{batch_id}/jobs
Cancel a job
POST /v1/transcode-jobs/{job_id}/cancel
Webhook for completion
Send events like:
{
"event": "transcode.completed",
"job_id": "job_456",
"output_url": "https://cdn.example.com/video_1080p.mp4"
}
4) Model your data
Jobs table
idbatch_idsource_asset_idstatus(queued,processing,completed,failed,canceled)presetpriorityprogressattempt_counterror_messagecreated_atupdated_atstarted_atcompleted_at
Assets table
idsource_urlstorage_keydurationcodecresolutionsize_byteschecksum
Outputs table
idjob_idformatstorage_keycdn_urlmanifest_urlfor HLS/DASH
5) Use presets for consistency
Instead of letting users pass arbitrary ffmpeg flags, define presets:
h264_1080ph264_720phevc_4khls_1080paudio_only_aac
Example preset config:
{
"name": "h264_1080p",
"video_codec": "libx264",
"audio_codec": "aac",
"resolution": "1920x1080",
"bitrate": "5000k",
"container": "mp4"
}
This keeps jobs predictable and safer.
6) Worker execution pattern
Worker steps:
- Pull job from queue
- Mark
processing - Download source from storage
- Run ffmpeg with preset
- Upload output(s)
- Update job status and progress
- Emit webhook or event
Example ffmpeg command:
ffmpeg -i input.mp4 -c:v libx264 -preset slow -b:v 5M -c:a aac -movflags +faststart output.mp4
For HLS:
ffmpeg -i input.mp4 \
-c:v libx264 -c:a aac \
-hls_time 6 -hls_playlist_type vod \
-f hls output.m3u8
7) Make it resilient
Important features
- Idempotency keys for job creation
- Retries with exponential backoff
- Dead-letter queue for failures
- Job locks to avoid double processing
- Timeouts for hung transcodes
- Chunked uploads for large files
- Rate limiting per tenant/user
- Validation of input media before queueing
8) Security considerations
- Require API authentication: API keys, OAuth, or JWT
- Use signed URLs for uploads/downloads
- Validate file types and size limits
- Sandbox workers
- Avoid exposing raw ffmpeg arguments unless heavily restricted
- Scan uploads if needed
- Encrypt storage at rest and in transit
9) Scaling strategy
For a large video library:
- Split into batches
- Use autoscaling worker containers
- Prioritize jobs by tenant or SLA
- Separate queues by job type:
- thumbnails
- transcode
- packaging
- caption burn-in
- Use media-aware instance types if encoding is CPU/GPU heavy
- Consider managed services if you don’t want to operate ffmpeg infrastructure
10) If you want to build faster, consider managed services
Instead of managing workers yourself, you can use:
- AWS Elemental MediaConvert
- Google Transcoder API
- Azure Media Services alternatives / partner services
- Cloudinary or Mux for higher-level video workflows
This can simplify scaling, monitoring, and format support.
11) Minimal implementation plan
If you want a practical first version:
- Build
POST /jobsandGET /jobs/{id} - Store source files in S3
- Use a Redis queue
- Run ffmpeg in Docker workers
- Store outputs back in S3
- Add status polling
- Add batch creation
- Add webhook notifications
- Add retries and dead-letter handling
- Add presets and auth
12) Suggested tech stack example
Option A: Python
- FastAPI
- Celery
- Redis
- Postgres
- ffmpeg
- boto3 for S3
Option B: Node.js
- NestJS or Express
- BullMQ
- Redis
- Postgres
- ffmpeg via child process
- AWS SDK
Option C: Go
- Gin/Fiber
- Asynq or custom queue
- Postgres
- ffmpeg subprocesses
If you want, I can also provide:
- a full OpenAPI spec for this API,
- a database schema, or
- a reference implementation in Python/FastAPI or Node.js.