Prompt
How do I process image uploads asynchronously?
Latest observation
To process image uploads asynchronously, the usual pattern is:
- Accept the upload quickly
- Store the file
- Queue a background job
- Return a fast response to the client
- Process the image in a worker
- Store/update the result
- Let the client poll or subscribe for status
Typical flow
1) Upload endpoint
Your API receives the image and saves it to durable storage:
- local disk
- S3 / GCS / Azure Blob
- a database only if the files are tiny, though that’s usually not ideal
Then it creates a job record like:
job_idstatus = queuedinput_pathcreated_at
2) Queue a background task
Push a message to a queue such as:
- Celery + Redis/RabbitMQ
- Sidekiq
- Resque
- AWS SQS + worker
- BullMQ / RabbitMQ / Kafka, depending on your stack
The queue message should contain the file reference, not the whole file.
3) Return immediately
Respond with something like:
{
"job_id": "12345",
"status": "queued"
}
This keeps the upload endpoint responsive.
4) Worker processes the image
A separate worker process picks up the job and does things like:
- resize / thumbnail generation
- format conversion
- OCR / ML inference
- virus scanning
- EXIF stripping
- optimization/compression
Then it writes the output back to storage and updates the job status:
processingcompletedfailed
5) Client checks status
Provide a status endpoint:
GET /jobs/12345
Response:
{
"job_id": "12345",
"status": "completed",
"result_url": "https://..."
}
You can also use:
- WebSockets
- Server-Sent Events
- webhooks
- push notifications
if you want live updates.
Example architecture
Client → Upload API → Object Storage → Queue → Worker → Result Storage/DB
Best practices
Don’t process large images in the request thread
Avoid blocking the web server with CPU-heavy work.
Validate uploads early
Check:
- file type
- file size
- image dimensions
- corrupted files
Use unique job IDs
Track every upload and its processing state.
Make jobs idempotent
Workers should safely handle retries without duplicating output.
Handle failures
Store error details so clients can see what went wrong.
Secure the upload
- sanitize filenames
- scan for malware if needed
- restrict MIME types
- strip metadata if sensitive
Simple pseudo-example
Upload handler
def upload_image(file):
path = save_to_storage(file)
job_id = create_job_record(path, status="queued")
enqueue_job(job_id)
return {"job_id": job_id, "status": "queued"}
Worker
def process_image_job(job_id):
job = get_job(job_id)
update_job(job_id, status="processing")
try:
image = load_image(job.input_path)
result = resize(image)
output_path = save_to_storage(result)
update_job(job_id, status="completed", result_path=output_path)
except Exception as e:
update_job(job_id, status="failed", error=str(e))
If you want, I can show you:
- a Node.js example with BullMQ
- a Python/FastAPI example with Celery
- a Django example
- a Java/Spring example
- or a cloud-native approach using S3 + SQS + Lambda