Prompt
How do I add moderation to an upload flow with an API?
Latest observation
A good moderation pattern for an upload flow is:
- Accept the upload
- Store it in a quarantined/private location
- Run moderation asynchronously
- Only publish/activate the asset if it passes
- Reject, redact, or flag it if it fails
That keeps your app fast and avoids exposing unreviewed content.
Common architecture
1) Client uploads file
Your frontend calls your API to create an upload.
Example:
POST /uploads- API returns:
upload_id- a pre-signed URL or upload endpoint
- an initial status like
pending
2) Store as “pending”
Save metadata in your DB:
iduser_idfile_urlstatus = pendingmoderation_status = queued- timestamps
Do not make it publicly accessible yet.
3) Trigger moderation
After upload completes, enqueue a background job to inspect the asset.
Moderation can include:
- image/video safety checks
- text checks if the file contains text/metadata/captions
- metadata validation
- virus scanning
- OCR + text moderation for images/screenshots
- audio transcription + moderation for audio/video
4) Update result
Set one of these statuses:
approvedrejectedneeds_reviewblocked
Then:
- if approved: move to public bucket / mark visible
- if rejected: keep private or delete
- if needs review: route to human moderation
Recommended API design
Create upload
POST /uploads
Response:
{
"upload_id": "up_123",
"upload_url": "https://storage.example.com/presigned...",
"status": "pending"
}
Check status
GET /uploads/up_123
Response:
{
"upload_id": "up_123",
"status": "processing",
"moderation_status": "queued"
}
Later:
{
"upload_id": "up_123",
"status": "approved",
"moderation_status": "passed"
}
Or:
{
"upload_id": "up_123",
"status": "rejected",
"moderation_status": "failed",
"reason": "inappropriate content"
}
Optional webhook callback
If your system supports it, notify the client or another service:
POST /webhooks/upload-moderated
Best practices
Don’t trust client-side moderation
Client checks are useful for UX, but the server must enforce moderation.
Keep unverified content private
Use:
- private buckets
- signed URLs
- internal-only processing endpoints
Make moderation asynchronous
Don’t block the upload request waiting for a model to finish. Return quickly and process in the background.
Support manual review
Some content will be ambiguous. Add a human review queue.
Log decisions
Store:
- model/version used
- scores/confidence
- moderation reason codes
- reviewer IDs if manual
This helps with debugging and appeals.
Handle retries safely
Use idempotency keys so the same upload isn’t moderated multiple times accidentally.
Example flow
- User uploads profile image
- API stores it privately
- Moderation job checks:
- nudity
- violence
- hateful symbols
- If passed, image becomes public
- If failed, user sees a rejection message and can upload another
If you want to use a moderation service
Many teams use:
- OpenAI moderation for text
- image/video moderation providers
- cloud-provider scanning tools
- custom ML models
- combined pipelines
A common pattern is:
- scan for malware first
- run automated moderation second
- human review for borderline cases
Minimal implementation checklist
- Uploads land in private storage
- DB row created with
pending - Background job moderates content
- Status updates stored
- Public access only after approval
- Rejection and appeal path
- Logging and audit trail
If you want, I can also give you:
- a Node.js/Express example,
- a Python/FastAPI example, or
- a full architecture diagram for this flow.
Brands