What should I use for batch data delivery API?

For a batch data delivery API, a good default choice is:

Recommended approach

Use a REST API with asynchronous batch upload

• POST /batches to submit a batch
• return a batch_id
• process it asynchronously
• expose GET /batches/{batch_id} for status/result

This is usually the best fit when:

• the payload is large
• processing takes time
• you want retries and idempotency
• you need clear status tracking

Common options

1. REST + async processing

Best for most batch delivery use cases.

• Simple for clients
• Works well with JSON or NDJSON
• Easy to document and test
• Can support pagination, status polling, and callbacks/webhooks

2. Message queue / event-based delivery

Best if you have high volume or internal service-to-service delivery.

• Kafka, RabbitMQ, SQS, Pub/Sub
• More scalable and resilient
• Not ideal if external clients need a simple API

3. File-based upload API

Best for very large batches.

• Upload CSV/JSONL/Parquet to object storage
• Send a pointer/manifest via API
• Good for millions of records or multi-GB transfers

4. GraphQL

Usually not ideal for batch delivery.

• Better for query flexibility than bulk ingestion
• More complexity for uploads and async workflows

Practical recommendation

If you’re designing this from scratch, use:

• HTTP REST
• JSON Lines (NDJSON) or CSV for record batches
• async job model
• idempotency key
• webhook or polling for completion
• object storage presigned upload if batches are big

Example shape

• POST /batch-deliveries
• GET /batch-deliveries/{id}
• GET /batch-deliveries/{id}/errors
• optional: POST /batch-deliveries/{id}/cancel

If you want the simplest answer

Use REST over HTTP with asynchronous batch jobs and JSON/CSV payloads.

If you tell me:

1. batch size,
2. whether clients are external or internal,
3. latency requirements,
4. file type/format,

I can recommend a more specific API design.