What should I use for a data API with bulk export?

For a data API with bulk export, the best choice depends on how big the exports are, how often they run, and whether clients need the data immediately or can fetch it asynchronously.

Good default pattern

Use:

• REST API for normal queries and smaller responses
• Asynchronous bulk export jobs for large datasets
• Export files in CSV, JSONL, or Parquet
• Store exports in object storage like S3, GCS, or Azure Blob
• Return a job ID and let the client poll or receive a webhook when ready

Recommended architecture

1. Client requests export:
   • POST /exports
2. API creates a background job:
   • filters data
   • writes file to storage
3. Client checks status:
   • GET /exports/{id}
4. When complete, client downloads via:
   • signed URL or temporary authenticated link

Format choices

• CSV: best for spreadsheets and simple tabular data
• JSONL: good for API-native consumption and streaming
• Parquet: best for analytics and very large exports
• ZIP: useful if exporting multiple files

If you need very large-scale / analytics use

Consider:

• GraphQL only if clients need flexible field selection, but it’s not ideal for bulk export
• gRPC for service-to-service transfers, not usually for external bulk export
• A data warehouse/ETL pipeline if exports are frequent and huge

Practical recommendation

If you’re building a general-purpose external data API:

• Use REST
• Support paged reads
• Add async bulk export endpoints
• Export to S3 + signed URLs
• Prefer CSV/JSONL, and offer Parquet for power users

If you want, I can suggest a concrete API design with endpoints and response examples.