Need dataset API with bulk download and filters

Here’s a solid dataset API design that supports both bulk download and filters.

Core requirements

• List datasets
• Filter datasets by metadata
• Download a single dataset
• Bulk download multiple datasets, ideally as a ZIP or async export
• Support pagination and sorting

---

Suggested endpoints

1) List / search datasets

GET /api/v1/datasets

Query params

• q — full-text search
• category
• source
• owner
• tag
• created_after
• created_before
• updated_after
• updated_before
• format — e.g. csv, parquet, json
• public — true/false
• sort — e.g. created_at, name
• order — asc|desc
• page
• page_size

Example

GET /api/v1/datasets?category=finance&tag=transactions&sort=updated_at&order=desc&page=1&page_size=20

Response

{
  "data": [
    {
      "id": "ds_001",
      "name": "Finance Transactions 2024",
      "description": "Monthly transaction records",
      "category": "finance",
      "tags": ["transactions", "monthly"],
      "format": "parquet",
      "size_bytes": 123456789,
      "created_at": "2024-01-01T00:00:00Z",
      "updated_at": "2024-06-01T00:00:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "page_size": 20,
    "total": 245
  }
}

---

2) Get dataset details

GET /api/v1/datasets/{dataset_id}

Returns metadata, schema, available files, and download links.

---

3) Download one dataset

GET /api/v1/datasets/{dataset_id}/download

Query params

• format — optional conversion format
• version — optional version id

Response

• Direct file download, or
• A signed URL if files are stored in object storage

---

4) Bulk download datasets

You have two good options:

Option A: Synchronous ZIP for small batches

POST /api/v1/datasets/bulk-download

Body

{
  "dataset_ids": ["ds_001", "ds_002", "ds_003"],
  "format": "zip"
}

Response

• 200 OK with a ZIP file if small enough

Option B: Async export job for large batches

POST /api/v1/datasets/bulk-download-jobs

Body

{
  "filter": {
    "category": "finance",
    "tag": "transactions",
    "updated_after": "2024-01-01T00:00:00Z"
  },
  "format": "zip"
}

Response

{
  "job_id": "job_123",
  "status": "queued"
}

Then:

• GET /api/v1/download-jobs/{job_id}
• GET /api/v1/download-jobs/{job_id}/download

This is better for large exports.

---

Recommended filter model

Support filters either as query params or a JSON body for bulk export.

Query param filters

Best for listing/searching:

• exact matches: category=finance
• multi-select: tag=a&tag=b
• ranges: created_after, created_before

JSON filters for bulk download

{
  "filter": {
    "category": ["finance", "sales"],
    "tags": ["transactions", "2024"],
    "size_bytes": { "gte": 1000000, "lte": 500000000 }
  }
}

---

Nice-to-have features

• Cursor-based pagination for large result sets
• Presigned download URLs
• Rate limiting
• Job progress tracking for bulk export
• Include/exclude fields in listing responses
• Saved filters

---

Minimal OpenAPI shape

paths:
  /api/v1/datasets:
    get:
      summary: List datasets
  /api/v1/datasets/{dataset_id}:
    get:
      summary: Get dataset metadata
  /api/v1/datasets/{dataset_id}/download:
    get:
      summary: Download dataset
  /api/v1/datasets/bulk-download-jobs:
    post:
      summary: Create bulk download job
  /api/v1/download-jobs/{job_id}:
    get:
      summary: Check job status
  /api/v1/download-jobs/{job_id}/download:
    get:
      summary: Download completed bulk export

---

Best practice recommendation

If your datasets can be large, use:

• GET /datasets for filtering and discovery
• POST /bulk-download-jobs for filtered bulk export
• GET /download-jobs/{id}/download for file retrieval

This scales much better than trying to stream huge ZIPs immediately.

If you want, I can also provide:

1. a full OpenAPI spec,
2. a database schema, or
3. an implementation example in FastAPI/Express.