How do I set up a data versioning tool for reproducible training on large image datasets with active learning?

A good way to do this is to combine:

• DVC (or lakeFS) for data/version tracking
• Git for code/config
• object storage (S3/GCS/Azure Blob/local NAS) for the image files
• experiment tracking (MLflow/W&B) for runs, metrics, and model artifacts

For large image datasets + active learning, DVC is a common choice because it handles large files efficiently by storing only metadata in Git and the actual data in remote storage.

---

Recommended setup

1) Organize your project

Example structure:

project/
├─ data/
│  ├─ raw/
│  ├─ interim/
│  └─ splits/
├─ models/
├─ src/
├─ configs/
├─ dvc.yaml
├─ params.yaml
└─ .gitignore

Keep:

• raw/ = original images
• interim/ = transformed/cached outputs
• splits/ = train/val/test or AL pool/query sets
• annotations/labels in versioned files too

---

2) Initialize Git and DVC

git init
dvc init
git add .dvc .gitignore
git commit -m "Initialize DVC"

Add a remote for data storage:

dvc remote add -d storage s3://my-bucket/my-project
# or gs://..., azure://..., or local path
dvc remote modify storage access_key_id ...
dvc remote modify storage secret_access_key ...

---

3) Track the dataset with DVC

If you have a folder of images:

dvc add data/raw/images
git add data/raw/images.dvc .gitignore
git commit -m "Track raw images with DVC"

If labels are in CSV/JSON, version those too:

dvc add data/raw/labels.csv
git add data/raw/labels.csv.dvc
git commit -m "Track labels"

Then push actual data to remote:

dvc push

Now Git stores the pointer; DVC remote stores the images.

---

Reproducible training pipeline

Use dvc.yaml to define stages like:

• prepare
• split
• train
• evaluate

Example:

stages:
  split:
    cmd: python src/split.py --input data/raw --output data/splits
    deps:
      - src/split.py
      - data/raw/images.dvc
      - data/raw/labels.csv.dvc
    outs:
      - data/splits

  train:
    cmd: python src/train.py --data data/splits/train --labels data/splits/train_labels.csv --model models/model.pt
    deps:
      - src/train.py
      - data/splits
    params:
      - train.epochs
      - train.lr
    outs:
      - models/model.pt
    metrics:
      - metrics.json

Then run:

dvc repro

This makes training reproducible because the exact code, data version, and parameters are all pinned.

---

Active learning workflow

For active learning, you usually have:

• labeled set
• unlabeled pool
• query-selected samples
• newly labeled set

Version each of these as separate DVC-tracked artifacts.

Suggested approach

Store AL state in versioned files, for example:

data/al/
├─ labeled/
├─ unlabeled/
├─ query/
├─ annotations_round_01.csv
├─ annotations_round_02.csv
└─ selection_state.json

Each AL round:

1. Start from a known dataset snapshot
2. Train model
3. Score unlabeled pool
4. Select top uncertain samples
5. Save selected image IDs into query_round_N.csv
6. Human annotates them
7. Merge new labels into labeled set
8. Commit new DVC/Git state

This lets you reproduce exactly:

• which images were labeled at each round
• which model chose them
• what parameters were used

---

Best practice: version IDs, not only files

For very large datasets, avoid duplicating data unnecessarily. Keep stable image IDs and version:

• manifests (csv/json)
• label files
• split files
• selection files

Example manifest:

image_id,path,label,round
0001,img_0001.jpg,cat,1
0002,img_0002.jpg,dog,1

This is especially useful when active learning changes only labels or membership, not the underlying image files.

---

Make runs fully reproducible

To reproduce training exactly, version these together:

• code in Git
• data pointers in DVC
• params in params.yaml
• random seeds
• environment file (requirements.txt, conda.yaml, poetry.lock, or Dockerfile)
• model metrics/artifacts

Example params.yaml:

train:
  epochs: 20
  lr: 0.0001
  seed: 42

Then your train code should read these values, not hardcode them.

---

Handling huge image datasets efficiently

For scale, consider:

• object storage for data remote
• image manifests instead of copying images around
• DataLoader caching
• webdataset/tar shards if training throughput matters
• content-addressed storage via DVC or lakeFS
• partitioning by round/class/date

If your dataset is extremely large and multi-user, lakeFS may be better than DVC because it provides Git-like branching/commits directly on object storage. But DVC is usually easier to adopt for ML teams.

---

Example workflow for each active learning round

# After selecting new samples
python src/select_uncertain.py --model models/model.pt --pool data/al/unlabeled --out data/al/query_round_03.csv

# After annotation
python src/merge_labels.py --base data/al/labeled.csv --new data/al/query_round_03_labeled.csv --out data/al/labeled_round_03.csv

# Track new dataset state
dvc add data/al/labeled_round_03.csv data/al/unlabeled_round_03.csv
git add data/al/labeled_round_03.csv.dvc data/al/unlabeled_round_03.csv.dvc
git commit -m "Active learning round 3 dataset update"
dvc push

# Re-train reproducibly
dvc repro

---

What to avoid

• Don’t store huge image files directly in Git
• Don’t rely on “latest” folders without versioning
• Don’t mutate data in place without recording the snapshot
• Don’t keep active learning selections only in ad hoc notebooks

---

Quick recommendation

If you want a practical, common setup:

• DVC + S3 + Git + MLflow
• version:
  • raw images
  • labels/annotations
  • splits
  • active learning query lists
  • training params
  • model metrics

---

If you want, I can give you:

1. a minimal DVC + active learning repo template, or
2. an example for S3/GCS + PyTorch training pipeline.