A practical way to set up ground-truth dataset management for corner-case curation in autonomy research is to treat it like a versioned data product with clear provenance, annotation standards, and review gates—not just a folder of labels.

Here’s a solid setup.

---

1) Define the dataset’s purpose

Be explicit about what “ground truth” means in your project.

Examples:

• Perception: object boxes, lanes, traffic lights, free space, depth, occupancy
• Prediction: future trajectories, intent, interaction labels
• Planning/control: expert action, safe/unsafe, intervention reason
• Scene understanding: scene graph, weather, road type, risk factors
• Corner cases: rare, ambiguous, high-risk, out-of-distribution events

Write down:

• the task
• the label taxonomy
• the unit of annotation (frame, sequence, event, scene)
• the intended use (training, validation, stress testing, regression)

This prevents label drift later.

---

2) Build a data model around “events,” not just samples

Corner cases are usually temporal and contextual. Don’t manage them only as isolated frames.

Use a hierarchy like:

• Dataset
  • Scenario / drive / route
    • Event
      • Segment / clip
        • Frame / timestamp
          • Annotations

For each event, store:

• unique event ID
• sensor sync info
• geographic/environment context if allowed
• trigger reason for inclusion
• severity score
• novelty/rareness tags
• source system or vehicle
• annotation status
• reviewer and audit history

This makes later filtering much easier.

---

3) Standardize a corner-case taxonomy

Create a shared vocabulary for why a case matters.

Typical tags:

• OOD: out of distribution
• Rare actor: wheelchair, animal, construction vehicle
• Unusual behavior: jaywalking, wrong-way driving, abrupt stop
• Ambiguous perception: occlusion, glare, heavy rain, motion blur
• Complex interaction: merging, negotiation, unprotected turns
• Infrastructure anomaly: missing lane markings, temporary signage
• Safety-critical: near collision, disengagement, hard brake
• Map mismatch: map outdated or inconsistent
• Sensor anomaly: dropout, calibration issue

Use:

• a primary tag
• optional secondary tags
• severity level
• confidence level

Keep the taxonomy versioned.

---

4) Design annotation guidelines before labeling starts

The biggest source of bad ground truth is inconsistent labeling.

Your guidelines should specify:

• exact label definitions
• boundary rules
• occlusion rules
• truncation rules
• how to annotate uncertain cases
• how to handle multi-label scenes
• what to do when sensors disagree
• how to annotate temporal persistence

Include examples of:

• positive/negative cases
• ambiguous cases
• edge cases
• “do not annotate” cases

If possible, put the guidelines in a living document with changelog and version number.

---

5) Separate raw data, curated data, and gold ground truth

Use a clean pipeline:

Raw zone

• immutable sensor logs
• metadata
• no edits to source files

Curated zone

• selected candidate clips
• normalized formats
• extracted frame windows
• candidate annotations

Gold zone

• reviewed, approved ground truth
• final label set
• locked version

This prevents accidental overwrites and lets you trace everything back.

---

6) Use version control for data and labels

Treat annotations like code.

Recommended practices:

• dataset version IDs
• annotation schema version IDs
• label ontology version IDs
• change logs for every update
• immutable releases for experiments

Tools can include:

• DVC or LakeFS for dataset versioning
• object storage for raw media
• Git for schemas, guidelines, and metadata
• SQL/NoSQL metadata store for queries

A good rule: every experiment result should reference the exact dataset version used.

---

7) Add provenance and lineage metadata

For every sample/event, record:

• data source
• capture time
• vehicle/platform
• sensor suite version
• calibration version
• map version
• weather/time-of-day context
• annotation tool and annotator ID
• reviewer ID
• label confidence
• whether it was auto-suggested or manually selected

This is crucial in autonomy because sensor and map differences can explain label differences.

---

8) Create a curation workflow with clear gates

A reliable workflow looks like this:

Step 1: Ingest

Collect logs from vehicles, simulators, or closed-course testing.

Step 2: Detect candidates

Find corner cases using:

• rule-based triggers
• disengagements
• hard braking
• near-miss metrics
• anomaly detectors
• clustering/embedding-based novelty detection
• active learning uncertainty scores

Step 3: Triage

A curator decides:

• keep
• discard
• needs review
• duplicate
• insufficient quality

Step 4: Annotate

Annotators label based on guidelines.

Step 5: QA/review

A second reviewer checks:

• completeness
• consistency
• taxonomy compliance
• temporal continuity
• disagreement resolution

Step 6: Promote to gold

Only fully reviewed data moves to gold.

Step 7: Release

Package dataset version with schema and metadata.

---

9) Build quality control into the process

Ground truth is only useful if it’s trustworthy.

Track:

• inter-annotator agreement
• reviewer disagreement rates
• annotation throughput
• label distribution shifts
• missing-label rates
• duplicate rates
• sensor/metadata completeness
• uncertainty rates

Useful QA checks:

• frame-to-frame temporal consistency
• impossible geometry checks
• box overlap sanity checks
• label ontology validation
• class balance monitoring
• random spot audits
• adversarial audits of corner cases

For corner cases, disagreement is expected more often, so capture it rather than hiding it.

---

10) Make uncertainty first-class

In autonomy, many corner cases are inherently ambiguous.

Instead of forcing a false binary label, support:

• confidence scores
• “uncertain” labels
• multiple plausible interpretations
• provenance of disagreement
• adjudicated final label
• unresolved cases queue

This helps downstream teams understand where the ground truth is weak.

---

11) Manage duplicates and near-duplicates

Corner-case repositories often fill up with repeated scenes.

Store similarity features for:

• visual embeddings
• map/route similarity
• event signatures
• trigger patterns

Use them to:

• deduplicate exact repeats
• cluster near-duplicates
• track families of similar events
• ensure diversity in validation/test sets

This improves coverage and avoids overfitting to repeated incidents.

---

12) Build search and retrieval around metadata

You want to query things like:

• all rainy nighttime pedestrian occlusion events
• all disengagements near unprotected left turns
• all clips with sensor dropout and construction signage
• all severe corner cases from a specific vehicle build

So index by:

• tags
• severity
• environment
• actor types
• route segment
• label version
• model uncertainty
• failure mode

A good metadata layer saves huge amounts of curator time.

---

13) Protect privacy and compliance

Autonomy datasets often include sensitive data.

Plan for:

• face/license plate redaction
• location access controls
• retention policy
• consent/legal review
• access logging
• role-based permissions
• encryption at rest and in transit

If using real-world road data, this can be essential for compliance.

---

14) Create a regression test set from corner cases

Separate your “interesting corner cases” from your “stable benchmark.”

Maintain:

• train set
• validation set
• gold regression set
• challenge set

The regression set should be:

• frozen
• curated
• representative of known failure modes
• used to detect model regressions over time

This is often the most valuable subset for autonomy teams.

---

15) Instrument the system for feedback loops

Once models run against the dataset, feed results back into curation.

Useful signals:

• model uncertainty
• false positives/false negatives
• intervention events
• near-miss scoring
• disagreement between models
• active learning acquisition scores

This turns the dataset into a living system for continuous improvement.

---

A simple reference architecture

Storage

• Object store: raw logs, video, point clouds, images
• Metadata DB: event records, annotations, tags, lineage
• Versioning layer: releases, schema versions, audit trails

Tools

• Annotation tool with temporal support
• Dataset versioning system
• Search UI for filtering corner cases
• QA dashboard
• Label review queue

Roles

• Data engineer: ingestion, storage, versioning
• Curator: candidate selection, deduping, tagging
• Annotator: ground-truth creation
• Reviewer/adjudicator: QA and conflict resolution
• Researcher/modeler: consumes frozen releases

---

Minimal schema to start with

For each event or clip, store:

• event_id
• dataset_version
• source_id
• capture_start_time
• capture_end_time
• location_hash or region ID
• sensor_suite_version
• map_version
• corner_case_tags[]
• severity
• novelty_score
• annotation_status
• annotator_id
• reviewer_id
• label_schema_version
• confidence
• provenance_notes
• release_state

---

Practical rollout plan

If you’re starting from scratch:

Phase 1

• define taxonomy and guidelines
• build metadata schema
• choose storage/versioning stack
• pilot on a small curated set

Phase 2

• add annotation and review workflow
• establish QA metrics
• create first gold release

Phase 3

• automate candidate discovery
• add deduplication and search
• create regression/challenge subsets

Phase 4

• integrate model feedback and active learning
• formalize release process and audits

---

Common mistakes to avoid

• labeling before finalizing taxonomy
• mixing raw and gold data
• no versioning of labels or schema
• ignoring uncertainty
• using frames instead of events for corner cases
• not tracking provenance
• no QA/review step
• letting test sets leak into training
• no policy for duplicates or conflicts

---

If you want, I can also give you:

1. a sample database schema for this system,
2. a workflow diagram, or
3. a tool stack recommendation for a small research lab vs. a production autonomy team.