How do I set up a model metadata store for tracking runs, artifacts, and promotion history across teams?

A good model metadata store should act like a system of record for the model lifecycle: training runs, artifacts, evaluations, approvals, and promotions across environments/teams.

1) Define the core entities

At minimum, model metadata usually centers on these objects:

• Model: the logical model name or family
• Model version / artifact: a specific trained package, file, or container
• Run: a training or evaluation execution
• Dataset / data snapshot: what data the run used
• Metrics / evaluation results: accuracy, AUC, latency, fairness, etc.
• Experiment / job: a grouping of runs
• Environment: dev / staging / prod
• Deployment / promotion event: when a version moved between environments
• Approval / governance record: who approved and why
• Lineage / dependencies: code version, feature set, data sources
• Ownership / team / project: accountability and access boundaries

2) Use a hybrid metadata architecture

Most teams use:

• Object store for large artifacts
  Examples: S3, GCS, Azure Blob, MinIO
• Relational database for structured metadata
  Examples: Postgres, MySQL
• Optional event log / message bus for async updates
  Examples: Kafka, Pub/Sub, SQS

Typical split:

• Store model binaries, ONNX files, Docker images, reports in object storage
• Store references, hashes, tags, metrics, statuses, and promotion history in the DB

3) Design the data model

A practical relational schema might look like this:

teams

• team_id
• name
• owner
• cost_center

models

• model_id
• name
• description
• team_id
• created_at

model_versions

• version_id
• model_id
• version_name or semver
• artifact_uri
• artifact_hash
• framework
• created_at
• created_by
• git_commit
• status (registered, candidate, approved, deprecated)

runs

• run_id
• model_id
• version_id nullable until registered
• experiment_name
• status
• started_at
• ended_at
• triggered_by
• compute_env
• code_version

run_inputs

• run_id
• dataset_uri
• dataset_version
• feature_store_snapshot
• data_hash

run_metrics

• run_id
• metric_name
• metric_value
• metric_type
• threshold
• passed

artifacts

• artifact_id
• run_id
• version_id
• type (model, plot, confusion_matrix, log, profile)
• uri
• hash
• size_bytes

environments

• environment_id
• name (dev, staging, prod)
• team_scope

promotions

• promotion_id
• version_id
• from_environment_id
• to_environment_id
• status
• requested_by
• approved_by
• approved_at
• reason
• policy_result

audit_events

• event_id
• entity_type
• entity_id
• action
• actor
• timestamp
• details_json

This gives you:

• run tracking
• artifact lineage
• environment promotion history
• auditability across teams

4) Capture lineage and immutability

For trustworthy metadata:

• Make artifacts immutable once registered
• Store content hashes for artifacts and datasets
• Record git commit hash, branch, and build ID
• Track exact dataset versions and feature snapshots
• Record who approved each promotion and what policy was evaluated

This is what lets you answer:

• “What exact data trained prod model X?”
• “Which code commit produced this artifact?”
• “Who promoted it and when?”
• “Was it approved by policy or manually overridden?”

5) Support multi-team access and governance

Across teams, you need:

• RBAC/ABAC
  • Team members can see their models
  • Platform/admins can see everything
  • Only approvers can promote to prod
• Namespace or tenancy model
  • team_id on every major object
  • Optional project/org hierarchy
• Audit logs
  • All reads/writes for sensitive actions
• Policy checks
  • Block promotion if metrics regress, bias thresholds fail, or artifacts are unsigned
• Tags/labels
  • owner, business_unit, pii_sensitive, critical_system

6) Decide how metadata gets written

Good patterns:

Option A: Model registry API

Each training pipeline calls an internal service:

• POST /runs
• POST /artifacts
• POST /metrics
• POST /promotions

Best for centralized control and consistency.

Option B: SDK

Provide a Python/Java SDK so training jobs can log metadata easily:

• log_run()
• log_metric()
• log_artifact()
• register_model_version()
• promote_version()

Best for developer adoption.

Option C: Event-driven

Training jobs emit events; a metadata service consumes them. Best for scale and decoupling, but more complex.

7) Include promotion workflow states

A robust promotion lifecycle often looks like:

1. trained
2. validated
3. candidate
4. approved
5. deployed
6. rolled_back or deprecated

Each state transition should be recorded in promotions or an events table.

You may also want:

• manual approvals
• automated gates
• canary status
• rollback history

8) Add search and discovery

Teams need to find models quickly. Index:

• model name
• tags
• owner/team
• metric values
• environment
• creation date
• framework
• status

If your relational DB search is limited, mirror metadata into:

• Elasticsearch / OpenSearch
• or use Postgres full-text search

9) Build for scale and reliability

Practical implementation tips:

• Use Postgres first unless scale demands otherwise
• Partition large tables like audit_events and run_metrics
• Add indexes on:
  • model_id
  • version_id
  • team_id
  • created_at
  • status
• Keep artifact blobs out of the DB
• Use idempotent writes for pipeline retries
• Version your schema and APIs

10) Example end-to-end flow

1. Data scientist starts training job
2. Job creates a run
3. Job logs input dataset snapshot, code commit, and metrics
4. Job uploads model artifact to object storage
5. Metadata service registers a model_version
6. Evaluation pipeline validates metrics and policy checks
7. If approved, create a promotion record from dev to staging/prod
8. Deployment system reads the metadata store to fetch the correct artifact
9. Audit events capture every transition

11) If you want to start simple

A minimal but solid v1:

• Postgres metadata DB
• S3/GCS artifact storage
• SDK to log runs/metrics/artifacts
• Basic RBAC by team
• Promotion table with approval history
• Audit log table
• Dashboard/search UI

12) Useful existing tools

If you don’t want to build everything:

• MLflow Model Registry
• Weights & Biases
• SageMaker Model Registry
• Vertex AI Model Registry
• Kubeflow Metadata
• OpenLineage / Marquez for lineage

These can be integrated with your own governance layer if needed.

If you want, I can also provide:

1. a sample Postgres schema,
2. a REST API design, or
3. a reference architecture diagram for this metadata store.