Are there any multi-modal annotation tools that track inter-annotator agreement and versioning for model training?

Yes — there are several multimodal annotation platforms that support some combination of:

• image / video / audio / text / sensor data annotation
• inter-annotator agreement (IAA) tracking
• versioning / audit history
• export for model training

A few strong options:

1) Labelbox

• Supports image, video, text, and some multimodal workflows
• Built-in quality control and review workflows
• Can calculate agreement metrics through consensus/review setups
• Tracks dataset/task history and annotation changes
• Good for training pipelines and active learning

2) Supervisely

• Strong for computer vision and multimodal datasets
• Supports images, videos, point clouds, 3D, and audio-related workflows
• Has project/version history and collaboration tools
• Can compare annotations across users and support review/consensus processes
• Flexible for research and production use

3) CVAT

• Open-source and widely used for image/video annotation
• Supports task versioning via exports, backups, and some project organization features
• Has review and team workflows
• Inter-annotator agreement is not always a first-class built-in metric, but it can be managed via task duplication/comparison or custom tooling
• Best if you want open source and control

4) Doccano

• Best for text and sequence labeling rather than rich multimodal data
• Supports collaboration and project management
• Versioning and IAA are more limited than in enterprise tools
• Useful if your “multimodal” workflow is mostly text-centric

5) Prodigy

• Annotation tool designed for ML workflows, especially NLP
• Strong support for active learning and iterative annotation
• Versioning is mostly handled through your data/code pipeline rather than built-in dataset version control
• IAA can be measured externally or with custom scripts

6) Argilla

• Focused on NLP / LLM data curation
• Good dataset tracking and human feedback loops
• Useful for annotation versioning and review, though not a full multimodal CV platform
• IAA support may require external analysis

---

If you specifically need both IAA and versioning

The most practical choices are usually:

• Labelbox for enterprise-grade multimodal workflows
• Supervisely for vision-heavy multimodal projects
• CVAT + external version control / custom metrics if you want open source

---

Important caveat

In many tools, true inter-annotator agreement isn’t a fully automatic “one-click” feature. Often the platform supports:

• assigning the same item to multiple annotators,
• reviewer/consensus workflows,
• exporting annotations,
• then computing Cohen’s kappa, Fleiss’ kappa, IoU-based agreement, etc., externally.

---

If you want, I can also give you:

1. a comparison table,
2. open-source only options, or
3. recommendations based on your modality (image/video/audio/text/3D).