How do I set up a video labeling software workflow for time-synced trajectory annotation on robot datasets?

Here’s a practical way to set up a video labeling workflow for time-synced trajectory annotation on robot datasets.

1) Define the annotation target

First decide exactly what “trajectory” means in your project:

• Object trajectory in image space: bounding boxes / keypoints over time
• Robot pose trajectory: base pose or end-effector pose over time
• State trajectory: action/state labels aligned to frames
• Multi-agent trajectories: robot + humans + tools

This determines the annotation schema and tool features you need.

---

2) Prepare the dataset for synchronization

For robot datasets, the main challenge is aligning video with other sensors.

Recommended data format

Store each recording as a session with:

• video files: camera streams
• timestamps: per-frame or per-video PTS
• robot telemetry: joint states, odometry, IMU, control commands
• calibration metadata: camera intrinsics/extrinsics, camera-to-robot transform
• event markers: task start/end, contact events, gripper open/close, etc.

Key requirement

Make sure every modality shares a common time base, such as:

• ROS time
• Unix epoch
• monotonic clock with known offset

If you don’t already have that, create a preprocessing step to align streams by:

• exact timestamps
• interpolation of sparse telemetry
• nearest-neighbor frame matching for annotations

---

3) Choose labeling software with timeline support

You want a tool that supports:

• video playback
• frame-by-frame annotation
• interpolation between keyframes
• timeline or segment labeling
• track propagation
• keyboard shortcuts
• export in structured format

Good options

• CVAT: strong for video tracking and interpolation
• Label Studio: flexible, easier to customize
• Supervisely: strong workflow and collaboration
• Scalabel: good for video/object tracking
• VIA: lightweight, but less powerful for synced workflows

For robot trajectory annotation, CVAT is often the best starting point.

---

4) Set up annotation classes and schema

Define labels before annotation begins.

Example schema

• robot_base
• end_effector
• tool
• human_hand
• target_object
• contact_event
• grasp_start
• grasp_end
• failure_event

For trajectories

Decide whether each trajectory is:

• a track of a visual entity
• a polyline path
• a sequence of keypoints
• a time interval label
• a combination of the above

For robot motion, it’s common to annotate:

• object tracks per frame
• event timestamps
• pose keyframes at selected times

---

5) Create synchronized video tasks

Split recordings into annotation units.

Best practice

Create tasks by:

• single episode
• fixed time windows, e.g. 30–120 seconds
• task-specific segments, e.g. “pick-and-place”

Include in each task:

• the relevant video
• linked metadata JSON
• a timeline panel if your tool supports it
• frame timestamps

If you have multiple camera views:

• keep them in the same task
• synchronize playback using timestamps
• use the same segment boundaries across all views

---

6) Add a timeline overlay or sidecar data

For robot data, labeling improves a lot if the annotator can see telemetry.

Useful overlays

• joint positions
• gripper open/close state
• end-effector pose
• robot speed
• event flags

If the labeling tool doesn’t support this natively, add:

• an auxiliary dashboard
• custom video overlay
• sidecar visualization synchronized to the same timestamps

A common workflow is:

1. video frame shown in labeling tool
2. telemetry shown in a separate synchronized panel
3. annotator labels events/trajectories using both

---

7) Use keyframes plus interpolation

For trajectory annotation, don’t manually label every frame unless necessary.

Workflow

• Annotate an object or robot part at a few keyframes
• Let the tool interpolate between frames
• Correct only the frames where tracking drifts
• Mark occlusions or lost tracking explicitly

This is much faster for long robot videos.

For robot datasets, make sure the annotator can:

• jump to exact timestamps
• move frame-by-frame
• see frame index/timecode
• adjust keyframes precisely

---

8) Include quality control

Build in QA from the start.

QC steps

• double-annotate a subset of sequences
• measure inter-annotator agreement
• inspect drift over long tracks
• validate timestamp alignment
• check that tracks don’t jump between objects

Automated checks

• track continuity
• impossible speed jumps
• labels outside valid time ranges
• missing frames / broken sync
• consistent class names and IDs

---

9) Export annotations in a robotics-friendly format

Your export should preserve time and identity.

Ideal export fields

• episode_id
• frame_id
• timestamp
• track_id
• label
• geometry
  • bbox / polygon / keypoints / pose
• confidence or annotator
• occluded, truncated, lost

Common downstream formats

• COCO-like for vision
• MOT format for tracking
• custom JSON for trajectory and event labels
• ROS bag-derived metadata for robotics pipelines

For robot learning, many teams use a custom JSON/Parquet format because it handles timestamps and multi-modal data better than standard vision formats.

---

10) Automate the boring parts

You can make the workflow much more efficient with scripts.

Automation ideas

• split raw logs into tasks
• extract video from ROS bags
• generate frame timestamps
• convert telemetry to CSV/JSON
• precompute model predictions for pre-labeling
• import existing tracks for refinement
• batch export annotations into training format

Good pattern

Use a pipeline like:

1. raw data ingestion
2. synchronization and preprocessing
3. task creation in labeling tool
4. human annotation
5. QA
6. export
7. training set build

---

11) Recommended workflow for robot trajectory annotation

A robust setup looks like this:

1. Record
   • synchronized camera + robot sensor data
2. Preprocess
   • extract video and timestamps
   • align streams
   • create metadata
3. Pre-label
   • run detector/tracker to generate initial tracks
4. Annotate
   • human corrects tracks and adds events
5. QA
   • review difficult segments and sync issues
6. Export
   • save as timestamped trajectories
7. Train
   • feed into perception or imitation learning models
8. Iterate
   • sample failure cases for relabeling

---

12) Practical software recommendation

If you want a concrete stack:

Option A: Most common

• CVAT for annotation
• ROS/bag or custom preprocessing scripts
• Python export pipeline
• Custom sync dashboard for telemetry

Option B: More customizable

• Label Studio
• custom frontend for robot telemetry
• custom annotation schema and export code

Option C: Enterprise/collaborative

• Supervisely
• built-in collaboration and video tracking tools

---

13) Important design tips

• Always store timestamps alongside frame numbers
• Don’t rely on frame count alone if video is variable FPS
• Use track IDs consistently across occlusion
• Treat occlusion/lost as first-class states
• Keep raw data immutable; annotate on derived tasks
• Version your label schema

---

If you want, I can also give you:

1. a CVAT-based setup guide,
2. a custom JSON schema for time-synced robot trajectories, or
3. a sample Python preprocessing pipeline for ROS/video synchronization.