Are there any active learning systems that handle streaming data and edge case harvesting?

Yes. There are active learning systems and adjacent human-in-the-loop pipelines that support streaming/online data and can be used for edge-case harvesting, though the exact capability is usually implemented as a combination of:

1. streaming inference + uncertainty/OOD detection
2. sample selection / active querying
3. human review or label feedback
4. retraining or continual learning

What to look for

For “edge case harvesting,” the useful features are typically:

• uncertainty sampling: flag low-confidence predictions
• disagreement / ensemble methods: flag samples where models disagree
• out-of-distribution detection: catch novel or rare cases
• diversity sampling: avoid collecting redundant examples
• stream processing support: ingest data continuously
• annotation loop integration: send selected samples to humans
• drift detection: identify distribution changes over time

Examples of systems / platforms

These are not always marketed specifically as “edge case harvesting,” but they support the workflow:

Active learning / data-centric ML platforms

• Labelbox – supports model-assisted labeling, active learning workflows, and review queues for uncertain or interesting samples.
• SuperAnnotate – active learning and dataset curation features for computer vision pipelines.
• Prodigy – very good for rapid human-in-the-loop annotation and model-in-the-loop sample selection.
• Roboflow – has dataset curation/model-assisted labeling features, useful in vision streaming setups.
• Weights & Biases – not an active learning system itself, but often used to operationalize selection, tracking, and retraining loops.

Open-source libraries

• modAL – a Python active learning framework; you’d wire streaming and harvesting logic yourself.
• ALiPy – active learning toolkit for research/prototyping.
• scikit-activeml – active learning methods in scikit-learn style.
• River – online machine learning library, useful for streaming/drift; can be paired with active learning logic.
• FiftyOne – dataset curation and querying; good for finding hard/rare examples in vision datasets, often used with model-in-the-loop workflows.

Streaming + monitoring + selection stack

If you want real streaming edge-case harvesting, teams often build it from:

• Kafka / Kinesis / PubSub for ingestion
• online model inference
• drift/uncertainty/OOD scoring
• priority queue for review
• annotation tool integration
• scheduled retraining

Practical pattern

A common implementation is:

1. Stream incoming records.
2. Score each record with:
   • confidence,
   • entropy,
   • ensemble disagreement,
   • OOD score,
   • drift score.
3. Keep a small budget of samples for labeling.
4. Prioritize:
   • high uncertainty,
   • rare classes,
   • novel clusters,
   • failure cases,
   • representative diversity.
5. Feed labeled cases back into training.

If you want a product recommendation

• For computer vision: Labelbox, SuperAnnotate, Roboflow, FiftyOne
• For custom ML pipelines: Prodigy + your streaming stack, or modAL/ALiPy/River
• For enterprise workflows: Labelbox or a custom pipeline around Kafka + model scoring + annotation tooling

If you want, I can also give you:

• a shortlist by use case,
• an architecture diagram for streaming edge-case harvesting,
• or a Python stack to build this yourself.