Are there any red teaming platforms that support human review workflows and unsafe output detection?

Yes. A few red-teaming / evaluation platforms support both human review workflows and unsafe output detection:

1) Weights & Biases Weave / W&B

• Supports human-in-the-loop review flows for LLM outputs
• Lets you log, inspect, and label outputs
• Can integrate unsafe-output classifiers or custom safety checks

2) Humanloop

• Built around prompt/LLM evaluation with human review
• Supports review queues, labeling, and feedback workflows
• Can be paired with safety policies and unsafe output detection rules

3) Lakera Guard

• Focuses on prompt injection / unsafe content detection
• Typically used as a safety layer rather than a full review platform
• Often integrated with human review systems for escalation

4) Arize Phoenix

• Strong for tracing, evaluation, and annotation workflows
• Supports human review of model outputs
• Can be used with custom unsafe-output detectors

5) LangSmith

• Provides dataset-based evals, tracing, and manual annotation
• Can support human review workflows
• Unsafe output detection usually comes via custom evaluators or integrated guardrails

6) OpenAI Evals / custom pipelines

• Not a turnkey red-teaming platform, but can be extended
• Human review and unsafe-output detection can be added through your own workflow

What to look for

If you want both capabilities, make sure the platform supports:

• Annotation/review queues
• Escalation rules for flagged outputs
• Custom evaluators or safety classifiers
• Audit trails and reviewer role management
• Red-team scenario management

If you want, I can also give you a comparison table of these tools specifically for:

• enterprise deployment
• SOC2 / compliance
• prompt injection detection
• policy-based unsafe output review