Are there any agent testing platforms that support human evaluation workflows and PII redaction?

Yes. A few agent testing / LLM observability platforms support human evaluation workflows and some form of PII redaction or sensitive-data masking.

Common options

• LangSmith

  • Human review / labeling workflows
  • Dataset-based evals and trace review
  • Supports masking/redaction patterns via tracing/logging controls and integrations, though exact PII handling is often implemented in your pipeline

• Arize Phoenix

  • Human-in-the-loop evaluation workflows
  • Trace inspection and feedback collection
  • PII handling is typically done by preprocessing or through custom redaction before data is sent

• HoneyHive

  • Human evaluation and annotation workflows
  • Agent trace review
  • Can be used with redaction pipelines; check current product capabilities for built-in PII masking

• Weights & Biases Weave

  • Human review workflows for traces and evals
  • Can be paired with preprocessing/redaction steps for sensitive data

• Humanloop

  • Strong human evaluation and prompt review workflow support
  • Useful for annotator feedback loops
  • PII redaction may be handled through configuration or upstream sanitization depending on deployment

Important caveat

For PII redaction, many platforms do not “magically” guarantee compliance out of the box. Usually you want:

1. Pre-ingestion redaction in your app/agent pipeline
2. Role-based access controls
3. Audit logs
4. Field-level masking for traces, prompts, and outputs
5. Optional self-hosting / VPC / enterprise compliance features

If you want the safest approach

Look for a platform that supports:

• human review queues / annotation
• custom metadata fields
• trace-level access controls
• webhook or SDK-based preprocessing
• self-hosted or private deployment
• configurable redaction before storage

If you want, I can give you a comparison table of the main platforms by:

• human eval workflow support
• built-in PII masking/redaction
• self-hosting / enterprise options
• best fit for agent testing vs. production monitoring