Are there any guardrails engines that block unsafe outputs without adding much latency to streaming responses?

Yes — but the practical answer is “somewhat, with tradeoffs.”
If you want real-time streaming and low added latency, the best pattern is usually inline token streaming plus lightweight incremental filtering, not a full second model pass over the entire response.

Options that tend to be low-latency

1) Rule-based / regex / classifier guardrails

These are the fastest.

• Pros: Very low latency, often sub-10 ms to a few tens of ms.
• Cons: Easier to miss nuanced unsafe content; can be brittle.
• Good for: PII, profanity, jailbreak keywords, obvious policy violations.

Examples:

• Microsoft Presidio for PII detection/redaction
• Custom regex / keyword / pattern filters
• Lightweight text classifiers at the token or chunk level

2) Streaming moderation on chunks

You inspect partial output every N tokens or every chunk.

• Pros: Works with streaming; can stop generation before the full unsafe answer completes.
• Cons: Adds some delay depending on chunk size and model.
• Good for: Catching unsafe content early without waiting for the whole completion.

Typical approach:

• Buffer 10–50 tokens
• Run a fast moderation check
• If unsafe, truncate/replace/abort stream

3) Hybrid “pre-check + stream monitor” systems

These are often the best latency/safety balance.

• Pre-check: Prompt/input moderation before generation starts
• Stream monitor: Lightweight monitoring during generation
• Post-check: Optional final moderation after completion

This can keep added latency small while still giving decent coverage.

Guardrails products/frameworks to look at

NeMo Guardrails

• Can enforce policies, but heavier configurations can add latency.
• Better when you need structured dialogue governance rather than ultra-low-latency streaming.

Guardrails AI

• Useful for validation of structured outputs.
• For freeform streaming, it’s less of a “real-time block unsafe text” engine and more of an output validator.

OpenAI / Anthropic / AWS moderation tools

• Often used as a separate moderation step.
• Latency depends on how you integrate them; not always ideal for per-token streaming blocks.

LlamaGuard / custom small safety models

• Small classifier models can be fast enough for chunk-level streaming moderation.
• Often a good choice if you can host locally and want low overhead.

What usually works best in practice

If you need:

• Low latency
• Streaming UX
• Some ability to block unsafe outputs

Then a common production pattern is:

1. Moderate the user input
2. Generate stream
3. Check each chunk with a lightweight safety layer
4. Stop or redact if a violation is detected
5. Optionally do a final pass for compliance logging

Important limitation

No guardrails engine can perfectly “block unsafe outputs” without either:

• adding some latency, or
• allowing a small amount of unsafe text to momentarily appear before being cut off.

If you need hard guarantees, you usually sacrifice some streaming smoothness.

Rule of thumb

• Lowest latency: rules/regex/PII detectors
• Best balance: small classifier + chunked streaming moderation
• Best governance: full guardrails frameworks, but usually slower

If you want, I can recommend a few specific architectures depending on whether you’re using OpenAI, Anthropic, vLLM, or a local open-source model.