How do I set up a telephony AI stack for an IVR replacement with call routing, recording consent, and fallback to a human agent?

Here’s a practical blueprint for setting up a telephony AI stack to replace or augment IVR with:

• call routing
• recording consent handling
• fallback to a human agent

I’ll describe the architecture, core components, call flow, and implementation considerations.

---

1) High-level architecture

A typical stack looks like this:

Telephony provider

• Twilio, Vonage, Plivo, Telnyx, or SIP trunk to your PBX/contact center
• Handles inbound/outbound calls, PSTN connectivity, recording, transfers

Voice AI orchestration layer

• A backend service that manages the call state machine
• Receives call events, decides routing, invokes AI, and coordinates tools/actions

Speech pipeline

• ASR/STT: speech-to-text, e.g. Deepgram, AssemblyAI, Whisper, Azure Speech
• LLM / dialog engine: determines intent, extracts entities, generates responses
• TTS: text-to-speech, e.g. ElevenLabs, Azure TTS, Amazon Polly, Google TTS

Business systems

• CRM, ticketing, order systems, identity verification, scheduling, knowledge base

Human handoff

• Contact center/agent desktop: Zendesk, Genesys, Five9, Amazon Connect, Intercom, custom queue
• Warm transfer or cold transfer based on confidence and policy

Observability and compliance

• Call logs, transcripts, analytics, QA, consent records, redaction, retention controls

---

2) Core call flow

A common inbound flow:

1. Call arrives

   • Telephony provider hits your webhook.
   • Your orchestration service answers and starts the session.

2. Greet + disclosure

   • “Hi, this is the virtual assistant for X. This call may be recorded for quality purposes. Do I have your consent?”
   • If recording is required before collecting info, ask consent immediately.

3. Consent handling

   • If user agrees:
     • start recording
     • persist consent timestamp, phone number, call ID, policy version
   • If user declines:
     • either continue without recording if allowed, or route to a human agent / end call depending on policy

4. Intent capture

   • ASR converts speech to text
   • LLM classifies intent: billing, tech support, appointment, sales, etc.

5. Routing decision

   • If intent is supported, AI handles it.
   • If intent is ambiguous, high-risk, or user requests an agent, transfer to human.
   • If caller fails verification or is upset, route to human.

6. Action execution

   • AI can look up account, create ticket, schedule callback, check order status, etc.

7. Fallback / escalation

   • Confidence below threshold
   • Policy trigger
   • User asks for human
   • System error / timeout
   • Transfer to agent queue with context

---

3) Recommended system design

A. Telephony layer

Use a provider that supports:

• webhooks for call events
• media streaming for real-time audio
• call transfer / bridging
• recording controls
• DTMF capture
• SIP or PSTN termination

Twilio example capabilities

• Voice webhooks
• TwiML for call control
• Media Streams for live audio to your AI stack
• Recording APIs
• Conference bridging and transfers

B. Orchestration service

This is your “brain” for the call. It should:

• maintain per-call state
• prompt the user
• manage consent
• decide when to invoke AI tools
• track confidence thresholds and escalation rules
• pass context to a human agent upon transfer

Implement as:

• Node.js, Python, or Go service
• with Redis for short-lived call state
• PostgreSQL for call records and consent logs
• queue/event bus if integrating many systems

C. Real-time speech pipeline

Two common patterns:

Pattern 1: streaming real-time

Best for conversational experiences.

• Telephony streams audio to your ASR
• ASR returns partial transcripts
• LLM decides responses
• TTS streams back audio

Pattern 2: turn-based

Best for simpler IVR replacement.

• User speaks
• ASR processes utterance
• LLM responds
• TTS plays response
• Repeat

Real-time streaming feels much more natural.

---

4) Recording consent flow

This is critical for legal and trust reasons. Requirements vary by jurisdiction, so implement policy-based consent handling.

Suggested approach

1. Announce recording
   • “This call may be recorded for quality and training purposes.”
2. Ask for consent
   • “Do I have your permission to continue?”
3. Capture response
   • Voice or DTMF:
     • “Yes” / “No”
     • Press 1 / Press 2
4. Store proof
   • Call ID
   • Timestamp
   • Disclosure text version
   • User response
   • Jurisdiction/policy applied
5. Branch behavior
   • Consent granted: recording on
   • Consent denied:
     • stop recording immediately
     • continue unrecorded if allowed
     • otherwise transfer to a human or end call

Best practice

• Use double opt-in in two-party consent jurisdictions if needed.
• Make sure your agent desktop also shows recording status.
• If transferring to a human, communicate whether the call is being recorded.

---

5) Routing logic

Use a policy engine or rules service for routing.

Example routing inputs

• caller-selected reason
• ASR/LLM intent
• confidence score
• customer tier
• account status
• language detected
• business hours
• queue wait time
• compliance flags

Routing examples

• Sales → sales queue or AI-assisted sales flow
• Billing → AI handles simple invoices; escalation for disputes
• Technical support → AI gathers symptoms; transfer if complex
• VIP → priority human queue
• Unknown/low confidence → agent fallback

Simple routing rule sample

• If confidence < 0.75 → human agent
• If user says “agent”, “representative”, or “someone” → human agent
• If intent is “cancel service” or “charge dispute” → human agent
• If verification fails twice → human agent
• If after 2 clarification attempts still ambiguous → human agent

---

6) Human fallback design

You want the handoff to be smooth and contextual.

Include context in transfer

Send the agent:

• caller name / phone number
• verified account ID if available
• call reason
• transcript summary
• key entities collected
• consent status
• error reason for escalation

Transfer types

• Cold transfer: route directly to queue
• Warm transfer: AI or IVR briefs the agent first
• Conference bridge: AI stays on until agent joins, then drops off

Recommended fallback flow

1. AI says: “I’m connecting you to a specialist.”
2. Place caller in queue
3. Push a summary to agent desktop via API/webhook
4. Optionally have AI speak a short handoff summary to the agent in a whisper channel
5. Agent joins, takes over

---

7) Implementation components

Backend services

• Call session service: state machine, event handling
• Consent service: recording permissions and legal flags
• Conversation engine: prompt management and intent routing
• Integration service: CRM/ticketing/order lookup
• Transfer service: agent routing and queue logic
• Audit log service: immutable logging of key events

Data stores

• Redis: live call state, short-lived session cache
• Postgres: consent logs, transcripts metadata, routing decisions
• Object storage: recordings, if retained
• Vector DB: knowledge base retrieval, if using RAG

Security

• Encrypt recordings at rest
• Restrict transcript access
• Minimize PII exposure in prompts/logs
• Redact payment data, SSNs, DOBs
• Use signed webhooks and mTLS where possible

---

8) Example call state machine

A simple state machine:

1. INIT
2. GREETING
3. CONSENT_REQUESTED
4. CONSENT_GRANTED or CONSENT_DENIED
5. IDENTIFY_INTENT
6. HANDLE_AUTOMATION
7. ESCALATE_TO_AGENT if needed
8. TRANSFER
9. END

Include transitions for:

• timeout
• silence
• barge-in
• ASR failure
• TTS failure
• repeated clarification failure

---

9) Practical tech stack example

Option A: Twilio + Python

• Twilio Voice + Media Streams
• FastAPI backend
• Deepgram ASR
• OpenAI or another LLM
• ElevenLabs or Amazon Polly TTS
• Redis + Postgres
• Zendesk or Genesys for human handoff

Option B: AWS-native

• Amazon Connect for telephony/contact center
• Lex or custom LLM orchestration
• Amazon Transcribe + Polly
• Lambda/ECS for orchestration
• Connect agent queues for transfer

Option C: SIP/contact center integrated

• SIP trunk into your PBX/contact center
• Custom media gateway
• ASR/LLM/TTS services behind your orchestration layer
• Best if you already have a large call center environment

---

10) Key design decisions

Latency

Aim for low turn latency:

• partial ASR
• short LLM prompts
• streaming TTS
• prefetch likely responses

Confidence thresholds

Define:

• intent confidence threshold
• entity extraction confidence
• verification confidence
• escalation thresholds

Compliance

• consent rules by region
• retention policies
• data subject requests
• call recording disclosure
• agent notification requirements

Reliability

• fallback greeting if AI fails
• retry on ASR/LLM/TTS failures
• circuit breaker to route to humans during outages

---

11) Minimal production checklist

• inbound call webhook
• call state machine
• recording consent capture
• transcription pipeline
• LLM intent/response pipeline
• human transfer/queue integration
• recording controls
• audit logs
• PII redaction
• monitoring and alerts
• legal/compliance review
• test scripts for edge cases

---

12) A good starting MVP

If you want to move fast, build this first:

1. Inbound phone number
2. Greeting + consent prompt
3. Simple intent classification:
   • sales
   • support
   • billing
   • agent request
4. Two or three automated workflows
5. Human transfer for everything else
6. Transcript and call summary sent to CRM

This gives you an IVR replacement quickly without over-engineering.

---

If you want, I can also provide:

1. a reference architecture diagram
2. a Twilio + FastAPI example
3. a sample call state machine
4. a vendor comparison table for telephony/ASR/TTS/LLMs