To build live speech-to-text into an app, you generally need four pieces:

1. Capture microphone audio
2. Stream audio to a speech-to-text engine
3. Receive partial and final transcripts in real time
4. Display/update the UI as text arrives

There are two main approaches:

---

1) Use a cloud speech API

Best if you want accuracy and less infrastructure work.

Common providers

• Google Cloud Speech-to-Text
• OpenAI transcription APIs (often batch-oriented; live streaming depends on your architecture and product needs)
• Azure Speech
• AWS Transcribe
• Deepgram
• AssemblyAI

Typical flow

• User grants mic permission
• App records audio in small chunks
• Chunks are sent over WebSocket/gRPC/HTTP streaming
• Provider returns:
  • interim results while speaking
  • final results when a phrase ends
• You append/update the transcript in the UI

Pros

• High accuracy
• Speaker diarization, punctuation, language detection often available
• Minimal ML ops

Cons

• Ongoing cost
• Latency depends on network
• Privacy/compliance considerations

---

2) Run speech-to-text locally/on-device

Best if you need privacy, offline support, or low recurring cost.

Options

• Whisper / faster-whisper on a server
• Vosk for lighter on-device/offline use
• Native OS APIs:
  • iOS Speech framework
  • Android SpeechRecognizer / on-device speech services
  • Web Speech API in browsers that support it

Pros

• Better privacy
• Can work offline
• No per-minute cloud cost

Cons

• More engineering
• Model size / performance constraints
• Accuracy may vary by device and language

---

Recommended architecture

For a web or mobile app

Client

• Requests microphone permission
• Captures audio frames (e.g. 16 kHz mono PCM)
• Sends frames to backend over WebSocket

Backend

• Buffers/transcodes audio if needed
• Streams to STT provider or local model
• Sends transcript updates back to client

UI

• Shows interim transcript in gray
• Commits final transcript in black
• Handles reconnects and errors

---

Key technical details

Audio format

Most speech engines prefer:

• 16-bit PCM
• Mono
• 16,000 Hz sample rate

If your app records in another format (like AAC/Opus), you may need to decode/transcode.

Chunk size

Send small chunks, often:

• 20–100 ms of audio per packet

Smaller chunks reduce latency, but too small can increase overhead.

Interim vs final results

• Interim: text can change as the model refines its guess
• Final: committed transcript segment

Your UI should treat these differently.

Silence detection / utterance segmentation

You often need:

• VAD (voice activity detection)
• Push-to-talk or automatic end-of-speech detection
• Timeouts for stopping after silence

This helps finalize phrases cleanly.

---

Example implementation pattern

Frontend

• Use Web Audio API / MediaRecorder / native audio recorder
• Convert mic stream to the format required by your STT backend
• Send through WebSocket

Backend

• WebSocket endpoint receives audio
• Forward to STT service
• Emit transcript events back to client:
  • partial
  • final
  • error

---

Very simple UI event model

{ "type": "partial", "text": "hello worl" }
{ "type": "partial", "text": "hello world" }
{ "type": "final", "text": "hello world" }

---

Practical tips

• Add VAD to cut down on costs and improve finalization
• Use a low-latency transport like WebSocket
• Throttle UI updates to avoid excessive rerenders
• Handle network drops gracefully
• Test with noisy environments and multiple accents
• Measure end-to-end latency from speech to transcript

---

If you want the fastest path

If you want something working quickly:

Web app

• Use browser mic capture
• Send audio to your backend via WebSocket
• Use a provider with streaming support like Deepgram, Google, Azure, or AWS Transcribe

Mobile app

• Use native audio capture
• Stream to backend
• Pick a provider or local engine

Desktop/offline

• Run Whisper/faster-whisper or Vosk locally

---

If you want, I can help with one of these next:

1. A concrete architecture for your stack
2. Sample code for web live transcription
3. Mobile implementation plan
4. Choosing the best STT provider
5. Building an offline/low-latency solution

If you tell me your platform (web, iOS, Android, desktop, backend language), I can give you a step-by-step implementation.