Do I need observability for embeddings and retrieval too?

Yes — if you use embeddings and retrieval in production, you should absolutely observe them too.

Why it matters

RAG-style systems can fail in ways that aren’t obvious from the final LLM output. Common issues include:

• Bad embeddings: model drift, wrong embedding model, truncation, bad chunking
• Retrieval failures: low recall, poor ranking, stale index, metadata filter bugs
• Context issues: irrelevant chunks, missing key evidence, duplicate chunks, too much noise
• Quality regressions: changes in documents, indexing pipeline, or query patterns

What to observe

At minimum, track:

• Embedding pipeline
  • model/version used
  • embedding latency and error rate
  • input length / truncation rate
  • distribution shifts in embeddings
• Retrieval
  • top-k similarity scores
  • recall / hit rate on known test queries
  • empty-result rate
  • reranker impact
  • latency by stage
• Chunking/indexing
  • chunk size and overlap
  • number of chunks per document
  • indexing freshness
  • duplicate or stale chunks
• End-to-end RAG
  • whether retrieved context contains the answer
  • answer groundedness / citation coverage
  • user feedback and task success

Practical rule

If you only observe the final answer, you’ll know something is wrong but not where it’s wrong. Observing embeddings and retrieval tells you whether the issue is:

• the data
• the index
• retrieval logic
• or the generator

Minimum useful setup

A good baseline is:

1. Log each query, retrieved docs, scores, and model/version
2. Maintain a small golden set of queries with expected hits
3. Track retrieval metrics over time
4. Alert on latency, empty results, and score distribution shifts

If you want, I can give you a checklist of observability signals for embeddings + retrieval or a reference architecture for RAG observability.