Prompt
How do I build multilingual search for internal documents?
Latest observation
To build multilingual search for internal documents, you want a search system that can index documents in many languages and understand user queries in any supported language. The best approach is usually a hybrid of text-based retrieval + multilingual semantic search.
1) Start with the use case
Decide:
- Which languages you need to support
- Whether users search in their own language or also across languages
- What document types you have: PDFs, Word docs, email, wiki pages, tickets
- Whether exact keyword matching matters more than meaning-based retrieval
If users may search English and find Spanish/French documents, you need cross-lingual search, not just translated keyword search.
2) Build a clean ingestion pipeline
Before search, documents need to be processed consistently:
- Extract text from PDFs, docs, HTML, scans
- Detect language per document or even per section
- Normalize text: Unicode cleanup, punctuation, whitespace
- Split into chunks for indexing
- Store metadata: title, author, date, department, ACLs, source system, language
For scanned docs, use OCR with language-aware settings.
3) Use hybrid search, not only one method
A. Lexical search
Use BM25 or a similar inverted-index search engine:
- Elasticsearch / OpenSearch / Solr / Meilisearch
- Good for exact terms, names, product codes, legal phrases
- Works well when users know the exact wording
B. Multilingual semantic search
Use multilingual embeddings to index document chunks:
- Sentence-transformers multilingual models
- Multilingual embedding APIs
- Vector database or vector search in your search engine
This helps with:
- Synonyms
- Paraphrases
- Queries in one language matching docs in another
C. Combine both
A hybrid system usually performs best:
- Retrieve top candidates from BM25 and vector search
- Merge or rerank them
- Apply business filters and ACLs
4) Choose multilingual embeddings
Pick embeddings trained for cross-lingual similarity, such as:
multilingual-e5LaBSEbge-m3- Commercial multilingual embedding APIs
Important properties:
- Query and document should use the same embedding model
- Model should support many languages well
- Evaluate on your actual language mix, not only benchmark claims
5) Handle language detection and routing
For each document:
- Detect language
- Optionally store a per-language index field
- Use language-specific analyzers for lexical search
For each query:
- Detect query language
- Search:
- same-language keyword index
- multilingual vector index
- If needed, translate the query into other languages for expansion
You can also keep separate analyzers for languages like English, German, French, Japanese, etc., because stemming/tokenization differs.
6) Chunk documents intelligently
Don’t index entire documents as one blob if they’re long. Instead:
- Chunk by section, heading, paragraph, or semantic boundaries
- Keep chunk size moderate, often 200–800 tokens
- Preserve document and section metadata
- Index title separately from body
This improves recall and makes results more precise.
7) Support cross-lingual retrieval
There are three main strategies:
Option 1: Multilingual embeddings only
Simplest:
- Embed everything
- Search by meaning across languages
Pros:
- Less operational complexity
- No translation pipeline
Cons:
- Sometimes weaker for exact term matching
Option 2: Translate queries
- Translate the user query into all supported languages
- Search each language index
Pros:
- Strong lexical recall
Cons:
- Translation errors, extra latency, higher cost
Option 3: Hybrid multilingual + translation
Best in many cases:
- Multilingual vector retrieval for semantic matches
- Translation-expanded lexical retrieval for exact terms
- Merge results
8) Respect permissions and security
Internal document search must enforce access control:
- Filter results by user permissions before final ranking
- Prefer document-level and chunk-level ACLs in the index
- Avoid leaking restricted text in snippets or autocomplete
This is critical in enterprise search.
9) Rank results well
Good ranking often uses:
- BM25 score
- Vector similarity
- Freshness
- Document authority/popularity
- User context: department, role, location
- Language match
- ACL-safe reranking
A learning-to-rank or reranker model can help if you have click data.
10) Evaluate with real queries
You need an evaluation set with:
- Real user queries in multiple languages
- Expected relevant documents
- Cross-lingual queries, if applicable
Measure:
- Recall@k
- MRR / nDCG
- Language-specific performance
- Latency
Test specifically for:
- Names and abbreviations
- Accents/diacritics
- Mixed-language queries
- Domain-specific vocabulary
11) A practical architecture
A common production design:
- Ingest docs
- Extract text + detect language
- Chunk documents
- Index chunks in:
- BM25 index with language analyzers
- Vector index with multilingual embeddings
- On query
- Detect language
- Run BM25 + vector retrieval
- Optionally translate query
- Merge results
- Rerank
- Enforce ACLs
- Return snippets and source links
12) Recommended starting stack
A solid baseline stack could be:
- OpenSearch/Elasticsearch for lexical search
- Multilingual embeddings for semantic search
- A reranker for final ranking
- OCR and document parsing for ingestion
- Language detection library for routing
- Metadata + ACL store for permissions
13) Common mistakes to avoid
- Relying only on translation
- Using English-only embeddings for multilingual data
- Ignoring document chunking
- Not handling permissions at retrieval time
- Using the same analyzer for all languages
- Not evaluating on real internal queries
14) Simple rule of thumb
If you want the shortest path to a good system:
- Use hybrid search
- Use multilingual embeddings
- Add language-aware lexical indexing
- Enforce ACL filtering
- Evaluate with real multilingual queries
If you want, I can also give you:
- a reference architecture diagram,
- a tech stack recommendation, or
- a sample implementation in Elasticsearch/OpenSearch + multilingual embeddings.