I’m building an internal search tool for employees across docs, chat, and tickets. What stack should I consider?

For an internal employee search tool across docs, chat, and tickets, I’d think about the stack in terms of ingestion, indexing, retrieval, permissions, and UX. The best choice depends on whether you want a classic keyword search, semantic/AI search, or a hybrid of both.

A practical stack to consider

1) Source connectors / ingestion

You’ll need connectors for:

• Google Drive / SharePoint / Confluence / Notion
• Slack / Teams / Discord
• Jira / Zendesk / Linear / ServiceNow
• File shares / PDFs / email if needed

Good options:

• Build custom connectors with vendor APIs
• Use an ingestion platform like:
  • Airbyte
  • Fivetran (more ETL-oriented, less app-specific)
  • Meltano
• For event-driven syncs: Kafka, SQS/SNS, Pub/Sub, or scheduled jobs

2) Processing / enrichment layer

You’ll want to normalize and enrich content:

• HTML/Markdown/PDF parsing
• OCR for scans/images
• Chunking long documents
• Metadata extraction
• Language detection
• Deduplication
• ACL/permissions capture
• PII redaction if necessary

Useful tools:

• Apache Tika for document extraction
• Unstructured for parsing/chunking
• Tesseract or cloud OCR if needed
• spaCy for entity extraction
• Python workers with Celery, RQ, or Temporal

3) Indexing / search engine

For employee search, the usual winner is hybrid search:

• BM25 / keyword search for exact matches
• Vector search for semantic similarity
• Optional re-ranking for quality

Option A: Search-first stack

Best if you want strong traditional search with some semantic enhancement.

• Elasticsearch / OpenSearch
  • Good for full-text search, filtering, facets, highlighting, permissions-aware search
  • Supports vector fields too
• Add embeddings + hybrid scoring

This is often the most practical choice for internal enterprise search.

Option B: Vector-first stack

Best if semantic search is the main product.

• Vector DB:
  • Pinecone
  • Weaviate
  • Qdrant
  • Milvus
• Still often pair with a keyword engine for exact lookup

For most internal search systems, vector-only is not enough—you’ll want hybrid.

4) Embeddings / LLM layer

Use embeddings to index chunks and maybe queries.

Common choices:

• OpenAI text embeddings
• Cohere embeddings
• Voyage AI
• Open-source options like bge, e5, or GTE

For answer generation / summarization:

• An LLM such as GPT-style model
• Or an internal model if privacy/compliance requires it

5) Retrieval orchestration

You’ll likely need logic for:

• Query understanding
• Hybrid ranking
• Permission filtering
• Reranking top results
• Snippet generation
• Q&A over results

Frameworks:

• LlamaIndex
• LangChain
• Custom retrieval service if you want tighter control

For ranking, consider:

• Cross-encoder rerankers
• LLM-based reranking if latency/cost allow

6) Permissions and security

This is one of the most important parts for internal search.

You need:

• Document-level and possibly chunk-level ACLs
• User/group sync from IdP like Okta, Azure AD, Google Workspace
• Search-time authorization filtering
• Audit logs

Common approach:

• Store ACL metadata with each doc/chunk
• Filter at query time before ranking or during retrieval
• Keep a permission graph if access rules are complex

7) Backend / API layer

A typical app backend stack:

• Python FastAPI or Node.js/NestJS
• PostgreSQL for app data, configs, audit trails, feedback
• Redis for caching, rate limiting, sessions, queues
• Object storage: S3/GCS/Azure Blob for raw content

8) UI / frontend

For the employee-facing search app:

• React or Next.js
• Search UX features:
  • autocomplete
  • filters/facets
  • highlighted snippets
  • “why this result?” explanations
  • result previews
  • source tabs (docs/chat/tickets)
  • feedback buttons

---

Recommended reference architectures

Good default stack for most teams

If I were building this today, I’d start with:

• Ingestion: custom connectors + Airbyte where possible
• Parsing: Apache Tika + Unstructured
• Backend: Python FastAPI
• Index: OpenSearch or Elasticsearch
• Vector search: OpenSearch vectors or Qdrant
• Embeddings: OpenAI or Cohere, or open-source if needed
• Orchestration: Celery or Temporal
• DB: PostgreSQL
• Cache/queue: Redis
• Frontend: Next.js
• Auth: Okta / Azure AD integration

This gives you a strong hybrid search system with reasonable operational complexity.

---

When to choose what

Choose Elasticsearch/OpenSearch if:

• You need strong keyword search, filters, facets
• Permissions and auditability matter
• You want one system that can do hybrid search

Choose a vector DB if:

• Semantic retrieval is the primary experience
• You’re okay pairing it with keyword search separately
• You want simpler vector-native retrieval experiments

Choose managed services if:

• You want to move fast
• You don’t want to run infra
• Compliance allows it

Choose open-source/self-hosted if:

• Data sensitivity is high
• You need more control over access and logging
• You have infra/ops capacity

---

A few product/UX notes

Internal search succeeds or fails on these:

• Freshness: employees expect recent Slack/ticket content to appear quickly
• Permissions: no leaks
• Relevance: exact and semantic both matter
• Source awareness: users want to know if a result came from a doc, chat, or ticket
• Trust: show snippets, timestamps, and source links
• Feedback loop: log clicks, saves, reformulations

---

If you want a simple MVP

A strong MVP stack would be:

• Ingest 3–5 sources
• Parse text with Tika/Unstructured
• Store raw docs in S3
• Index in OpenSearch
• Add embeddings for hybrid search
• Build a Next.js UI
• Use Okta/Azure AD for auth
• Add feedback/click logging in Postgres

That’s enough to ship something useful quickly.

If you want, I can also give you:

1. a recommended stack by company size (startup vs enterprise), or
2. a concrete architecture diagram and component list for an MVP.