How do I build search across docs, tickets, chat, and wiki pages?

To build search across docs, tickets, chat, and wiki pages, treat it as an enterprise search problem with one common pipeline and source-specific connectors.

1) Define the search experience

Decide what users need:

• Unified search bar across all content
• Filters by source: docs / tickets / chat / wiki
• Filters by team, date, author, status, channel
• Sorting by relevance, recency, or popularity
• Snippets with highlights
• Permission-aware results

2) Ingest content from each source

Create connectors for each system:

• Docs: Google Drive, SharePoint, Notion, Confluence, Box, etc.
• Tickets: Jira, Zendesk, ServiceNow, Linear, etc.
• Chat: Slack, Microsoft Teams, Discord, etc.
• Wiki: Confluence, Notion, MediaWiki, internal wiki

For each item, extract:

• id
• source
• title
• body
• author
• created_at
• updated_at
• tags
• url
• permissions / ACLs
• any source-specific metadata

Use:

• Initial full sync
• Incremental sync via webhooks or polling
• Deletion handling
• Version updates

3) Normalize the data

Convert everything into a common schema so search behaves consistently.

Example document schema:

{
  "id": "string",
  "source": "slack|jira|confluence|docs",
  "source_id": "original-source-id",
  "title": "string",
  "body": "string",
  "summary": "string",
  "url": "string",
  "author": {
    "id": "string",
    "name": "string"
  },
  "created_at": "timestamp",
  "updated_at": "timestamp",
  "tags": ["string"],
  "permissions": ["user:123", "group:eng"],
  "type": "message|ticket|page|file",
  "metadata": {}
}

4) Index into a search engine

Use a search engine that supports:

• Full-text search
• Filtering/faceting
• Ranking
• Highlighting
• Synonyms / stemming / typo tolerance
• ACL filtering at query time

Common choices:

• Elasticsearch / OpenSearch
• Solr
• Typesense / Meilisearch for simpler needs
• Postgres full-text search for smaller/internal setups

Typical index fields:

• title
• body
• tags
• source
• author
• created_at
• updated_at
• permissions

5) Handle permissions correctly

This is critical.

At query time, only return items the user can access. Common approaches:

• Store ACLs in the index and filter by user/group IDs
• Use document-level security in the search layer
• Precompute allowed principals for each doc

Be careful with chat and ticket data, since private channels and restricted projects are common.

6) Relevance ranking

Rank results using a blend of:

• Text relevance in title > body
• Source boosts by query intent
• Freshness
• Engagement/popularity
• Exact match boost
• User/team affinity
• Recency for chat/tickets, completeness for docs/wiki

Example:

• For “how to reset vpn”: boost docs/wiki
• For “bug 1234”: boost tickets
• For “incident yesterday”: boost chat and tickets

7) Search UX

Good UX makes search feel great:

• Autocomplete / typeahead
• Result snippets with highlights
• Source badges and icons
• Filter chips
• Recent searches
• Saved searches
• “Did you mean” suggestions
• Zero-result fallback suggestions

8) Pipeline architecture

A common architecture:

1. Connectors pull from APIs/webhooks
2. Normalizer maps data into common schema
3. Enricher adds NLP, tags, embeddings, permissions
4. Indexer writes to search engine
5. Query API serves search requests to UI
6. UI renders results and filters

9) Optional: semantic search

If users ask natural-language questions, add semantic retrieval:

• Generate embeddings for titles/bodies
• Store vectors in a vector DB or hybrid search engine
• Combine keyword + vector search
• Use reranking for better results

This helps for queries like:

• “how do we onboard a new engineer?”
• “what was decided about the billing migration?”

10) Operational concerns

Make sure to handle:

• Rate limits from source APIs
• Incremental sync failures
• Reindexing and backfills
• Deduplication
• Audit logs
• Content freshness SLAs
• Security and encryption

---

Practical stack example

A common setup:

• Ingestion: Python/Node workers + webhooks
• Queue: Kafka / SQS / PubSub
• Index: OpenSearch
• Metadata store: Postgres
• Vector search: OpenSearch k-NN or Pinecone/Weaviate
• API: FastAPI / Node.js
• UI: React

---

Recommended implementation order

1. Start with 1–2 sources
2. Define the shared schema
3. Build permission filtering
4. Add full-text search and filters
5. Improve ranking and snippets
6. Add semantic search later

If you want, I can also give you:

• a reference architecture diagram
• a database/search index schema
• or a step-by-step implementation plan in Python or Node.js