I’m building a content portal and need near real-time indexing from multiple sources — what should I use?

For near real-time indexing from multiple sources, the best fit is usually a search engine with streaming/ingestion connectors plus a message queue or change-data-capture pipeline.

Good options

1) Elasticsearch / OpenSearch

Best when you need:

• Full-text search
• Faceting/filtering
• Near real-time document updates
• Multiple ingestion sources

Typical setup:

• Sources → Kafka / RabbitMQ / SQS / CDC → indexing workers → Elasticsearch/OpenSearch
• Or use built-in connectors / ingest pipelines if your sources are simple

Why it’s popular:

• Documents become searchable in seconds
• Flexible mapping and query capabilities
• Strong ecosystem for logs, content, and portal search

2) Algolia

Best when you want:

• Very fast hosted search
• Minimal ops
• Excellent relevance/search UX

Tradeoffs:

• Less flexible for complex backend indexing pipelines
• Can get expensive at scale
• Not ideal if you need deep control over indexing logic

3) Meilisearch

Best when you want:

• Simple deployment
• Fast text search
• Lightweight indexing for a content portal

Tradeoffs:

• Simpler feature set than Elasticsearch/OpenSearch
• Better for straightforward portal search than complex enterprise needs

4) Apache Solr

Best when:

• You already use Solr
• Need mature enterprise search features

Tradeoffs:

• More operational overhead than newer hosted options
• Less common for new builds than Elasticsearch/OpenSearch

---

Recommended architecture for your case

If you have multiple sources and need near real-time updates, I’d recommend:

Source systems
→ CDC / webhooks / polling / file drops
→ Kafka or another queue
→ Indexer service
→ Elasticsearch or OpenSearch

This gives you:

• Decoupling from source systems
• Retry handling
• Scalability
• Easy near real-time updates

---

My practical recommendation

• Need control + scale + multi-source ingestion? → OpenSearch or Elasticsearch
• Need fastest time to market and managed search? → Algolia
• Need simple portal search with low ops? → Meilisearch

If you want, I can suggest a 具体 architecture based on your stack (AWS/GCP/Azure, databases, content types, and expected update rate).