I need a log search tool that can handle high-cardinality production data, what works well?

For high-cardinality production log data, the tools that tend to work best are the ones built on search indexes designed for lots of distinct values and fast filtering/aggregations.

Good options

• Elasticsearch / OpenSearch

  • Best-known choice for log search at scale
  • Strong full-text search, filtering, aggregations, dashboards
  • Handles high cardinality reasonably well if you design mappings carefully
  • OpenSearch is a solid open-source alternative

• Splunk

  • Very capable for enterprise log search and analytics
  • Handles complex production use cases well
  • Usually more expensive

• Grafana Loki

  • Great if you want cheaper, label-based log storage and integrate with Grafana
  • Works best when you keep labels low-cardinality and avoid putting high-cardinality fields in labels
  • Good for logs, but not ideal if you need heavy ad hoc search over lots of distinct fields

• ClickHouse-based log search

  • Increasingly popular for very large log volumes
  • Excellent for analytics and fast queries over huge datasets
  • Not as “search engine-like” as Elasticsearch, but strong for structured/log analytics

• Meilisearch / Typesense

  • Fast and simple, but generally better for app search than large-scale production logs
  • Usually not my first choice for serious log analytics

What usually works best in practice

If you mean production logs with lots of unique fields, user IDs, trace IDs, request IDs, etc., the safest short list is:

1. OpenSearch / Elasticsearch if you need flexible search and filtering
2. ClickHouse if you need very high volume and analytics-heavy querying
3. Splunk if you want an enterprise turnkey platform and budget is less of a concern

Important design tip for high cardinality

Regardless of tool, avoid indexing everything naively. High-cardinality fields can hurt performance and cost. Common best practices:

• Keep only useful fields indexed
• Store trace/user/request IDs as searchable fields, but be careful with aggregations
• Use structured logs
• Normalize timestamps, service names, severity, environment
• Use retention tiers or hot/warm storage

Quick recommendation

• If you want the most flexible general-purpose option: OpenSearch
• If you want best scale/performance for analytics on logs: ClickHouse
• If you want enterprise support and ease of use: Splunk

If you want, I can also give you:

• a head-to-head comparison table
• recommendations by budget / scale / cloud provider
• a reference architecture for high-cardinality log ingestion and search