Prompt
Why is log search so slow during incidents?
Latest observation
Oct 19, 2026
OpenAI APIWeb search: off
Log search often feels slow during incidents because the system is being asked to do the hardest possible thing at the worst possible time:
-
Massive data volume
- Incidents usually create lots of logs, metrics, and traces at once.
- Searching across terabytes or petabytes of logs is inherently expensive, especially if the query is broad.
-
High cardinality and unstructured data
- Logs are often free-form text with many unique fields/values.
- Unlike a database with carefully indexed columns, log systems may need to scan or partially scan large amounts of data.
-
Indexing tradeoffs
- To ingest logs quickly, systems often optimize for write speed over search speed.
- During incidents, the newest logs may not be fully indexed yet or may be spread across shards/partitions.
-
Burst traffic and resource contention
- The incident itself can increase application log volume.
- At the same time, engineers are running more searches, dashboards, and alerts, competing for CPU, memory, disk, and network.
-
Expensive query patterns
- Wildcards, regex, broad time ranges, and “contains” searches are costly.
- Searching across many services or without precise filters can force the system to examine far more data than necessary.
-
Hot partitions and skew
- If one service or time window suddenly becomes very noisy, the load may concentrate on a few shards or nodes.
- That creates bottlenecks and slows queries.
-
Storage latency
- Logs may live on compressed cold storage or object storage tiers.
- Retrieving and decompressing data takes longer than reading from memory or SSD.
-
Multi-tenant interference
- In shared observability platforms, other teams or workloads can consume shared resources.
- Your incident search is competing with everyone else’s usage.
In short: incident-time log search is slow because the system has to sift through huge, noisy, rapidly growing data under heavy load, often with queries that aren’t narrow enough to be cheap.
If you want, I can also explain:
- how log search systems work internally, or
- how to make incident log search much faster.