Prompt
Why is Splunk missing logs from Kubernetes pods?
Latest observation
Oct 28, 2026
OpenAI APIWeb search: off
Splunk usually isn’t “missing” Kubernetes pod logs randomly — they’re often not being collected, enriched, or indexed the way you expect. Common causes include:
1) The logs never reach the collector
- Container writes to stdout/stderr incorrectly or logs to a file inside the container instead of stdout.
- Pod is very short-lived and the agent misses the log before the pod disappears.
- Node-level log path isn’t mounted or readable by the log collector.
2) The logging agent can’t see the pod logs
- DaemonSet permissions/RBAC are incomplete.
- The collector isn’t mounting:
/var/log/var/log/containers/var/log/pods
- SELinux/AppArmor or filesystem permissions block access.
3) Kubernetes metadata/labels are missing
- Splunk receives the log line, but metadata enrichment fails, so it may look like the pod logs are missing or hard to find.
- Misconfigured pod annotations, labels, or cluster metadata settings.
4) Filtering or routing is excluding them
- Fluentd/Fluent Bit/HEC pipeline filters out namespaces, labels, or container names.
- Index/sourcetype routing sends them somewhere else.
- Search filters in Splunk are too narrow.
5) Timestamp or parsing issues
- Bad timestamps can make logs appear “missing” because they’re indexed with an unexpected time.
- Multiline logs may be split or dropped depending on parser config.
6) Resource or backpressure problems
- Collector buffer overflow.
- HEC token throttling or HTTP errors.
- Network issues between collector and Splunk HEC/indexer.
- High log volume causing drops.
7) Rotation / ephemeral storage issues
- Docker/containerd log rotation may delete logs before collection.
- If the node is under pressure, logs can be removed or truncated.
8) The app isn’t actually emitting logs where you think
- Check if the app logs to a sidecar, a volume, or a different file path.
- Check for application log level changes or disabled logging.
Quick troubleshooting checklist
- Check if logs exist on the node
- Look under
/var/log/containers/and/var/log/pods/
- Look under
- Check the logging agent
- Fluent Bit/Fluentd/Splunk Connect for Kubernetes pods running?
- Any errors in their logs?
- Verify permissions
- RBAC, volume mounts, and service account permissions
- Check Splunk ingestion
- HEC errors, index size limits, dropped events, throttling
- Search broadly in Splunk
- Try searching by namespace, pod name, container name, and time window
- Verify timestamps
- Search wider time range to catch clock skew or parsing issues
- Look for pipeline filters
- Namespace exclusions, regex filters, routing rules
If you want, I can help you narrow this down with a step-by-step checklist for:
- Splunk Connect for Kubernetes
- Fluent Bit / Fluentd
- Splunk HEC
- EKS / AKS / GKE