Prompt
How do I ship logs from AWS, Azure, and GCP into one system?
Latest observation
To ship logs from AWS, Azure, and GCP into one system, the usual pattern is:
- Collect logs locally in each cloud
- Forward them to a central ingestion layer
- Normalize/enrich them
- Store and query them in one log platform
Common architectures
1) Central log platform with cloud-native forwarding
Use each cloud’s native logging service as the source, then export to a single destination.
- AWS: CloudWatch Logs → subscription filter → Kinesis / Lambda / Firehose / HTTPS endpoint
- Azure: Azure Monitor / Log Analytics → diagnostic settings → Event Hub / Storage / partner export
- GCP: Cloud Logging → sinks → Pub/Sub / BigQuery / Storage / HTTPS endpoint
Then send all three into:
- Splunk
- Elastic / OpenSearch
- Datadog
- New Relic
- Grafana Loki
- Sumo Logic
- A SIEM / data lake / Kafka pipeline
2) Agent-based collection on workloads
Install an agent on VMs, containers, or Kubernetes nodes and send logs directly to one backend.
Typical agents:
- Fluent Bit
- Vector
- OpenTelemetry Collector
- Filebeat / Elastic Agent
- Datadog Agent
This works well for:
- Kubernetes
- VM syslog/app logs
- On-prem + cloud hybrid
3) Unified event bus first
Send everything into a common stream layer first:
- Kafka / Confluent
- Kinesis
- Event Hubs
- Pub/Sub
Then route from there to your log store.
This is best when you want:
- replay
- transformation
- multiple consumers
- long-term flexibility
Practical cloud-by-cloud setup
AWS
- Enable CloudWatch Logs for EC2, EKS, Lambda, RDS, etc.
- Use:
- Subscription filters for real-time forwarding
- Kinesis Firehose for managed delivery
- Lambda for custom transforms
- Optional: CloudTrail, VPC Flow Logs, ALB/NLB logs → CloudWatch/S3 → export
Azure
- Enable Diagnostic settings on resources
- Route to:
- Event Hub for streaming
- Storage Account for archival
- Log Analytics for Azure-native analysis
- Common sources:
- Activity Logs
- Resource logs
- AKS container logs
GCP
- Use Cloud Logging sinks
- Route to:
- Pub/Sub for streaming
- BigQuery for analytics
- Cloud Storage for archive
- HTTPS/partner destinations via integrations
- Common sources:
- Audit logs
- GKE logs
- VPC Flow Logs
- App logs
Recommended design
If you want a robust multi-cloud setup, a strong default is:
AWS CloudWatch / Azure Monitor / GCP Cloud Logging → streaming layer (Kafka/Event Hubs/Pub/Sub bridge) → normalization pipeline → central log store
Or, if you want simpler operations:
Each cloud’s native logs → vendor integration/agent → central observability platform
Key design concerns
Normalize the schema
Different clouds label fields differently. Standardize:
- timestamp
- hostname / pod / instance
- service name
- environment
- region
- severity
- trace_id / span_id
- request_id
- cloud provider
A common schema like Elastic Common Schema (ECS) or OpenTelemetry Logs helps a lot.
Secure the transport
Use:
- TLS everywhere
- IAM/service principals
- least privilege
- private endpoints / peering where possible
- encryption at rest
Control cost
Logging gets expensive fast. Add:
- filtering at source
- sampling
- retention tiers
- archive to cheap storage
- route only what you need to hot search
Make routing resilient
Use:
- buffering
- retries
- dead-letter queues
- backpressure handling
Simple example
If your target is Elastic:
- AWS: CloudWatch Logs → Firehose → Elasticsearch/OpenSearch
- Azure: Diagnostic settings → Event Hub → Logstash/Fluent Bit → Elastic
- GCP: Cloud Logging sink → Pub/Sub → Dataflow/Logstash → Elastic
If your target is Splunk:
- AWS: CloudWatch → Kinesis Firehose → Splunk HEC
- Azure: Event Hub → Splunk Add-on / HEC forwarder
- GCP: Pub/Sub → Dataflow/forwarder → Splunk HEC
Easiest “one system” choices
If you want minimal glue code, pick one of these patterns:
- Datadog / New Relic / Splunk / Sumo: easiest multi-cloud ingestion
- OpenSearch / Elastic / Loki: more control, more setup
- Kafka + custom pipeline: most flexible, most ops
If you want, I can also give you:
- a reference architecture diagram,
- a Terraform-based setup for all three clouds, or
- a comparison of best log backends for this use case.
Brands