Prompt
I'm building a multi-cloud service and want centralized logs across AWS, Azure, and GCP
Latest observation
A good pattern is to centralize log collection in a cloud-agnostic observability backend and use each cloud’s native export/forwarding mechanisms to ship logs there.
Recommended architecture
1) Collect locally in each cloud
Use each provider’s native logging service as the source:
- AWS: CloudWatch Logs, CloudTrail, VPC Flow Logs
- Azure: Azure Monitor / Log Analytics, Activity Logs, Diagnostic Settings
- GCP: Cloud Logging, Audit Logs, VPC Flow Logs
2) Forward to a central destination
Common central targets:
- OpenSearch / Elasticsearch
- Splunk
- Datadog
- Grafana Loki
- Elastic Cloud
- A SIEM like Microsoft Sentinel or Chronicle if security-centric
- Object storage + processing pipeline (S3/GCS/Blob) if you want lower cost and batch analytics
3) Normalize the data
Make sure logs from all clouds are transformed into a common schema, ideally ECS, OpenTelemetry logs, or your own standard fields:
- timestamp
- service name
- environment
- cloud provider
- region
- account/subscription/project
- request ID / trace ID
- severity
- message
- user / principal
- resource IDs
4) Correlate with traces and metrics
If possible, add:
- trace IDs
- correlation IDs
- span IDs
- deployment/version tags
This makes cross-cloud debugging much easier.
Practical implementation options
Option A: Use OpenTelemetry + a log pipeline
- Run OTel Collector in each environment
- Send logs via OTLP or vendor-specific exporters
- Normalize and forward to your backend
Best if you want portability and control.
Option B: Native exports to a central SIEM/search platform
- AWS subscription filters / Kinesis Firehose
- Azure diagnostic settings to Event Hub or storage
- GCP log sinks to Pub/Sub or storage
- Then ingest into Splunk/Elastic/Datadog
Best if you want to use cloud-native collection with centralized analytics.
Option C: Centralize in object storage first
- Export all logs to S3 / Blob / GCS
- Use a batch ETL job to normalize and query them
Best for cost-sensitive archival and compliance, but weaker for real-time debugging.
Important design considerations
Access and security
- Use least privilege IAM/service principals
- Encrypt logs in transit and at rest
- Separate prod/non-prod
- Protect against log poisoning and sensitive data leakage
Retention and cost
- Hot/warm/cold tiering
- Short retention for verbose app logs
- Longer retention for audit/security logs
- Sample high-volume debug logs
Standardization
Define a common log format early:
- JSON structured logs
- consistent severity levels
- standardized metadata fields
Multi-cloud metadata
Always include:
cloud.providercloud.regioncloud.account_id/ subscription / projectk8s.clusterif applicableservice.namedeployment.environment
A simple reference design
AWS / Azure / GCP apps → native log service → export agent/sink → central Kafka / Event Hub / Pub/Sub bridge or collector → OpenSearch / Loki / Splunk / Datadog → dashboards + alerting + retention
If you want the simplest operational setup
For most teams, I’d recommend:
- Structured JSON logs everywhere
- OpenTelemetry Collector in each cloud
- Central backend like Datadog, Splunk, or Elastic
- Common schema with cloud and trace metadata
If you want, I can also give you:
- a reference architecture diagram
- a cloud-by-cloud implementation plan
- or a comparison of Splunk vs Elastic vs Loki vs Datadog for multi-cloud logs.