Prompt
How do I centralize logs across AWS, Azure, and GCP?
Latest observation
Oct 28, 2026
OpenAI APIWeb search: off
A good way to centralize logs across AWS, Azure, and GCP is to make one vendor-neutral log pipeline that collects from each cloud, normalizes the data, and stores it in a single searchable backend.
Common architecture
Cloud sources
- AWS: CloudTrail, CloudWatch Logs, VPC Flow Logs, ELB logs, EKS/container logs
- Azure: Azure Activity Logs, Azure Monitor / Log Analytics, NSG flow logs, AKS logs
- GCP: Cloud Audit Logs, Cloud Logging, VPC Flow Logs, GKE logs
Collection layer
- Use cloud-native exporters/forwarders or agents:
- Fluent Bit / Fluentd
- OpenTelemetry Collector
- Vendor connectors / sinks / subscriptions / event hubs / pub-sub / Kinesis / Service Bus
- Forward logs from each cloud into a central ingestion point or streaming bus.
Normalization layer
- Map logs into a common schema such as:
- Elastic Common Schema (ECS)
- OpenTelemetry semantic conventions
- A custom schema with fields like:
timestampcloud.providercloud.regionservice.namehost.nameseveritymessagetrace_id,span_idaccount/project/subscription id
Central storage/search Choose one:
- SIEM / observability platform: Splunk, Datadog, Elastic, Sumo Logic, New Relic, Grafana Loki
- Cloud-neutral storage + query: S3/ADLS/GCS + Athena/BigQuery/Synapse, or a lakehouse
- Security-focused: Microsoft Sentinel, Splunk ES, Elastic Security, Google Chronicle
A practical pattern that works well
Option 1: Push everything to a central SIEM
Best if you want fast deployment and unified search.
- AWS:
- CloudTrail → Kinesis Firehose / Lambda → SIEM
- CloudWatch Logs → subscription filter → Firehose/Lambda → SIEM
- Azure:
- Azure Monitor diagnostic settings → Event Hub → SIEM
- GCP:
- Cloud Logging sinks → Pub/Sub → Dataflow/forwarder → SIEM
Pros:
- Fast to implement
- Good for security and compliance
- Unified dashboards and alerting
Cons:
- Can get expensive at high volume
- Some schema/cost tuning needed
Option 2: Central log lake
Best for large volumes and cost control.
- Send all logs to object storage:
- AWS S3
- Azure Blob Storage / ADLS Gen2
- GCS
- Use a common format:
- JSON, Parquet, or Avro
- Query with:
- Athena, Synapse serverless, BigQuery, Trino, Databricks
Pros:
- Lower cost for long retention
- Flexible analytics
- Good for audit/compliance archives
Cons:
- More engineering effort
- Less “real-time” unless you add streaming/search
Option 3: Hybrid
Best overall for many organizations.
- Real-time/security logs → SIEM
- Bulk/long-term logs → data lake
- Use the same normalization pipeline for both
Recommended implementation steps
-
Define log scope
- Security, infrastructure, app, audit, network, Kubernetes?
- Decide retention and compliance requirements.
-
Pick a common schema
- ECS or OpenTelemetry is a strong default.
-
Create cloud-specific collection
- AWS: CloudWatch subscription filters, CloudTrail to S3/Kinesis
- Azure: Diagnostic settings to Event Hub
- GCP: Log sinks to Pub/Sub
- Add agents for VM/container/app logs if needed.
-
Use a central ingestion pipeline
- OpenTelemetry Collector, Fluent Bit, or a streaming ETL layer
- Enrich with metadata:
- cloud, account/project, region, env, app, team
-
Store in your central platform
- SIEM for search/alerts
- Data lake for retention/analytics
- Ensure encryption, RBAC, and tenant separation
-
Build dashboards and alerts
- Cross-cloud incident views
- Authentication failures
- Network anomalies
- Policy/compliance violations
-
Governance
- Normalize time zones/UTC
- Deduplicate if needed
- Control PII/secrets
- Define retention and lifecycle policies
Tools people often use
Collection/forwarding
- Fluent Bit
- OpenTelemetry Collector
- Vector
- Logstash
- Beats
Central platforms
- Splunk
- Elastic
- Datadog
- Sumo Logic
- Grafana Loki
- Microsoft Sentinel
- Google Chronicle
Storage/query
- S3 + Athena
- ADLS + Synapse
- GCS + BigQuery
- Databricks / Snowflake / Trino
Key design tips
- Use UTC everywhere
- Preserve raw logs as well as normalized records
- Tag every log with cloud/account/project/subscription
- Separate security logs from app noise
- Plan for cost: logs can explode in volume
- Watch egress charges when moving data cross-cloud
- Automate onboarding for new accounts/projects/subscriptions
If you want a simple recommended stack
A common, solid setup is:
- Fluent Bit or OpenTelemetry Collector in each cloud
- Cloud-native sinks:
- AWS → Kinesis/EventBridge/S3
- Azure → Event Hub/Log Analytics export
- GCP → Pub/Sub
- Central backend:
- Elastic, Splunk, or Grafana Loki
- Long-term archive:
- Object storage + Parquet
If you want, I can also give you:
- a reference architecture diagram,
- a vendor-neutral implementation plan, or
- a step-by-step setup for AWS + Azure + GCP.
Brands