Are there any autoscaling workers that handle bursty pipeline execution without high idle costs?

Yes — but the best option depends on what you mean by “pipeline” and where it runs.

If you want bursty workers with low idle cost, look for systems that support scale-to-zero or on-demand job workers. Common patterns:

Good fits

• Kubernetes + KEDA
  • Autoscale worker deployments based on queue depth, Kafka lag, HTTP metrics, etc.
  • Can scale down to zero in some setups.
• Cloud Run / AWS App Runner / Azure Container Apps
  • Great for containerized jobs or services that wake up on demand.
  • Usually better for stateless work than long-running pipeline executors.
• AWS Batch / Google Cloud Batch / Azure Batch
  • Designed for queued compute jobs.
  • Spawns compute only when jobs exist, so idle cost is low.
• Serverless queues + functions
  • For event-driven pipeline steps, e.g. SQS + Lambda, Pub/Sub + Cloud Functions.
  • Very cost-efficient for short tasks.
• Prefect / Dagster / Airflow with ephemeral workers
  • These orchestration tools can run tasks on ephemeral containers or Kubernetes jobs.
  • The scheduler stays up, but workers can scale with demand.
• Temporal workers on autoscaled containers
  • Good if your “pipeline” is really workflow execution with retries/state management.
  • Workers can be scaled based on task backlog.

If you need high burst throughput

A common architecture is:

1. Put work items on a queue
2. Run ephemeral workers in containers or batch jobs
3. Autoscale by queue depth / lag
4. Scale back to zero when the queue is empty

This avoids paying for idle workers while still handling spikes.

Things to watch

• Cold start latency if scaling from zero
• Long-running tasks may be better on batch or Kubernetes jobs than serverless functions
• Stateful pipelines may need orchestration support, not just autoscaling workers
• External dependencies like databases can become the real bottleneck during bursts

If you tell me:

• your cloud/provider,
• whether tasks are short or long,
• and whether you’re using Airflow/Dagster/Prefect/Temporal/Kubeflow/etc.,

I can suggest the most cost-efficient autoscaling setup.