Are there any message queue backed stores that handle ephemeral workload spikes without heavy persistence overhead?

Yes — if your goal is to absorb short-lived spikes without paying the cost of a fully durable, heavily persisted queue/store, there are several good patterns and products.

Common options

1. In-memory queues / brokers

Best when you want very low latency and can tolerate message loss on crash.

• Redis Streams / Lists

  • Fast, simple, often used as a lightweight buffer.
  • Can be configured with persistence, but you can also run it mostly as an in-memory spike absorber.
  • Good for ephemeral work if losing some messages is acceptable.

• RabbitMQ with transient queues/messages

  • Supports non-durable queues and transient messages.
  • Reduces persistence overhead a lot.
  • Good for workload smoothing, but not for strong durability guarantees.

• NATS / NATS JetStream

  • Core NATS is very lightweight and ephemeral.
  • JetStream adds persistence, but you can choose how durable you want it to be.
  • Excellent for bursty, low-latency systems.

• Memcached-style work buffers aren’t common as queues, but can be used for short-lived coordination in some architectures.

2. Log-based systems with tunable durability

These still persist, but can be configured to minimize overhead.

• Kafka

  • Usually not what you want if your main goal is “avoid persistence overhead.”
  • But it handles spikes very well.
  • You can tune retention, replication, and producer acks to reduce cost.
  • Better for high throughput than for “ephemeral only.”

• Pulsar

  • Similar story: durable by design, but flexible.
  • Usually overkill if you truly want a lightweight ephemeral buffer.

3. Managed cloud queues with short retention

If you want ephemeral behavior without operating infra:

• AWS SQS

  • Messages can expire quickly via retention settings.
  • Still a durable service, so persistence overhead is not zero.
  • Good operationally, but not “lightweight.”

• Google Pub/Sub / Azure Service Bus

  • Also durable, but can be used for burst absorption with short retention.

4. Spill-to-disk / hybrid systems

If you need to handle spikes but only persist when necessary:

• Redis with maxmemory policies

  • Keep hot burst data in memory, evict under pressure.
  • Can be useful when you’d rather drop than persist.

• NATS JetStream / RabbitMQ quorum vs classic queues

  • Classic/non-durable modes can reduce overhead.
  • Hybrid designs often balance memory and disk carefully.

---

If your requirement is specifically:

“Handle ephemeral workload spikes without heavy persistence overhead”

The most relevant choices are usually:

1. NATS core — if you can tolerate at-most-once semantics.
2. RabbitMQ non-durable queues/messages — if you want brokered queuing with reduced persistence.
3. Redis Streams/lists — if you want a simple buffer and fast processing.
4. In-memory queue with backpressure + optional loss — if the spike is truly temporary.

---

Rule of thumb

Choose based on whether losing messages is acceptable:

• Loss acceptable → in-memory / non-durable broker
• Loss not acceptable → you’ll need some persistence, even if tuned lightly
• Very high burst + durability → Kafka/Pulsar/SQS style systems

If you want, I can recommend a specific queue/store based on:

• expected message volume,
• acceptable loss rate,
• single-node vs clustered,
• and whether you need ordering or consumer replay.