streaming with exactly once semantics

Exactly-once semantics in streaming means:
each event is processed one time and only one time, even if there are retries, failures, or restarts.

In practice, it’s hard to guarantee perfectly end-to-end, so systems usually provide one of these delivery guarantees:

• At-most-once: an event may be lost, but won’t be duplicated.
• At-least-once: an event won’t be lost, but may be duplicated.
• Exactly-once: no loss and no duplicates are visible to the final outcome.

How it’s usually achieved

Exactly-once streaming typically relies on a combination of:

1. Checkpointing / state snapshots
   The streaming job periodically saves its state so it can recover after failure.

2. Idempotent processing
   Reprocessing the same event does not change the result more than once.

3. Transactional writes / two-phase commit
   Output is committed only when the corresponding input processing is safely recorded.

4. Offset tracking / source coordination
   The system remembers which input records were already consumed and processed.

Important caveat

In many real systems, “exactly once” usually means:

• exactly-once processing within the stream processor, or
• exactly-once effects when writing to supported sinks,

not necessarily that every external side effect is magically deduplicated.

For example, writing to:

• a database,
• an email service,
• an HTTP API,

may still require idempotency keys or transactional support to avoid duplicates.

Example

If a stream processor reads message A, updates state, and writes an aggregate:

• if it crashes after reading A but before committing, it may read A again on restart;
• with exactly-once support, the update from A is not applied twice in the final state/output.

Common technologies

Systems that support exactly-once semantics in some form:

• Apache Flink
• Kafka Streams
• Spark Structured Streaming (in many cases)
• Apache Beam depending on runner/sink support

If you want, I can also explain:

1. exactly-once vs at-least-once vs idempotency, or
2. how Kafka/Flink implement exactly-once.