Durable Subscriptions and Consumer Checkpoints
A durable subscription gives a named consumer a server-tracked position in the event log. After reconnect, event store replays committed events after last acknowledged position, then switches consumer to live delivery. Processing remains at least once, so handlers must be idempotent.
Problem
Why this pattern exists
Ephemeral pub/sub drops events while consumer is offline. Client-managed offsets scatter recovery state across services and can advance past failed work. Durable server-side cursor makes restart behavior explicit, but still needs single ownership, acknowledgement policy, filters, lag monitoring, and poison-event handling.
Consumer name is stateful identity, not cosmetic label. Reusing one name for two replicas can race cursor updates; reusing after reducer change can skip history new version needs. Stable name per logical consumer version turns reset, replay, blue-green migration, and operational ownership into manageable choices.
Design decisions
Make boundaries explicit
- 01
Name consumer by output contract
Include purpose and version, such as search_index_v3. New reducer or destination schema receives new identity so it can replay independently beside current consumer.
- 02
Acknowledge in batches deliberately
Small interval reduces replay after crash but increases checkpoint writes. Large interval improves throughput but expands duplicate window. Tune against processing cost and recovery target.
- 03
Design lag and poison policy
Measure distance from live head, cap retry loops, and preserve failed event evidence. Pause or dead-letter deliberately rather than acknowledge malformed facts silently.
AllSource implementation
Apply pattern to durable Core history
AllSource Core durable-consumer protocol registers consumers, delivers replay frames over WebSocket, and accepts acknowledgement positions through consumer endpoint. Cursor metadata is event-sourced through Core system streams, so service caches can rebuild after restart. Reconnection uses stored position rather than process memory.
Rust ProjectionWorker wraps protocol with event filters, checkpoint interval, exponential reconnect backoff, replay-to-live transition, and per-entity version dedup. It is single-instance per consumer name; shard with explicit distinct identities and partition-aware reducers. Watch replay count, checkpoint age, reconnect rate, lag notices, reducer latency, and current position versus Core head.
POST /consumers -> register search_index_v3
connect WebSocket ?consumer_id=search_index_v3
receive replay positions 81200..81500
POST /consumers/search_index_v3/ack { position: 81500 }
receive replay_complete
continue with live committed eventsFailure modes
Detect weak implementations early
Two workers with same identity move one cursor unpredictably.
Fix: Run single owner per name or implement explicit partitioned consumer identities.
Slow reducer falls behind live broadcast repeatedly.
Fix: Measure lag, optimize or shard reducer, and rely on replay after reconnect.
Bad event retries forever without operator visibility.
Fix: Expose failure, retain payload reference, and define pause, fix, replay, or dead-letter policy.
Production checklist
Ready when each statement is true
- Consumer ID names logical output and reducer version.
- One active owner or explicit sharding model exists.
- Acknowledgement interval matches replay tolerance.
- Handlers are idempotent under duplicate delivery.
- Lag, reconnect, error, position, and replay metrics are visible.
Related patterns
Continue through adjacent decisions
Idempotent consumers
An idempotent event consumer produces the same durable result when it receives an event more than once. Use stable event IDs, entity versions, unique effect keys, and acknowledge only after processing succeeds; at-least-once delivery then becomes recoverable instead of corrupting state.
Read nextProjections and read models
A projection folds ordered events into a query-specific read model. It is derived state, not a second source of truth: operators must be able to discard it, replay source events, and build a replacement without rewriting event history or stopping existing readers.
Read nextSnapshots and checkpoints
A snapshot stores derived aggregate state at a known point in its history; a checkpoint stores how far a consumer processed the event log. Both speed recovery, but neither replaces source events. Snapshot validity depends on reducer compatibility, while checkpoint validity depends on consumer identity and delivery semantics.
Read nextStore history once