How AllSource Core Works: WAL, Parquet, and DashMap

AllSource Core is a purpose-built event store written in Rust. Published reference benchmarks report 469K events/sec on the batch-ingest path and 11.9us p99 on the indexed-read path. Those paths use a write-ahead log for recovery, Parquet files for long-term persistence, and a concurrent in-memory map for reads. This post explains each layer and its trade-offs.

The three-layer storage model

Write path:  Event → WAL (fsync) → DashMap (memory) → Parquet (periodic flush)
Read path:   Query → DashMap (11.9us) → done
Recovery:    Startup → Parquet (bulk load) → WAL (replay delta) → DashMap (ready)

Accepted events enter the WAL and in-memory map before periodic Parquet checkpoints. The configured fsync policy determines when WAL bytes are forced to disk; the in-memory concurrent map serves hot reads; Parquet provides compact columnar persistence.

Core is not in-memory only: startup loads Parquet and replays valid WAL entries. Process-crash and power-loss behaviour still depends on fsync configuration, storage, and replication.

Layer 1: Write-Ahead Log (WAL)

The WAL is the durability guarantee. When an event arrives:

1. Serialize the event to bytes
2. Compute a CRC32 checksum over the bytes
3. Write [length][checksum][bytes] to the WAL file
4. Apply the configured fsync interval
5. Make the accepted event available to the in-memory read path

The CRC32 checksum detects corruption — if a WAL entry's checksum doesn't match its payload during recovery, we skip the corrupted entry and log a warning. This catches bit-rot, partial writes from crashes, and filesystem corruption.

// Simplified WAL write (actual code in infrastructure/persistence/wal.rs)
pub fn append(&mut self, event: &Event) -> Result<()> {
    let bytes = bincode::serialize(event)?;
    let checksum = crc32fast::hash(&bytes);
    let len = bytes.len() as u32;
 
    self.file.write_all(&len.to_le_bytes())?;
    self.file.write_all(&checksum.to_le_bytes())?;
    self.file.write_all(&bytes)?;
 
    // fsync on interval, not every write — 100ms default
    if self.should_sync() {
        self.file.sync_data()?;
    }
    Ok(())
}

The fsync interval is a durability/performance trade-off:

• 100ms (default): at most 100ms of events lost on power failure. Good enough for most use cases.
• 0ms (every write): strongest configured acknowledgement boundary, with materially lower throughput. Hardware and operating-system failure modes still apply.
• 1s: higher throughput, up to 1s of potential loss.

For financial use cases or audit trails, set ALLSOURCE_FSYNC_INTERVAL=0. For IoT telemetry where occasional loss is acceptable, 1s is fine.

Layer 2: DashMap (in-memory concurrent reads)

After the WAL write, the event is inserted into a DashMap — a lock-free concurrent hash map from the dashmap crate. This is where reads come from.

DashMap uses sharded locks internally, allowing reads and writes on different shards to proceed concurrently. The published reference benchmark measured 11.9us p99 indexed reads on its stated hardware; end-to-end API, time-travel, graph, and vector queries have different paths.

// Simplified query (actual code in store.rs)
pub fn query(&self, filter: &QueryFilter) -> Vec<Event> {
    self.events
        .iter()
        .filter(|e| filter.matches(e))
        .take(filter.limit)
        .map(|e| e.clone())
        .collect()
}

The trade-off: all events must fit in memory. With 164K events on a 1GB VM (our current production deployment), each event averages ~6KB including metadata. For a million events, you'd need ~6GB of RAM. This is why AllSource's pricing tiers are event-count-based — the cost of the service is proportional to the memory required.

Layer 3: Parquet (columnar persistence)

Periodically (default: every 4 hours or 10K events), the event store flushes a Parquet checkpoint:

1. Snapshot all events in the current window
2. Write them to a Parquet file with Snappy compression
3. Truncate the WAL (events are now safely in Parquet)

Parquet files are columnar — meaning queries that filter on event_type or tenant_id only read those columns, not the entire event payload. This makes analytical queries fast even on large datasets.

At startup, Core loads events from Parquet first (bulk load), then replays any WAL entries that were written after the last checkpoint. This gives you fast recovery: the Parquet load is a single sequential read, and the WAL replay is typically small (only events since the last checkpoint).

Why not PostgreSQL?

We get this question a lot. Here's the concrete comparison:

PostgreSQL is a general-purpose database designed for mutable state. AllSource Core is a purpose-built event store designed for append-only, immutable events. The append-only constraint lets us make optimizations that PostgreSQL can't:

• No MVCC overhead: events are immutable, so we never update or delete
• Purpose-built hot index: entity lookups use the in-memory concurrent map rather than a general relational query plan
• No vacuum: no dead tuples, no bloat, no autovacuum pauses

The durability guarantee

When POST /api/v1/events accepts an event, it is:

1. Appended to the WAL with a CRC32 checksum
2. In the DashMap (queryable immediately)
3. Pending Parquet flush (will be checkpointed within the configured interval)

WAL replay recovers entries that reached durable storage. With interval-based fsync, the newest accepted writes can remain in the operating system's buffers until the next sync; use per-write fsync when that loss window is unacceptable. Replication and backups address disk or volume failure.

We run a durability test that writes events, kills the process, restarts, and verifies all events survived. It passes. Every time.

What this means for your agent

If you're using AllSource as memory for an AI agent:

• Accepted history is recoverable according to your durability policy. WAL and Parquet restore the event log after restart.
• Indexed reads are fast. The published Core benchmark measured 11.9us p99 for its indexed read path; reconstructive and semantic queries should be measured separately.
• Audit evidence has a durable base. Timestamps, checksums, and event metadata can support controls, but do not establish compliance on their own.

This is why we say AllSource Core IS the database. It's not a cache in front of PostgreSQL. It's not an in-memory store that loses data on restart. It's a purpose-built, durable, high-performance event store.

Start at all-source.xyz or read the API docs.