Self-hosted Core capabilityv0.5+

Transform event streams beside the source history.

AllSource Core matches accepted event types and runs ordered Filter, Map, Reduce, Window, and Branch operators in process. Source events stay unchanged; pipeline counters remain inspectable.

Current boundary: results are not persisted or published automatically, definitions and state are in memory, and hosted gateway routes are not exposed.

Inspect Core implementation Core API reference

Core ingestion path

Accepted source event

order.placed

01 matchorder.placed
02 filterstatus = paid
03 windowtumbling · 86400s
04 reducesum total · currency

Computed result

in process

{ "group": "GBP", "value": 4827 }

Source event

Stored

Counters

Tracked

Result sink

Not automatic

Execution contract

Know where state lives.

Where: Inside AllSource Core, on each accepted event that matches `source_event_types`.
State: Reduce values, window buffers, definitions, and counters remain in memory.
Restart: Register pipelines again; replay source events when derived state must be rebuilt.

Ordered operators

Each result feeds the next stage.

Operators execute in array order. Filter can halt processing; stateful operators keep values inside that pipeline instance.

01
Filter
Available
Keep or drop an event by field value using eq, ne, gt, lt, or contains.
field: "status" · op: "eq" · value: "paid"
02
Map
Available
Update one field with uppercase, lowercase, trim, multiply, or add transforms.
field: "currency" · transform: "uppercase"
03
Reduce
Available
Maintain count, sum, average, minimum, or maximum state with optional grouping.
function: "sum" · group_by: "currency"
04
Window
Available
Run a nested aggregation over tumbling, sliding, or session windows.
tumbling · sliding · session
05
Branch
Available
Map a string field value to a route name stored on the computed result.
field: "region" · branches: { "uk": "gbp" }
06
Enrich
Scaffold only
Operator shape exists, but external lookups are not wired. Current Core code emits placeholder values.
Do not use for production joins yet

Valid Core payload

Register a typed pipeline definition.

This example matches the current Rust `PipelineConfig` serde shape, including required `id` and `output` fields and nested window aggregation.

daily-revenue.jsonPOST /api/v1/pipelines

{
  "id": "7af7b1c8-6165-4c42-9081-e9d20d811780",
  "name": "daily-revenue",
  "description": "Paid revenue grouped by currency",
  "source_event_types": ["order.placed"],
  "operators": [
    {
      "type": "filter",
      "field": "status",
      "value": "paid",
      "op": "eq"
    },
    {
      "type": "window",
      "config": {
        "window_type": "tumbling",
        "size_seconds": 86400,
        "slide_seconds": null,
        "session_timeout_seconds": null
      },
      "aggregation": {
        "type": "reduce",
        "field": "total",
        "function": "sum",
        "group_by": "currency"
      }
    }
  ],
  "enabled": true,
  "output": "daily-revenue-by-currency"
}

Lifecycle endpoints

Operate through Core.

These routes live on Core's internal server, normally port 3900. Keep Core behind your network boundary.

POST/api/v1/pipelines

GET/api/v1/pipelines

List registered definitions

GET/api/v1/pipelines/stats

Read processing statistics

GET/api/v1/pipelines/{id}

Read one definition

GET/api/v1/pipelines/{id}/stats

Read one pipeline's statistics

PUT/api/v1/pipelines/{id}/reset

Clear operator state and counters

DELETE/api/v1/pipelines/{id}

Remove a definition

`output` is currently descriptive configuration. It does not create a projection, write another event, or publish a topic.

System boundary

Pick by delivery requirement.

“Stream processing” covers several jobs. AllSource Core owns one of them: ordered computation beside accepted history.

Core pipelines

Use for inline filtering, field transforms, aggregation, and routing calculations beside source events.

Understand Core

Query Service WebSockets

Use for live accepted-event feeds to clients. These feeds do not publish Core pipeline results.

See real-time analytics

Replay and projections

Use source events to rebuild queryable state. This remains separate from a pipeline's configured output name.

See replay use case

Kafka, Redpanda, or Flink

Use when transport, durable output topics, connectors, consumer offsets, or independent scaling are primary.

Direct answers

Stream-processing questions

What is AllSource stream processing?

AllSource Core can match accepted event types and run an ordered, in-process pipeline of Filter, Map, Reduce, Window, Branch, and Enrich operators. Source events remain unchanged. Enrich external lookups and automatic output persistence are not complete today.

Are AllSource pipeline results persisted automatically?

No. Core computes matching pipeline results in process and tracks processing statistics, but the configured output name does not currently persist a projection or publish a topic. Applications that need a durable output must add that integration.

Do AllSource pipelines survive a Core restart?

Not yet. Pipeline definitions, window buffers, reduction state, and counters live in memory. Register definitions again after restart and rebuild any required state from source events.

Can hosted AllSource customers configure pipelines through the public gateway?

Not currently. Pipeline lifecycle endpoints exist on the self-hosted Core server. The hosted public gateway does not expose those routes, so this page treats stream processing as a self-hosted Core capability.

Does AllSource stream processing replace Kafka or Flink?

No. Core pipelines suit transformations that should execute beside the event store. Use Kafka, Redpanda, Flink, or another stream platform when you need cross-service transport, independent scaling, connector ecosystems, durable output topics, or separate consumer offsets.

Build from evidence

Test your own pipeline against retained events.

Published 469K events/sec result measures Core batch ingestion, not arbitrary operator chains. Benchmark your definition, durability mode, and hardware.

Self-host Core Read benchmark method