Pipe-and-Filter Architecture: Composable Transformation at Any Scale

The shape

A pipe-and-filter system is a directed graph of filters — independent components that read from an input, transform, and write to an output — connected by pipes, which carry data and carry nothing else. A filter never knows what is upstream or downstream. It knows two contracts.

A canonical ingestion pipeline. Any stage can be scaled, replaced or reordered independently.

Because the coupling is only in the data contract, you can insert a new stage without touching its neighbours, run ten copies of the slow stage, or swap a stage's implementation language entirely.

Kinds of filters

Filter type	Behaviour	Example
Producer / source	Emits, does not consume	Kafka consumer, file reader, API poller
Transformer	1 in → 1 out	Parse JSON, normalise units, redact PII
Tester / filter	1 in → 0 or 1 out	Drop malformed records, sample 1%
Splitter	1 in → N out	Explode a batch into records; fan out formats
Aggregator	N in → 1 out	Window sums, join by key, batch for a sink
Consumer / sink	Consumes, does not emit	Write to warehouse, publish, notify

Design rule

Keep filters pure and single-purpose. A filter that both enriches and writes to a database cannot be reordered, retried or tested independently — and you have lost the entire benefit.

Pipes: the underrated part

The pipe determines almost every operational property of the pipeline. The filter code looks the same in each case; the system does not.

Pipe	Durability	Coupling	Typical use
Function call / iterator	None	Same process	Compilers, in-app transforms
Unix pipe / stdio	None	Same host	Shell pipelines, batch tooling
In-memory channel	None	Same process, concurrent	Go channels, reactive streams
Durable queue	Yes, ephemeral after ack	Cross-service	Work distribution, ETL steps
Log (Kafka-style)	Yes, replayable	Cross-service	Stream processing, event pipelines
Object storage handoff	Yes, cheap and large	Cross-service, batch	Media, big-file ETL

Choosing a durable log as the pipe is what turns a fragile script into a production pipeline: it decouples stage availability, absorbs bursts, and makes reprocessing possible.

Backpressure and buffering

Every pipeline has one slowest stage. What happens to the data arriving faster than that stage can process is the central operational question.

Blocking backpressure: the pipe refuses writes until there is room, and the slowdown propagates back to the source. Correct and simple — the default for in-process reactive streams.
Buffered: a bounded queue absorbs bursts. Choose the bound deliberately; unbounded buffers turn a throughput problem into an out-of-memory crash.
Load shedding: drop or sample when the buffer is full. Legitimate for metrics and telemetry, unacceptable for payments.
Scale the slow stage: the real fix when the pipe is a partitioned log — add consumers, one partition each.

per stage:  input rate, output rate, lag (queue depth), p99 duration, error rate

pipeline health = max(lag) across stages, and whether it is growing
a stage whose lag grows monotonically is the bottleneck — everything else is noise

Measure the pipeline by its stages, not as a whole.

Delivery guarantees and replay

At-least-once is the practical default. A stage crashes after writing output but before acknowledging input, so the record reappears. Make every stage idempotent on a record key.
Exactly-once is achievable per-framework (transactional offsets plus idempotent sinks) but only inside the framework's boundary — the external sink still needs an idempotency key.
Dead-letter every stage. A single malformed record must not stop the pipeline; route it aside with the error and the original payload.
Design for replay from day one. Immutable raw input plus deterministic filters means a bug is fixed by reprocessing, not by manual data repair.

Stateful stages and windows

Pure transformation is easy. Aggregation is where pipelines get interesting, because you must decide when a window is complete in a world where records arrive late and out of order.

Event time vs processing time. Aggregate by when the event happened, not when you received it, or every network hiccup corrupts your numbers.
Watermarks declare "we believe all events before T have arrived" and trigger window emission.
Allowed lateness plus an update or side-output path for stragglers.
State stores must be checkpointed, or a restart loses partial aggregates.

Where you already use it

Unix: cat access.log | grep 500 | awk '{print $7}' | sort | uniq -c — the archetype.
Compilers: lex → parse → typecheck → optimise → emit.
ETL/ELT: extract → clean → conform → load → model.
Media: demux → decode → filter → encode → mux → package.
Stream processing: Kafka Streams, Flink, Spark Structured Streaming.
Web middleware chains and CI pipelines — same structure, different data.

Limits

Poor fit for interactive, low-latency request-response work — each stage adds a hop.
Bad when stages need a shared global view of state mid-stream.
Serialisation between stages can dominate cost for high-volume small records; batch.
End-to-end debugging needs a correlation ID threaded through every stage.
Long pipelines have a cumulative failure probability — measure end-to-end, not per stage.

Interview framing

Strong answer

"I'd model ingestion as pipe-and-filter with a partitioned log between stages: ingest writes raw immutable events, then parse, validate, enrich and aggregate each run as independent consumer groups. That lets me scale enrichment — the slow stage, since it calls an external API — to twenty consumers without touching anything else, and replay from raw when we fix an enrichment bug. Every stage is idempotent on event ID with a dead-letter topic, and I aggregate on event time with watermarks and five minutes of allowed lateness. The metric I alert on is per-stage consumer lag."

Follow-ups

What happens when one stage is 10× slower than the rest?
How do you reprocess three days of data without double-counting?
Where would a pipeline be the wrong choice?