How I traced backpressure to the actual bottleneck in a real-time pipeline

A real-time system can finish a test with zero data loss and still fail the test.

I ran into exactly that while working on a real-time telemetry pipeline. The system consumed a large stream of events over many WebSocket connections, persisted raw data, parsed and normalized messages, updated internal state, and passed events through downstream processing.

One longer test looked almost ideal:

  • all 14,558 events I was tracking were delivered;
  • zero were lost;
  • the writer was clearly not saturated;
  • output finalized cleanly.

The run still failed with:

raw_ingest_queue_full

Nothing had been dropped yet, but the system was already telling me something important:

this workload was not sustainable.

It took several iterations to work through the problem. The most useful result was not one particular optimization. It was a better way to reason about backpressure without optimizing the wrong stage.

The pipeline

At a high level, the data path looked like this:

WebSocket connections
        ↓
raw ingest queues
        ↓
parsing / normalization
        ↓
state processing
        ↓
observer
        ↓
writer

The actual system had multiple sources and message types, dozens of connections, bounded queues, telemetry, compression, and state updates.

One early broad test opened roughly 50 WebSocket connections and handled 1,676 subscriptions while using most of the available capacity on a four-core host.

The first run failed during shutdown. The pipeline could not drain cleanly and left partial artifacts behind.

After fixing shutdown and finalization, the next run looked much healthier:

  • no partial files;
  • zero write failures;
  • zero dropped events.

Then, about 21 seconds in, it stopped again:

raw_ingest_queue_full

This was a different failure. Shutdown was fixed. The pipeline simply could not consume the incoming stream fast enough.

Why I did not just make the queue bigger

When a bounded queue fills up, increasing its size is an obvious response:

queue_size *= 10

That can help with a short burst.

It does not solve a sustained condition where:

producer_rate > consumer_rate

A larger queue only delays the failure while allowing more backlog and latency to accumulate.

For a real-time system, this can be worse than an immediate failure. The events may still be present, but they are increasingly stale.

data loss = 0
latency → ∞

Technically complete. Operationally no longer real-time.

So instead of changing capacity, I wanted to know which stage was actually falling behind.

The queue that fills is not necessarily the root cause

raw_ingest_queue_full tells you where backpressure became visible.

It does not tell you what caused it.

The slow stage might be parsing, normalization, state processing, the observer, the writer, or simply CPU contention across several stages.

The next change therefore was not an optimization. It was instrumentation.

I started measuring, per stage and where useful per connection:

  • queue capacity;
  • maximum backlog;
  • peak utilization;
  • producer and consumer rates;
  • produced and consumed event counts.

That led to a rule I now use regularly:

If you cannot tell which stage is accumulating backlog, it is too early to optimize.

My first hypothesis was downstream pressure

The observer and writer were natural suspects. The writer performs I/O, so it was easy to assume that persistence was slowing down the pipeline.

After some changes, the system ran much longer. One important run lasted roughly 17 minutes.

During that run:

delivered = 14,558
lost      = 0

The writer queue peaked at only:

8.86%

The observer also had plenty of headroom.

Yet the run still ended with:

raw_ingest_queue_full

Some raw queues had reached:

100%

That changed the investigation.

The downstream path was no longer the problem. Removing one bottleneck had exposed another one earlier in the pipeline, around raw ingestion and parsing.

A new bottleneck after a fix can be good information

Suppose a pipeline looks like:

A → B → C → D

If D handles 1,000 events per second while the other stages handle 10,000, D is the visible bottleneck.

After optimizing D, you may discover that B can only sustain 4,000 events per second.

B did not get slower.

Its limit was simply hidden before.

That was the pattern here. Once observer/writer pressure was no longer dominant, the capacity boundary moved upstream to the raw/parser path.

Why zero data loss was still a FAIL

If:

delivered = 14,558
lost      = 0

why fail the run?

Because zero loss describes what has happened so far.

A saturated bounded queue tells you what happens next if the same load continues.

A capacity test therefore cannot stop at:

lost_events == 0

It also needs signals such as:

queue utilization
producer/consumer balance
freshness
bounded latency

In this case, 100% raw queue utilization was enough to fail closed even though no protected events had been lost.

Moving upstream to the parser path

Per-connection telemetry now showed a much clearer picture:

observer queue ≈ mostly idle
writer queue   ≈ mostly idle
raw queue      = 100%

The next place to investigate was the work between raw ingestion and the following stage.

The candidates included:

  • parsing;
  • lock acquisition;
  • telemetry accounting;
  • frequent queue.qsize() calls;
  • per-event statistics updates;
  • processing batches that were too small.

Each operation may be inexpensive in isolation. Inside a hot loop executed hundreds of thousands of times, small costs become throughput limits.

A small experiment instead of a large rewrite

At this point I did not want to redesign the pipeline around multiprocessing or split every stage into separate processes.

That would have been a large architectural change before the bottleneck was fully understood.

Instead, I made a much smaller change:

  • reduced parser-side accounting overhead;
  • batched some telemetry updates;
  • removed unnecessary work from the hot path;
  • left queue capacity unchanged.

Local tests passed. Then I ran a 180-second smoke test against the real data stream.

The result was very different:

early termination: none

raw queue peak:      14.8%
observer queue:       4.87%
writer queue:         9.74%

Produced and consumed state-event counts matched:

317,329
317,329

For the event stream I was explicitly protecting:

delivered: 2,537
lost:          0

Before the change, raw queues had reached 100%.

After it, the peak was 14.8%.

That mattered much more than getting another lost=0.

The system now had capacity headroom.

Why headroom matters more than a nice zero-loss number

Compare:

loss                0
queue utilization 100%

with:

loss                0
queue utilization  15%

Both runs are zero-loss.

Only the second one has meaningful capacity margin.

In this case the raw queue peak dropped from 100% to 14.8% while zero-loss behavior was preserved.

That was the real improvement: not just completeness, but headroom before saturation.

Broad stress tests are useful, but poor at localization

The early workload combined multiple sources and stream types, 1,676 subscriptions, 50 connections, raw capture, parsing, state processing, telemetry, and compression.

That is useful for stress characterization.

It is less useful for answering one specific question: where is the sustainable operating boundary?

For localization, a staged progression is much more informative:

raw only
↓
raw + parsing
↓
raw + parsing + state
↓
full pipeline

Instead of one large red result, you start building a capacity map.

What helped most

The apparent problem changed several times:

shutdown
↓
suspected writer pressure
↓
downstream proved healthy
↓
raw/parser saturation

Each run needed to do more than return PASS or FAIL. It needed to reduce uncertainty.

The workflow that emerged was:

failure
↓
instrument
↓
localize the bottleneck
↓
make the smallest useful change
↓
unit tests
↓
short real-data smoke test
↓
compare capacity metrics
↓
only then run longer tests

That is more useful than repeatedly increasing queue sizes and rerunning the same workload.

Rules I kept from the investigation

A full queue is a symptom. Find its producer and consumer.

Zero loss is not enough. A real-time pipeline also needs freshness, bounded latency, bounded backlog, and capacity headroom.

A larger queue is not more throughput. It can absorb a burst, but it cannot fix a sustained producer > consumer condition.

Instrumentation often comes before optimization. If measurement prevents you from optimizing the wrong component, it has already paid for itself.

A new bottleneck after a fix is not necessarily bad news. It may mean the previous bottleneck is actually gone.

One green metric does not make a system healthy.

In this run:

lost_events == 0

was true.

But:

sustainable_capacity == true

was not.

Takeaway

Zero loss by itself does not mean a real-time pipeline is sustainable.

In this case, the important signal was not lost=0. It was that the raw queue peak dropped from 100% to 14.8% after the change.

The difference was not just avoiding data loss. It was gaining distance from saturation.