How a single malformed message brought our pricing pipeline to a halt
by Sergej Subkov
Our pricing pipeline suddenly stopped processing new data. Consumer lag kept rising, downstream systems worked with stale prices, and every restart produced exactly the same result: the application came up, crashed and stalled again.
The cause was not a complex infrastructure failure.
It was a single malformed JSON message.
One record blocked the entire partition
Our Kafka Streams application processed pricing and product data in an e-commerce context. Within a Kafka partition, messages are read in order. That very guarantee was what worked against us in this moment.
The deserializer could not read one record because two JSON objects had been incorrectly concatenated. Processing aborted. After the restart, the application hit the same offset again and failed again.
All subsequent messages in that partition remained blocked.
There is a name for exactly this pattern in the Kafka world: a poison pill — a message a consumer cannot process and keeps failing on with every new attempt.
A small bug in a producer had effectively brought a business-critical pipeline to a standstill.
The immediate fix
The stack trace gave us topic, partition and offset. Using the kafka-console-consumer, we read the affected record straight from Kafka.
The raw data confirmed the suspicion immediately: the JSON was invalid and could not be meaningfully repaired on the consumer side.
So we worked on two things in parallel:
- We preserved the malformed record and skipped it in a controlled way so the pipeline could catch up on its backlog.
- We fixed the bug in the producer so no further invalid messages were generated.
A few hours later, consumer lag was back to zero.
But that only resolved the incident, not the underlying class of failure.
The real solution: a dead letter queue
One thing was clear to us: in a distributed system landscape, malformed messages can never be prevented entirely.
Producers change. Schemas evolve. Teams deploy independently. Sooner or later the next poison pill will arrive — a record a consumer cannot process.
So the decisive question was no longer:
How do we prevent every malformed message?
But:
How do we prevent one malformed message from stopping all processing?
This shifted the focus from preventing errors to resilience: the pipeline had to expect bad data and keep running anyway.
Our answer was a dead letter queue.
On deserialization errors, a custom DeserializationExceptionHandler caught the malformed record. Instead of shutting down processing, it wrote the raw data together with topic, partition, offset and error details to a separate error topic.
After that, Kafka Streams could continue with the next record.
properties.put(
StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
DlqDeserializationExceptionHandler.class
);The goal was not to swallow errors. Quite the opposite: they were meant to become visible and actionable without blocking the entire data flow at the same time.
What changed afterwards
Since introducing the dead letter queue, a single invalid record no longer automatically led to a complete pipeline standstill.
The malformed message landed in the error topic. There we could immediately see:
- which topic it came from,
- which partition it was in,
- at which offset it was processed,
- and why it had failed.
That shortened diagnosis considerably. Instead of searching logs at night and reading records out of Kafka by hand, we had all the relevant information in one place.
Working with producer teams also became easier. We no longer had to guess which change had caused the error. We could show the exact record and the exact exception.
The most important lesson
A dead letter queue is not a free pass to simply ignore bad data.
Especially in pricing, a skipped record can mean a product temporarily keeps running with an outdated price. That is why you need alerts, clear ownership and a process for reprocessing later.
But a single broken message must not additionally block all subsequent valid price changes.
That was the decisive difference for us.
The short-term fix got the pipeline running again.
The dead letter queue made sure the same type of failure could never again paralyze the entire processing.
Conclusion
A single malformed message had been enough to stop a business-critical Kafka Streams pipeline.
The lasting solution was not to hope for error-free data. It was to design for the failure case deliberately.
Since then, problematic records are isolated, analyzed and reprocessed later, while valid messages keep flowing.
Good infrastructure does not prevent every error.
Resilient infrastructure makes sure a small error does not become a big outage.