I've been working with Kafka lately, exploring its concepts and going deeper into the parts that came up while dealing with some production cases.
On paper, it sounds simple: producers write events, consumers read them, everyone is happy. Then you connect a consumer to PostgreSQL, add a third-party API, deploy a few instances, and start running into lag, retries, duplicate deliveries, and offsets that don't quite line up with your business transactions.
Some of those situations showed up in real systems. Others are areas where I’m keen to keep learning and improving. This article is the result of that process: not a definitive guide, but a collection of patterns and trade-offs that helped me understand what was happening and why.
What happens when consumers can't keep up? What if a downstream API starts timing out? What happens when a consumer updates PostgreSQL successfully but crashes before committing its Kafka offset?
Those aren't rare edge cases.
In this article, we'll go through the patterns I've found most useful: backpressure, idempotency, retries, failure recovery, etc. More importantly, we'll look at when each one makes sense and what trade-offs it brings.
Consumer lag is the difference between the end of a partition's log and the consumer group's committed offset. It indicates how far behind the group is, but it does not necessarily reflect how much application processing is still pending.
When producers publish events faster than consumers can process them, the backlog grows. Simple in theory, stressful in practice.
flowchart LR
P[Producers] --> K[Kafka]
K --> C[Consumers]
C --> D[(PostgreSQL)]
C --> A[Third-party API]
style K fill:#dcecff,stroke:#3973ac,color:#172b4d
style D fill:#dff4e4,stroke:#42855b,color:#183d25
style A fill:#fff0d5,stroke:#c28b27,color:#49330d
Consider a concrete example:
The first reaction is to add more consumers. It might help, but only if the downstream dependencies have enough capacity to handle the additional load.
If PostgreSQL is already saturated or a third-party API is responding slowly, adding more consumers can increase contention, connection usage, and request failures.
The first step is to identify the bottleneck before changing the architecture.
Before touching anything, I try to look at a few numbers:
Consumer lag is a symptom, not a root cause. Basically it tells you that something is behind, not why.
Backpressure is the mechanism by which a system prevents incoming work from overwhelming its processing capacity. The name sounds fancy, but the idea is simple: if you can't process more, don't accept more.
It's easy to assume that backpressure simply means reducing the number of consumers. In practice, it involves several complementary strategies.
A consumer can receive messages faster than it can process them. If hundreds of messages trigger database operations simultaneously, they may exhaust the connection pool, increase latency, and cause timeouts.
There are two complementary ways to control this workload:
max.poll.records to limit the number of records returned by each poll() call. The configuration depends on the Kafka client library.For example, a consumer might receive up to 100 records per poll and process only 10 at a time. The remaining records wait until workers become available.
However, limiting the batch size does not prevent pending work from accumulating if the application continues receiving records faster than it can process them. Keep pending work bounded and use Kafka's partition pause/resume mechanism when processing capacity is exhausted.
Start with conservative limits and adjust them based on throughput, processing latency, and downstream resource usage.
Limiting concurrency prevents too much work from running simultaneously, but a consumer can still accumulate pending messages while its workers are busy.
Kafka supports pausing and resuming assigned partitions, allowing the application to stop receiving more records temporarily when processing capacity is exhausted.
For example, a consumer can pause its partitions when its processing queue reaches a predefined limit and resume them when enough capacity becomes available.
The exact API depends on the Kafka client library. The general flow is:
// Pseudocode: adapt to your Kafka client library.
if (processingQueueIsFull()) {
pauseAssignedPartitions();
}
if (processingQueueHasCapacity()) {
resumeAssignedPartitions();
}Pausing does not stop Kafka from retaining messages in the topic. It temporarily stops delivery from the paused partitions to that consumer.
The consumer must still poll or run its event loop as required by the client library and manage offset commits correctly. Otherwise, it may lose its group membership or trigger a rebalance.
Concurrency limits control how many operations run simultaneously. Rate limiting controls how many operations can start within a given period.
This matters when a consumer calls an external service with a strict request quota. For example, an API may allow only 500 requests per second. A consumer processing 2,000 messages per second could exceed that quota, even if its concurrency is limited.
A rate limiter can regulate outgoing requests to respect the provider's limit. Algorithms such as token bucket and leaky bucket are commonly used for this purpose.
When multiple consumer instances share the same API quota, the rate limit must account for their combined traffic. Otherwise, each instance may respect its individual limit while the application exceeds the provider's global limit.
So far, the focus has been on controlling how much work the consumer processes and how quickly it sends requests. Another way to improve throughput is to reduce the overhead of repeated operations.
For example, instead of inserting 100 events using 100 separate database round trips, the consumer can collect them and insert them using a single batch operation.
INSERT INTO processed_events (consumer_name, event_id)
VALUES
('order-projection', '00000000-0000-0000-0000-000000000001'),
('order-projection', '00000000-0000-0000-0000-000000000002'),
('order-projection', '00000000-0000-0000-0000-000000000003')
ON CONFLICT DO NOTHING;This reduces the number of database round trips, although the actual benefit depends on the database, driver, and workload.
Larger batches can improve throughput, but they may also increase processing latency, memory usage, and the amount of work affected by a failure.
Batch size should therefore be tuned against both throughput and latency requirements.
Each backpressure strategy controls a different aspect of the workload:
| Strategy | What it controls | When to use it |
|---|---|---|
| Limit concurrency | How many messages are processed simultaneously. | When the database or another dependency cannot handle too many concurrent operations. |
| Control message intake | How many messages are returned by each poll() call. |
When you want to limit how many messages the consumer receives at once. |
| Pause/resume partitions | When to temporarily stop receiving messages. | When pending work accumulates or a downstream system becomes overloaded. |
| Rate limiting | How many operations can start within a given period. | When an external API enforces a request quota. |
| Batching | How many operations are grouped into a single request. | When you want to reduce database round trips and improve throughput. |
Not every consumer needs all these strategies. Choose the ones that address your actual bottlenecks and resource constraints.
Backpressure helps up to a point. When lag keeps growing, the next question is whether you can simply add more consumers.
Scaling Kafka consumers looks easy on paper: add more instances. In practice, you're usually moving the bottleneck somewhere else.
Kafka consumers can scale horizontally, but scaling is constrained by partition count and downstream capacity.
Adding more consumer instances can increase parallelism up to the number of partitions assigned to the group. Beyond that, extra instances may remain idle.
Even before reaching that limit, the database or external API may become the bottleneck.
A practical scaling strategy should:
A temporary increase in consumer lag is not necessarily a problem. If incoming traffic returns to normal and the consumer can process messages faster than new ones arrive, the backlog will eventually decrease.
What matters is how long it takes to catch up and whether messages are processed within the expected time limits.
Of course, scaling is only half the story. Adding consumers is risky if they all keep hammering a dependency that is already struggling.
Sooner or later, every consumer depends on something that can fail: a database, an HTTP API, a payment provider, another microservice.
When a dependency becomes slow or unavailable, the consumer needs to avoid amplifying the failure.
A circuit breaker monitors failures and temporarily prevents calls to an unhealthy dependency.
It commonly has three states:
stateDiagram-v2
[*] --> Closed
Closed --> Open: Failure threshold reached
Open --> HalfOpen: Recovery timeout
HalfOpen --> Closed: Probe succeeds
HalfOpen --> Open: Probe fails
For example, if an external API starts returning repeated 503 Service Unavailable responses, the circuit breaker can open and prevent every Kafka consumer from continuing to send requests.
But a circuit breaker does not decide what happens to the messages that cannot be processed.
The application still needs a policy:
The right choice depends on whether the event can safely wait, whether ordering matters, and how much retry delay the business can tolerate.
A provider may return 429 Too Many Requests and include a Retry-After header.
When available, that information should inform the retry policy. Repeatedly retrying immediately can worsen throttling.
Requests should also have explicit timeouts. Without them, slow calls can occupy threads, connections, and in-flight capacity indefinitely.
Exponential backoff increases the delay between retry attempts.
A common formula is:
delay = min(maxDelay, baseDelay × 2^attempt)
Jitter introduces randomness into the delay to avoid synchronized retry bursts.
With full jitter:
delay = random(0, min(maxDelay, baseDelay × 2^attempt))
Imagine 1,000 consumers retrying an unavailable API. Without jitter, many of them may retry at nearly the same time. With jitter, retry attempts are spread across a time window.
Backoff reduces retry frequency; jitter reduces synchronization.
Retries should be bounded or governed by an explicit retry policy. They should distinguish transient failures from permanent errors, and they must not bypass idempotency safeguards (we'll get to that in section 6).
Before we talk about idempotency, it helps to be precise about delivery guarantees.
A message is processed zero or one times. A failure can cause work to be lost.
A message may be processed more than once. To avoid losing work, the consumer must only commit its offset after the required processing has succeeded.
The intended effect occurs once within a clearly defined scope.
Kafka supports idempotent producers to prevent duplicates caused by producer retries. Kafka transactions can atomically coordinate consumed offsets and records produced to Kafka topics.
However, these guarantees do not automatically extend to arbitrary PostgreSQL transactions, HTTP requests, or external payment providers.
For most consumers that update an external database, at-least-once delivery combined with idempotent processing is a practical reliability strategy.
The objective is not necessarily to prevent every duplicate delivery. It is to make duplicate deliveries safe.
An operation is idempotent when repeating it produces the same intended result as performing it once.
In distributed systems, duplicate delivery can occur for several reasons:
The key principle is:
Idempotency must be enforced at the business-effect boundary, not merely at the message transport boundary.
Every logical event should have a stable identifier, such as event_id.
That identifier must remain the same across retries of the same logical event. Otherwise, duplicate deliveries will look like new events.
A Kafka offset identifies a record within a particular topic partition, but a domain event ID is generally more useful when the same event may be republished or transported through different systems.
A stable event ID can also support application-specific idempotency rules, deduplication, auditing, and tracing an event across services.
When a consumer needs to prevent duplicate database updates, the database should enforce uniqueness rather than relying on an application-level check.
Two concurrent executions could both check whether an event exists, find nothing, and proceed. A unique constraint prevents both from successfully claiming the same event.
For PostgreSQL:
CREATE TABLE processed_events (
consumer_name TEXT NOT NULL,
event_id UUID NOT NULL,
PRIMARY KEY (consumer_name, event_id)
);Including the consumer name allows different logical consumers to process the same event independently.
The consumer can claim the event using:
INSERT INTO processed_events (consumer_name, event_id)
VALUES ('order-projection', :event_id)
ON CONFLICT DO NOTHING
RETURNING event_id;If the statement returns an ID, the event was newly claimed. If it returns no row, that consumer has already processed the event.
The claim must be part of the same database transaction as the business update.
The right approach depends on the business effect being protected and where that effect occurs.
Common strategies include:
For database updates, commit the transaction before committing the Kafka offset. If the consumer crashes after the database transaction succeeds but before the offset is committed, Kafka may deliver the event again. The idempotency record allows the consumer to skip the business update.
A database transaction cannot make an external API call atomic with a database update. External side effects require their own safeguards.
The goal is not simply to detect duplicate messages, but to ensure that processing the same logical event multiple times does not repeat its business effects.
Not every failure should be handled in the same way. Some errors are temporary and may disappear on retry. Others are permanent and will never succeed, no matter how many times the consumer tries.
A robust recovery strategy starts by classifying the failure:
Retry topics can separate failed messages from the main processing path, allowing them to be retried later without blocking other messages. For example, if a downstream service is temporarily unavailable, a message can be routed to a retry topic and processed again after a delay. The retry mechanism must control when the message becomes eligible for another attempt, since Kafka topics do not provide delayed delivery by themselves.
However, this approach can change event ordering. If a message fails and is moved to a retry topic, later messages may be processed first. If events for the same entity must be processed in order, the retry strategy must preserve that ordering or explicitly handle out-of-order events.
Retries should also be bounded. After a configured number of attempts, a message that continues to fail should be routed to a defined failure path, such as a dead-letter topic, rather than being retried indefinitely.
A poison message is one that repeatedly fails processing because of its content or another persistent problem.
A dead-letter topic provides a place to store messages that cannot be processed successfully. Include enough context to investigate the failure, such as the original topic, partition, offset, error details, and retry history, while avoiding sensitive data.
A dead-letter topic is not a complete recovery strategy on its own. You also need monitoring, alerting, an investigation process, and a safe way to replay messages after the underlying problem has been fixed.
Before replaying a failed message, determine whether its business effect has already occurred. This is especially important when a downstream API times out: the operation may have succeeded even though the consumer never received the response.
Use the provider's idempotency mechanism or another reliable way to verify the operation's status. If the outcome remains uncertain, keep the operation pending for reconciliation rather than blindly repeating an irreversible action.
Idempotent consumers make retries and replays safer because processing the same logical event again does not repeat its business effects.
After an incident, consumers may return to a large backlog. Restoring the service does not guarantee that the backlog will clear quickly.
Check how much data remains, how quickly consumers are processing it, and whether Kafka retention gives you enough time to catch up. Temporary scaling can help if the downstream systems can handle the extra load.
Measure recovery using consumer lag and the time required to return to normal. If messages need to be replayed, do so deliberately and monitor the operation.
The goal is not to retry every failure. It is to recover from temporary problems, isolate messages that cannot be processed, and make every retry or replay a controlled and observable operation.
These are some of the things I've learned recently while exploring how to build more reliable Kafka consumers. I hope you find them as useful and interesting as I did.
Building reliable consumers isn't just about processing messages faster. It's about handling backpressure, scaling responsibly, making processing idempotent, and recovering from failures safely.
There's always more to learn, but understanding these fundamentals is a great place to start.