Skip to main content

Kafka Producer Configuration

 This article will show the Kafka producer config.


Acks=0

Producer assumes a message written successfully to the topic at the same time the message is sent, without waiting for the broker to accept it all, it 

If the broker goes offline at the same will lose the data 

Suitable for the scenario where potentially okay to lose the data, like metrics collections.

In this case, producer throughput is highest as less network overhead 


Acks=1


Producer assumes a message written successfully to the topic if gets ack from the leader.

It's the default behavior for Kafka 1 and 2.8

Leader response is requested, but replication is not guaranteed as it happens in the background.

If the broker leader goes offline unexpectedly, at the same time replication does not happen possible data loss.

If acks are not received, the Producer will retry to send the message again.


Acks=All [-1]

it's default for kafka V3+.

In this case will wait for acceptance from all sync replicas. No data loss.

With min.insync.replicas Value, this is the optimization of this setting. 

Let's say if having 3 replicas, and setting for min.insync.replicas =2 , will just wait for leader and one replica response,


Idempotent Producer 

This producer guarantees no duplication of the message, in case of broker's response gets failed due to network error, it's by default behaviour of Kafka V3.


The default configs

retries = Integer.maxValue(2^31-1)

max.in.flight.requests=1 [kakfka 0.11]Or [Note number of parallel request]

max.in.flight.requests=5 [kakfka >V1, Higher performance and keep ordering KAFKA-5494]


The default config for Kafka v3+ is above.[Safe Kakfka]

acks=All

enabled.idempotence=true;

max.in.flight.requests=5

timeout  2 minute 

min.insync.replicas=2

The default config for Kafka v2.8 and below.

acks=1

enabled.idempotence=false;


Batching Performance Improvement 


Increase liger.ms time will help the producer to wait that many milliseconds to fill up the batch

The batch size can be increased to take more messages.

Add producer-level compression for efficiency.


Default Partitioner and Sticky Partitioner  

2.4 > above sticky partitioner [performance improvement in as message sent into batches with full capacity ]

The below version is a round-robin in this case more batches are less efficient.


Advanced Setting max.block.ms and buffer.memory

When the Broker is down or busy, the Producer keeps buffering the messages into the buffer as per setting 

producer send method and throws the exception as per the defined time max.block.ms if it is not able to deliver the messages that got buffered. It simply means the Broker is down or some issues need action on it.

























Comments

Post a Comment

Popular posts from this blog

Mastering Java Logging: A Guide to Debug, Info, Warn, and Error Levels

Comprehensive Guide to Java Logging Levels: Trace, Debug, Info, Warn, Error, and Fatal Comprehensive Guide to Java Logging Levels: Trace, Debug, Info, Warn, Error, and Fatal Logging is an essential aspect of application development and maintenance. It helps developers track application behavior and troubleshoot issues effectively. Java provides various logging levels to categorize messages based on their severity and purpose. This article covers all major logging levels: Trace , Debug , Info , Warn , Error , and Fatal , along with how these levels impact log printing. 1. Trace The Trace level is the most detailed logging level. It is typically used for granular debugging, such as tracking every method call or step in a complex computation. Use this level sparingly, as it can generate a large volume of log data. 2. Debug The Debug level provides detailed information useful during dev...
Post a Comment
Read more

Choosing Between Envoy and NGINX Ingress Controllers for Kubernetes

As Kubernetes has become the standard for deploying containerized applications, ingress controllers play a critical role in managing how external traffic is routed to services within the cluster. Envoy and NGINX are two of the most popular options for ingress controllers, and each has its strengths, weaknesses, and ideal use cases. In this blog, we’ll explore: How both ingress controllers work. A detailed comparison of their features. When to use Envoy vs. NGINX for ingress management. What is an Ingress Controller? An ingress controller is a specialized load balancer that: Manages incoming HTTP/HTTPS traffic. Routes traffic to appropriate services based on rules defined in Kubernetes ingress resources. Provides features like TLS termination, path-based routing, and host-based routing. How Envoy Ingress Controller Works Envoy , initially built by Lyft, is a high-performance, modern service proxy and ingress solution. Here's how it operates in Kubernetes: Ingress Resource : You d...
Post a Comment
Read more

Distributed Transactions in Microservices: Implementing the Saga Pattern

Managing distributed transactions is one of the most critical challenges in microservices architecture. Since microservices operate with decentralized data storage, traditional ACID transactions across services are not feasible. The Saga Pattern is a proven solution for ensuring data consistency in such distributed systems. In this blog, we’ll discuss: What is the Saga Pattern? Types of Saga Patterns : Orchestration vs. Choreography How to Choose Between Them Implementing Orchestration-Based Saga with Spring Boot An Approach to Event-Driven Saga with Kafka 1. What is the Saga Pattern? The Saga Pattern breaks a long-running distributed transaction into a series of smaller atomic transactions , each managed by a microservice. If any step fails, compensating actions are performed to roll back the preceding operations. Example: In an e-commerce system , a customer places an order: Payment is processed. Inventory is reserved. Shipping is scheduled. If inventory reservation fails, the paym...
Post a Comment
Read more