Skip to main content

Kafka Producer Configuration

 This article will show the Kafka producer config.


Acks=0

Producer assumes a message written successfully to the topic at the same time the message is sent, without waiting for the broker to accept it all, it 

If the broker goes offline at the same will lose the data 

Suitable for the scenario where potentially okay to lose the data, like metrics collections.

In this case, producer throughput is highest as less network overhead 


Acks=1


Producer assumes a message written successfully to the topic if gets ack from the leader.

It's the default behavior for Kafka 1 and 2.8

Leader response is requested, but replication is not guaranteed as it happens in the background.

If the broker leader goes offline unexpectedly, at the same time replication does not happen possible data loss.

If acks are not received, the Producer will retry to send the message again.


Acks=All [-1]

it's default for kafka V3+.

In this case will wait for acceptance from all sync replicas. No data loss.

With min.insync.replicas Value, this is the optimization of this setting. 

Let's say if having 3 replicas, and setting for min.insync.replicas =2 , will just wait for leader and one replica response,


Idempotent Producer 

This producer guarantees no duplication of the message, in case of broker's response gets failed due to network error, it's by default behaviour of Kafka V3.


The default configs

retries = Integer.maxValue(2^31-1)

max.in.flight.requests=1 [kakfka 0.11]Or [Note number of parallel request]

max.in.flight.requests=5 [kakfka >V1, Higher performance and keep ordering KAFKA-5494]


The default config for Kafka v3+ is above.[Safe Kakfka]

acks=All

enabled.idempotence=true;

max.in.flight.requests=5

timeout  2 minute 

min.insync.replicas=2

The default config for Kafka v2.8 and below.

acks=1

enabled.idempotence=false;


Batching Performance Improvement 


Increase liger.ms time will help the producer to wait that many milliseconds to fill up the batch

The batch size can be increased to take more messages.

Add producer-level compression for efficiency.


Default Partitioner and Sticky Partitioner  

2.4 > above sticky partitioner [performance improvement in as message sent into batches with full capacity ]

The below version is a round-robin in this case more batches are less efficient.


Advanced Setting max.block.ms and buffer.memory

When the Broker is down or busy, the Producer keeps buffering the messages into the buffer as per setting 

producer send method and throws the exception as per the defined time max.block.ms if it is not able to deliver the messages that got buffered. It simply means the Broker is down or some issues need action on it.

























Comments

Post a Comment

Popular posts from this blog

Mastering Java Logging: A Guide to Debug, Info, Warn, and Error Levels

Comprehensive Guide to Java Logging Levels: Trace, Debug, Info, Warn, Error, and Fatal Comprehensive Guide to Java Logging Levels: Trace, Debug, Info, Warn, Error, and Fatal Logging is an essential aspect of application development and maintenance. It helps developers track application behavior and troubleshoot issues effectively. Java provides various logging levels to categorize messages based on their severity and purpose. This article covers all major logging levels: Trace , Debug , Info , Warn , Error , and Fatal , along with how these levels impact log printing. 1. Trace The Trace level is the most detailed logging level. It is typically used for granular debugging, such as tracking every method call or step in a complex computation. Use this level sparingly, as it can generate a large volume of log data. 2. Debug The Debug level provides detailed information useful during dev...
Post a Comment
Read more

Choosing Between Envoy and NGINX Ingress Controllers for Kubernetes

As Kubernetes has become the standard for deploying containerized applications, ingress controllers play a critical role in managing how external traffic is routed to services within the cluster. Envoy and NGINX are two of the most popular options for ingress controllers, and each has its strengths, weaknesses, and ideal use cases. In this blog, we’ll explore: How both ingress controllers work. A detailed comparison of their features. When to use Envoy vs. NGINX for ingress management. What is an Ingress Controller? An ingress controller is a specialized load balancer that: Manages incoming HTTP/HTTPS traffic. Routes traffic to appropriate services based on rules defined in Kubernetes ingress resources. Provides features like TLS termination, path-based routing, and host-based routing. How Envoy Ingress Controller Works Envoy , initially built by Lyft, is a high-performance, modern service proxy and ingress solution. Here's how it operates in Kubernetes: Ingress Resource : You d...
Post a Comment
Read more

How to Identify High-Growth Stocks: Key Metrics and Analysis

Identifying high-growth stocks can significantly enhance your investment portfolio's performance. By analyzing key financial metrics, growth indicators, and market opportunities, you can pinpoint companies with the potential for exceptional returns. This blog outlines the critical factors to consider when selecting high-growth stocks. Key Metrics for High-Growth Stocks 1. Earnings Growth Consistent earnings growth is a hallmark of high-growth stocks. Look for companies with a double-digit EPS (Earnings Per Share) growth rate over several years, indicating strong profitability. 2. Revenue Growth Revenue growth shows the company’s ability to expand its market share or increase sales. Look for annual revenue growth rates above 15-20% . 3. Return on Equity (ROE) ROE measures how effectively a company uses shareholders' equity to generate profit. A high ROE (above 15-20% ) is ideal for high-growth companies. 4. Profit Margins Gross...
Post a Comment
Read more