Skip to main content

Kafka Producer Configuration

 This article will show the Kafka producer config.


Acks=0

Producer assumes a message written successfully to the topic at the same time the message is sent, without waiting for the broker to accept it all, it 

If the broker goes offline at the same will lose the data 

Suitable for the scenario where potentially okay to lose the data, like metrics collections.

In this case, producer throughput is highest as less network overhead 


Acks=1


Producer assumes a message written successfully to the topic if gets ack from the leader.

It's the default behavior for Kafka 1 and 2.8

Leader response is requested, but replication is not guaranteed as it happens in the background.

If the broker leader goes offline unexpectedly, at the same time replication does not happen possible data loss.

If acks are not received, the Producer will retry to send the message again.


Acks=All [-1]

it's default for kafka V3+.

In this case will wait for acceptance from all sync replicas. No data loss.

With min.insync.replicas Value, this is the optimization of this setting. 

Let's say if having 3 replicas, and setting for min.insync.replicas =2 , will just wait for leader and one replica response,


Idempotent Producer 

This producer guarantees no duplication of the message, in case of broker's response gets failed due to network error, it's by default behaviour of Kafka V3.


The default configs

retries = Integer.maxValue(2^31-1)

max.in.flight.requests=1 [kakfka 0.11]Or [Note number of parallel request]

max.in.flight.requests=5 [kakfka >V1, Higher performance and keep ordering KAFKA-5494]


The default config for Kafka v3+ is above.[Safe Kakfka]

acks=All

enabled.idempotence=true;

max.in.flight.requests=5

timeout  2 minute 

min.insync.replicas=2

The default config for Kafka v2.8 and below.

acks=1

enabled.idempotence=false;


Batching Performance Improvement 


Increase liger.ms time will help the producer to wait that many milliseconds to fill up the batch

The batch size can be increased to take more messages.

Add producer-level compression for efficiency.


Default Partitioner and Sticky Partitioner  

2.4 > above sticky partitioner [performance improvement in as message sent into batches with full capacity ]

The below version is a round-robin in this case more batches are less efficient.


Advanced Setting max.block.ms and buffer.memory

When the Broker is down or busy, the Producer keeps buffering the messages into the buffer as per setting 

producer send method and throws the exception as per the defined time max.block.ms if it is not able to deliver the messages that got buffered. It simply means the Broker is down or some issues need action on it.

























Comments

Post a Comment

Popular posts from this blog

Learning How to Map One-to-Many Relationships in JPA Spring Boot with PostgreSQL

  Introduction In this blog post, we explore how to effectively map one-to-many relationships using Spring Boot and PostgreSQL. This relationship type is common in database design, where one entity (e.g., a post) can have multiple related entities (e.g., comments). We'll dive into the implementation details with code snippets and provide insights into best practices. Understanding One-to-Many Relationships A one-to-many relationship signifies that one entity instance can be associated with multiple instances of another entity. In our case: Post Entity : Represents a blog post with fields such as id , title , content , and a collection of comments . Comment Entity : Represents comments on posts, including fields like id , content , and a reference to the post it belongs to. Mapping with Spring Boot and PostgreSQL Let's examine how we define and manage this relationship in our Spring Boot application: Post Entity  @Entity @Getter @Setter @Builder @AllArgsConstructor @NoArgsCon...
Post a Comment
Read more

Understanding the Advertisement Domain: A Comprehensive Overview Part 2

 The advertisement domain is a complex and dynamic ecosystem that involves various technologies and platforms working together to deliver ads to users in a targeted and efficient manner. The primary goal is to connect advertisers with their target audience, increasing brand visibility, user engagement, and revenue generation. In this blog, we will delve into the different components of the advertisement ecosystem, key concepts like programmatic advertising and real-time bidding (RTB), and provide a practical example to illustrate how it all works. Key Components of the Advertisement Domain The advertisement domain broadly consists of the following components: Advertisers : These are brands or companies that want to promote their products or services through advertisements. They set up ad campaigns targeting specific user segments. Publishers : These are websites, mobile apps, or digital platforms that display ads to users. Publishers monetize their content by selling ad space to ad...
Post a Comment
Read more

Tree Based Common problems and patterns

  Find the height of the tree. public class BinaryTreeHeight { public static int heightOfBinaryTree (TreeNode root) { if (root == null ) { return - 1 ; // Height of an empty tree is -1 } int leftHeight = heightOfBinaryTree(root.left); int rightHeight = heightOfBinaryTree(root.right); // Height of the tree is the maximum of left and right subtree heights plus 1 for the root return Math.max(leftHeight, rightHeight) + 1 ; } Find the Level of the Node. private static int findLevel (TreeNode root, TreeNode node, int level) { if (root == null ) { return - 1 ; // Node not found, return -1 } if (root == node) { return level; // Node found, return current level } // Check left subtree int leftLevel = findLevel(root.left, node, level + 1 ); if (leftLevel != - 1 ) { return leftLevel; // Node found ...
Post a Comment
Read more