Skip to main content

Kafka Consumer Configuration

At Most once 


Offsets are committed as soon as messages are received, in case of processing fails those messages will be lost, let's say the consumer went down, when the consumer comes back it will start reading from the point of the last committed offsets.



At least once

In this case, offsets are committed after processing the message batch, in case of consumer failure, the same messages are read twice so processing also happens twice, so make sure for idempotency for the system.


Exactly Once

This can be achieved with Kadka Transactional APIS, [easy with Kafka stream apis],



Consumer Offset Reset behavior 


auto.offset.reset:latest -> start reading from end 

auto.offset.reset:earlierest -> start reading from start.

auto.offset.reset.none-> throw NP if the offset was not found.


Note- consumers offset can be lost 

kafkaV<2.0 [consumer hasn't read data in 1 day ]

kafkaV>2.0 [consumer hasn't read data in 7 days]

This can be controlled by offset.retention.minutes  property



Comments

Post a Comment

Popular posts from this blog

Learning How to Map One-to-Many Relationships in JPA Spring Boot with PostgreSQL

  Introduction In this blog post, we explore how to effectively map one-to-many relationships using Spring Boot and PostgreSQL. This relationship type is common in database design, where one entity (e.g., a post) can have multiple related entities (e.g., comments). We'll dive into the implementation details with code snippets and provide insights into best practices. Understanding One-to-Many Relationships A one-to-many relationship signifies that one entity instance can be associated with multiple instances of another entity. In our case: Post Entity : Represents a blog post with fields such as id , title , content , and a collection of comments . Comment Entity : Represents comments on posts, including fields like id , content , and a reference to the post it belongs to. Mapping with Spring Boot and PostgreSQL Let's examine how we define and manage this relationship in our Spring Boot application: Post Entity  @Entity @Getter @Setter @Builder @AllArgsConstructor @NoArgsCon...
Post a Comment
Read more

Understanding the Advertisement Domain: A Comprehensive Overview Part 2

 The advertisement domain is a complex and dynamic ecosystem that involves various technologies and platforms working together to deliver ads to users in a targeted and efficient manner. The primary goal is to connect advertisers with their target audience, increasing brand visibility, user engagement, and revenue generation. In this blog, we will delve into the different components of the advertisement ecosystem, key concepts like programmatic advertising and real-time bidding (RTB), and provide a practical example to illustrate how it all works. Key Components of the Advertisement Domain The advertisement domain broadly consists of the following components: Advertisers : These are brands or companies that want to promote their products or services through advertisements. They set up ad campaigns targeting specific user segments. Publishers : These are websites, mobile apps, or digital platforms that display ads to users. Publishers monetize their content by selling ad space to ad...
Post a Comment
Read more

Tree Based Common problems and patterns

  Find the height of the tree. public class BinaryTreeHeight { public static int heightOfBinaryTree (TreeNode root) { if (root == null ) { return - 1 ; // Height of an empty tree is -1 } int leftHeight = heightOfBinaryTree(root.left); int rightHeight = heightOfBinaryTree(root.right); // Height of the tree is the maximum of left and right subtree heights plus 1 for the root return Math.max(leftHeight, rightHeight) + 1 ; } Find the Level of the Node. private static int findLevel (TreeNode root, TreeNode node, int level) { if (root == null ) { return - 1 ; // Node not found, return -1 } if (root == node) { return level; // Node found, return current level } // Check left subtree int leftLevel = findLevel(root.left, node, level + 1 ); if (leftLevel != - 1 ) { return leftLevel; // Node found ...
Post a Comment
Read more