Skip to main content

Kafka Consumer Configuration

At Most once 


Offsets are committed as soon as messages are received, in case of processing fails those messages will be lost, let's say the consumer went down, when the consumer comes back it will start reading from the point of the last committed offsets.



At least once

In this case, offsets are committed after processing the message batch, in case of consumer failure, the same messages are read twice so processing also happens twice, so make sure for idempotency for the system.


Exactly Once

This can be achieved with Kadka Transactional APIS, [easy with Kafka stream apis],



Consumer Offset Reset behavior 


auto.offset.reset:latest -> start reading from end 

auto.offset.reset:earlierest -> start reading from start.

auto.offset.reset.none-> throw NP if the offset was not found.


Note- consumers offset can be lost 

kafkaV<2.0 [consumer hasn't read data in 1 day ]

kafkaV>2.0 [consumer hasn't read data in 7 days]

This can be controlled by offset.retention.minutes  property



Comments

Post a Comment

Popular posts from this blog

Mastering Java Logging: A Guide to Debug, Info, Warn, and Error Levels

Comprehensive Guide to Java Logging Levels: Trace, Debug, Info, Warn, Error, and Fatal Comprehensive Guide to Java Logging Levels: Trace, Debug, Info, Warn, Error, and Fatal Logging is an essential aspect of application development and maintenance. It helps developers track application behavior and troubleshoot issues effectively. Java provides various logging levels to categorize messages based on their severity and purpose. This article covers all major logging levels: Trace , Debug , Info , Warn , Error , and Fatal , along with how these levels impact log printing. 1. Trace The Trace level is the most detailed logging level. It is typically used for granular debugging, such as tracking every method call or step in a complex computation. Use this level sparingly, as it can generate a large volume of log data. 2. Debug The Debug level provides detailed information useful during dev...
Post a Comment
Read more

Choosing Between Envoy and NGINX Ingress Controllers for Kubernetes

As Kubernetes has become the standard for deploying containerized applications, ingress controllers play a critical role in managing how external traffic is routed to services within the cluster. Envoy and NGINX are two of the most popular options for ingress controllers, and each has its strengths, weaknesses, and ideal use cases. In this blog, we’ll explore: How both ingress controllers work. A detailed comparison of their features. When to use Envoy vs. NGINX for ingress management. What is an Ingress Controller? An ingress controller is a specialized load balancer that: Manages incoming HTTP/HTTPS traffic. Routes traffic to appropriate services based on rules defined in Kubernetes ingress resources. Provides features like TLS termination, path-based routing, and host-based routing. How Envoy Ingress Controller Works Envoy , initially built by Lyft, is a high-performance, modern service proxy and ingress solution. Here's how it operates in Kubernetes: Ingress Resource : You d...
Post a Comment
Read more

Distributed Transactions in Microservices: Implementing the Saga Pattern

Managing distributed transactions is one of the most critical challenges in microservices architecture. Since microservices operate with decentralized data storage, traditional ACID transactions across services are not feasible. The Saga Pattern is a proven solution for ensuring data consistency in such distributed systems. In this blog, we’ll discuss: What is the Saga Pattern? Types of Saga Patterns : Orchestration vs. Choreography How to Choose Between Them Implementing Orchestration-Based Saga with Spring Boot An Approach to Event-Driven Saga with Kafka 1. What is the Saga Pattern? The Saga Pattern breaks a long-running distributed transaction into a series of smaller atomic transactions , each managed by a microservice. If any step fails, compensating actions are performed to roll back the preceding operations. Example: In an e-commerce system , a customer places an order: Payment is processed. Inventory is reserved. Shipping is scheduled. If inventory reservation fails, the paym...
Post a Comment
Read more