Skip to main content

Posts

🔄 Kafka Producer Internals: send() Explained with Delivery Semantics and Transactions

Kafka Producer Internal Working Apache Kafka is known for its high-throughput, fault-tolerant message streaming system. At the heart of Kafka's data pipeline is the Producer —responsible for publishing data to Kafka topics. This blog dives deep into the internal workings of the Kafka Producer, especially what happens under the hood when send() is called. We'll also break down different delivery guarantees and transactional semantics with diagrams. 🧠 Table of Contents Kafka Producer Architecture Overview What Happens When send() is Called Delivery Semantics Kafka Transactions & Idempotence Error Handling and Retries Diagram: Kafka Producer Internals Conclusion 🏗️ Kafka Producer Architecture Overview Kafka Producer is composed of the following core components: Serializer : Converts key/value to bytes. Partitioner : Determines which partition a record should go to. Accumulator : Buffers the records in memory befor...
Post a Comment
Read more
Recent posts

Project Reactor vs Java Virtual Threads: Which One to Choose for High-Concurrency Applications?

If you're building a high-concurrency Java backend —say, something that handles millions of requests per second —you’ve likely faced the dilemma: Should I use Project Reactor (Reactive Streams) or Java Virtual Threads (Project Loom)? After working with both, here’s a deep dive comparison that can help you decide. 🔧 What Are They? Project Reactor (Spring WebFlux) Asynchronous, event-driven, non-blocking model. Built on Mono and Flux , part of the reactive ecosystem. Ideal for streaming data and backpressure handling. Java Virtual Threads (JDK 21+) Lightweight threads, launched like regular threads ( Thread.ofVirtual() ). Enable writing traditional blocking code that scales like async. Great for simplifying code without compromising concurrency. ⚔️ Key Differences Feature Project Reactor Java Virtual Threads Programming Style Callback-based (Reactive) Imperative (Synchronous) Blocking I/O Not allowed (must use non-blocking APIs) Allowed (eff...
Post a Comment
Read more

Project Reactor Important Methods Cheat Sheet

🔹 1️⃣ subscribeOn – "Decides WHERE the Pipeline Starts" 📝 Definition: subscribeOn influences the thread where the data source (upstream) (e.g., data generation, API calls) runs . It affects the source and everything downstream (until a publishOn switches it). Flux<Integer> flux = Flux.range(1, 3) .doOnNext(i -> System.out.println("[Generating] " + i + " on " + Thread.currentThread().getName())) .subscribeOn(Schedulers.boundedElastic()) // Change starting thread .map(i -> { System.out.println("[Processing] " + i + " on " + Thread.currentThread().getName()); return i * 10; }); flux.blockLast(); Output: [Generating] 1 on boundedElastic-1 [Processing] 1 on boundedElastic-1 [Generating] 2 on boundedElastic-1 [Processing] 2 on boundedElastic-1 [Generating] 3 on boundedElastic-1 [Processing] 3 on boundedElastic-1 📢 Key Insight: subscribeOn affects the beginning (...
Post a Comment
Read more

Advanced Kafka Resilience: Dead-Letter Queues, Circuit Breakers, and Exactly-Once Delivery

Introduction In distributed systems, failures are inevitable—network partitions, broker crashes, or consumer lag can disrupt data flow. While retries help recover from transient issues, you need stronger guarantees for mission-critical systems. This guide covers three advanced Kafka resilience patterns: Dead-Letter Queues (DLQs) – Handle poison pills and unprocessable messages. Circuit Breakers – Prevent cascading failures when Kafka is unhealthy. Exactly-Once Delivery – Avoid duplicates in financial/transactional systems. Let's dive in! 1. Dead-Letter Queues (DLQs) in Kafka What is a DLQ? A dedicated Kafka topic where "failed" messages are sent after max retries (e.g., malformed payloads, unrecoverable errors). Why Use DLQs? Isolate bad messages instead of blocking retries. Audit failures for debugging. Reproce...
Post a Comment
Read more

Handling Kafka Retries in Spring Boot: Blocking vs. Reactive Approaches

  Introduction Apache Kafka is designed for high availability, but failures still happen—network issues, broker crashes, or cluster downtime. To ensure message delivery, applications must implement retry mechanisms. However, retries behave differently in traditional (blocking) vs. reactive (non-blocking) Kafka producers. This guide covers: ✅ Kafka’s built-in retries ( retries ,  retry.backoff.ms ) ✅ Blocking vs. non-blocking retry strategies ✅ Reactive Kafka retries with backoff ✅ Fallback strategies for guaranteed delivery ✅ Real-world failure scenarios and fixes 1. Kafka Producer Retry Basics When Do Retries Happen? Kafka producers automatically retry on: Network errors (e.g., broker disconnect) Leader election (e.g., broker restart) Temporary errors (e.g.,  NOT_ENOUGH_REPLICAS ) Key Configuration Properties Property Default Description retries 0 Number of retries for transient failures. retry.backoff...
Post a Comment
Read more