JackSparrow414

Kafka in Practice

From setup and messaging design to producers, consumers, retries, CDC, and message compatibility.

7 posts · In reading order

  1. Kafka (Part 1): A Single-Node KRaft Setup with Docker Compose, Kafka UI, and Prometheus JMX Exporter

    Builds a single-node Kafka environment in KRaft mode with Docker Compose and integrates Kafka UI and Prometheus JMX Exporter. It explains listeners, roles, memory, and monitoring configuration, then covers startup verification, JMX port conflicts, and resource-related troubleshooting.

  2. Kafka (Part 2): Designing a Messaging System to Decouple Email Delivery

    Uses email delivery to explain a design that decouples a business system from an email system through Kafka. It discusses responsibility boundaries, a unified message format, sending and callback flows, and a fallback when switching between old and new systems, providing background for subsequent producer and consumer implementations.

  3. Kafka (Part 3): Sending JSON with a Shared Serializer and Improving Producer Throughput

    Implements a Kafka producer that sends JSON messages, using a shared serializer for different objects and configuring asynchronous sending, failure recording, and retries. Based on actual email-message sizes, it discusses how compression, batch size, request size, and waiting time affect throughput.

  4. Kafka (Part 4): Consuming JSON, Sharing a Deserializer, and Improving Throughput

    This article implements Kafka consumers for JSON messages, including shared deserialization, deduplication, retries, and offset commits. Load tests inform partition counts and consumer settings, and the template method pattern reuses the consumption flow for emails of different priorities.

  5. Kafka (Part 5): Consumer Callbacks, Scheduled Retries, and Rebalancing

    This largely completes a messaging system built with Kafka. The key code has been explained across the articles; see the source repository for the complete design and implementation. The next Kafka article is planned to cover synchronizing Oracle data to PostgreSQL with Kafka Connect.

  6. Kafka (Part 6): Oracle-to-PostgreSQL CDC with Kafka Connect and Debezium, and Cache Consistency

    Introduces CDC and synchronizes Oracle data to PostgreSQL through a Kafka Connect and Debezium example. It discusses database synchronization, Elasticsearch imports, and cache consistency, comparing the conditions under which different approaches are suitable.

  7. Kafka (Part 7): Integrating Apache Avro and Apicurio Schema Registry to Ensure Message Compatibility Between Producers and Consumers

    To address deserialization failures caused by Kafka message format changes, this article walks through migrating from JSON to Avro and integrating Apicurio Schema Registry. It covers schema definition, Java class generation, producer and consumer configuration, and version compatibility checks, reducing the risk of poison-pill messages.