Tag: kafka
All the articles with the tag "kafka".
Kafka (Part 7): Integrating Apache Avro and Apicurio Schema Registry to Ensure Message Compatibility Between Producers and Consumers
To address deserialization failures caused by Kafka message format changes, this article walks through migrating from JSON to Avro and integrating Apicurio Schema Registry. It covers schema definition, Java class generation, producer and consumer configuration, and version compatibility checks, reducing the risk of poison-pill messages.
Kafka (Part 6): Oracle-to-PostgreSQL CDC with Kafka Connect and Debezium, and Cache Consistency
Introduces CDC and synchronizes Oracle data to PostgreSQL through a Kafka Connect and Debezium example. It discusses database synchronization, Elasticsearch imports, and cache consistency, comparing the conditions under which different approaches are suitable.
Kafka (Part 5): Consumer Callbacks, Scheduled Retries, and Rebalancing
This largely completes a messaging system built with Kafka. The key code has been explained across the articles; see the source repository for the complete design and implementation. The next Kafka article is planned to cover synchronizing Oracle data to PostgreSQL with Kafka Connect.
Kafka (Part 4): Consuming JSON, Sharing a Deserializer, and Improving Throughput
This article implements Kafka consumers for JSON messages, including shared deserialization, deduplication, retries, and offset commits. Load tests inform partition counts and consumer settings, and the template method pattern reuses the consumption flow for emails of different priorities.