All the highlights of the newest release

Apache Kafka 2.3 arrives with improved monitoring for partitions which have lost replicas

apache kafka
© Shutterstock / Lemau Studio  

The latest version of Apache Kafka is out and it brings a long list of improvements including, improved monitoring for partitions which have lost replicas and the addition of a Maximum Log Compaction Lag. Let’s take a look.

Apache Kafka 2.3 is here!

This version brings a long list of important improvements and new features including improved monitoring for partitions which have lost replicas and the addition of a Maximum Log Compaction Lag, which can help make your applications more GDPR compliant!

Let’s have a look at the most interesting highlights.

What’s new in Apache Kafka 2.3?

KIP-351 and KIP-427: Improved monitoring for partitions which have lost replicas – Adds additional metrics showing partitions that have exactly the minimum number of in-sync replicas. By monitoring these metrics, users can see partitions that are on the verge of becoming under-replicated. What’s more,  KIP-351 adds the –under-min-isr command line flag to the kafka-topics command. This allows users to easily see which topics have fewer than the minimum number of in-sync replicas.

KIP-354: Add a Maximum Log Compaction Lag – With KIP-354, it’s possible to set the maximum amount of time an old value will stick around. The new parameter specifies how long an old value may possibly live in a compacted topic. This can be used in complying with data retention regulations such as the GDPR.

KIP-402: Improve fairness in SocketServer processors – Prioritizes existing connections over new ones, which improves the broker’s resilience to connection storms.

KIP-461: Improve failure handling in the Replica Fetcher – If a single partition managed by a given replica fetcher thread fails, the thread continues handling the remainder of its partitions.

Other improvements and updates include:

  • KAFKA-7283: Reduce the amount of time the broker spends scanning log files when starting up
  • KIP-415: Incremental Cooperative Rebalancing in Kafka Connect
  • KIP-449: Add connector contexts to Connect worker logs
  • KIP-258: Allow Users to Store Record Timestamps in RocksDB
  • KIP-428: Add in-memory window store/KIP-445: Add in-memory Session Store
  • KIP-313: Add KStream.flatTransform and KStream.flatTransformValues

You can find the full list of new features, updates, and improvements for Apache Kafka 2.3 here.

SEE ALSO: Implement stream-aware, reactive, integration pipelines with Alpakka Kafka

You can also check out this informative video by Tim Berglund explaining what’s new in this release.


Eirini-Eleni Papadopoulou
Eirini-Eleni Papadopoulou was the editor for Coming from an academic background in East Asian Studies, she decided that it was time to go back to her high-school hobby that was computer science and she dived into the development world. Other hobbies include esports and League of Legends, although she never managed to escape elo hell (yet), and she is a guest writer/analyst for competitive LoL at TGH.

Inline Feedbacks
View all comments