High Water Mark no longer advances when topics are below configured min.insync.replicas

Description

In certain situations it is possible to produce to a topic-partition even if the partition falls below the configured min.insync.replicas-- for example if the producer is configured with acks=0 or acks=1.  In such situations, even though the producers are still able to produce to the partition below the configured min.insync.replicas, consumers will not consume any of the messages produced after the partition fell below the topic's configured min.insync.replicas.  Once the partition has the minimum required in-sync replicas the consumers will continue consuming.

This is because consumers are only able to consume up to the partition's High Water Mark.  Even though new messages are being actively produced, consumers will not have access to those new messages if the High Water Mark for the partition is not advanced.

In previous versions of Confluent Platform High Water Mark advancement was not dependent on min.insync.replicas.  However, with the implementation of KAFKA-15583 in Confluent Platform version 7.7.0, the High Water Mark is not allowed to progress until at least the configured min.insync.replicas is satisfied.

• Component Changelog
• KIP-966 with explanation of reason for the change

Applies To

Confluent Platform 7.7.0 and later

Apache Kafka 3.7 and later

Resolution

• Bring additional Brokers into ISR for the affected topic-partitions.
• Reduce the required min.insync.replicas for the affected topic to match the number of Brokers in ISR.
• To learn more about how the min.insync.replicas configuration affects topic durability, please see the following resource:
  • Configuring Durability, Availability, and Ordering Guarantees