KAFKA-19173: Add `Feature` for "streams" group #19509

mjsax · 2025-04-18T02:37:22Z

Add new StreamsGroupFeature, disabled by default, and add "streams" as
default value to group.coordinator.rebalance.protocols.

Reviewers: Chia-Ping Tsai chia7712@gmail.com, David Jacot
david.jacot@gmail.com, Lucas Brutschy lbrutschy@confluent.io,
Justine Olshan jolshan@confluent.io, Andrew Schofield
aschofield@confluent.io, Jun Rao jun@confluent.io

chia7712

@mjsax thanks for this patch. Please take a look at my two small questions. thanks!

tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java

server-common/src/main/java/org/apache/kafka/server/common/StreamsGroupVersion.java

mjsax · 2025-04-24T06:04:19Z

Updated this PR.

metadata/src/test/java/org/apache/kafka/metadata/storage/FormatterTest.java

...n-tests/src/test/java/org/apache/kafka/streams/integration/InternalTopicIntegrationTest.java

mjsax · 2025-04-25T04:42:54Z

server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java

@@ -147,7 +158,7 @@ public enum MetadataVersion {
     * <strong>Think carefully before you update this value. ONCE A METADATA VERSION IS PRODUCTION,
     * IT CANNOT BE CHANGED.</strong>
     */
-    public static final MetadataVersion LATEST_PRODUCTION = IBP_4_0_IV3;
+    public static final MetadataVersion LATEST_PRODUCTION = IBP_4_0_IV3; // when do we update this to `IBP_4_1_IV2`+


Side question...

See my comment above.

Well, this is still on 4_0 though -- when should it get bumped to 4_1 ?

It gets bumped to 4_1 when MV 4_1-IV0 is production ready.

Should we remove this comment?

Should we remove this comment?

Yes :)

It gets bumped to 4_1 when MV 4_1-IV0 is production ready.

@jolshan -- from our discussion, my understanding is, that we would only add a MV if something is stable (and IBP_4_1_IV1 added for QfK should actually be removed)? Is ELR not production ready yet? And if not, why was a new MV added for it?

We add the MV version when we create the feature, but don't mark the MV as LATEST_PRODUCTION until the feature should be enabled by default.

I think we could remove 4_1 from QfK.

For ELR, the first MV (4.0) was due to a dependency on a new MV record value. The second one (4.1) will be marked as stable soon I believe. But Calvin would know better than me. :)

lucasbru

LGTM, but a review from somebody with more background on features would be good.

jolshan · 2025-04-28T16:31:12Z

server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java

@@ -117,6 +117,9 @@ public enum MetadataVersion {
    // Enables share groups. Note, share groups are for preview only in 4.1. (KIP-932).
    IBP_4_1_IV1(27, "4.1", "IV1", false),

+    // Enables "streams" groups. Note, streams groups are for early access only in 4.1. (KIP-1071).


We don't need two MV versions if we aren't adding any new metadata records for this feature. If you don't want to enable it by default, we can remove this one and just keep ibp 4.1-2

(Seems like the share groups one was also done incorrectly.)

@jolshan Does that mean EligibleLeaderReplicasVersion is also incorrect. I used that as the basis of ShareVersion.

That one is different because it requires an update to the MV log. (See the "true" for the first MV under didMetadataChange)

@AndrewJSchofield : Added a clarification question in https://github.com/apache/kafka/pull/19293/files#r2064970001 regarding IBP_4_1_IV1. We could follow up there.

jolshan · 2025-04-28T16:33:28Z

server-common/src/main/java/org/apache/kafka/server/common/StreamsVersion.java

+    SV_0(0, MetadataVersion.MINIMUM_VERSION, Map.of()),
+
+    // Version 1 enables "streams" groups (KIP-1071).
+    // Using metadata version IBP_4_2_IV1 disables it in AK 4.1 release, and enables it in AK 4.2 release.


ditto, we don't need a MV to "disable it". We just don't update the LATEST_PRODUCTION in this file while it is still unstable, and we don't mark the MV listed here as stable until we want it to be set by default.

jolshan

thanks!

junrao

@mjsax : Thanks for the PR. Added a few comments.

junrao · 2025-04-28T22:24:37Z

server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java

+    // *** STREAMS GROUPS BECOME PRODUCTION-READY IN THE FUTURE. ITS DEFINITION ALLOWS A STREAMS ***
+    // *** GROUPS FEATURE TO BE DEFINED IN 4.1 BUT TURNED OFF BY DEFAULT, ABLE TO BE TURNED ON   ***
+    // *** DYNAMICALLY TO TRY OUT THE EARLY ACCESS CAPABILITY.                                   ***
+    IBP_4_2_IV1(29, "4.2", "IV1", false);


It's a bit awkward to add an unstable MV for each unstable feature level. Could we introduce a single unstable MV that's shared between Qfk and streams MV?

I did discuss this with @jolshan and her recommendation was to use two individual versions. I'll let her comment about it.

I think it is easier to understand 1 new MV per feature. That way when your feature is ready you can mark it and you don't have to worry about other folks being ready etc.

Thanks for the explanation, @jolshan. Sound good to me then.

junrao · 2025-04-28T22:24:59Z

server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java

@@ -147,7 +158,7 @@ public enum MetadataVersion {
     * <strong>Think carefully before you update this value. ONCE A METADATA VERSION IS PRODUCTION,
     * IT CANNOT BE CHANGED.</strong>
     */
-    public static final MetadataVersion LATEST_PRODUCTION = IBP_4_0_IV3;
+    public static final MetadataVersion LATEST_PRODUCTION = IBP_4_0_IV3; // when do we update this to `IBP_4_1_IV2`+


Should we remove this comment?

junrao · 2025-04-28T22:57:21Z

server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java

@@ -117,6 +117,9 @@ public enum MetadataVersion {
    // Enables share groups. Note, share groups are for preview only in 4.1. (KIP-932).
    IBP_4_1_IV1(27, "4.1", "IV1", false),

+    // Enables "streams" groups. Note, streams groups are for early access only in 4.1. (KIP-1071).


@AndrewJSchofield : Added a clarification question in https://github.com/apache/kafka/pull/19293/files#r2064970001 regarding IBP_4_1_IV1. We could follow up there.

Add new StreamsGroupFeature, disabled by default, and add "streams" as default value to `group.coordinator.rebalance.protocols`.

Add new StreamsGroupFeature, disabled by default, and add "streams" as default value to `group.coordinator.rebalance.protocols`. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Jacot <david.jacot@gmail.com>, Lucas Brutschy <lbrutschy@confluent.io>, Justine Olshan <jolshan@confluent.io>, Andrew Schofield <aschofield@confluent.io>, Jun Rao <jun@confluent.io>

mjsax added streams core Kafka Broker KIP-1071 PRs related to KIP-1071 labels Apr 18, 2025

github-actions bot added tools kraft labels Apr 18, 2025

chia7712 reviewed Apr 18, 2025

View reviewed changes

tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorConfig.java Show resolved Hide resolved

dajac reviewed Apr 18, 2025

View reviewed changes

lucasbru reviewed Apr 24, 2025

View reviewed changes

metadata/src/test/java/org/apache/kafka/metadata/storage/FormatterTest.java Outdated Show resolved Hide resolved

...n-tests/src/test/java/org/apache/kafka/streams/integration/InternalTopicIntegrationTest.java Show resolved Hide resolved

mjsax force-pushed the kafka-19173-kip-1071-feature-flag branch from dc5bc03 to 43e212a Compare April 25, 2025 04:01

mjsax commented Apr 25, 2025

View reviewed changes

lucasbru approved these changes Apr 28, 2025

View reviewed changes

jolshan reviewed Apr 28, 2025

View reviewed changes

jolshan approved these changes Apr 28, 2025

View reviewed changes

junrao reviewed Apr 28, 2025

View reviewed changes

mjsax added 7 commits April 28, 2025 16:30

KAFKA-19173: Add Feature for "streams" group

0a95f6c

Add new StreamsGroupFeature, disabled by default, and add "streams" as default value to `group.coordinator.rebalance.protocols`.

follow up

0b71a60

rebased and fixed tests

4264172

fix rebase

eb2aa7b

review comments

bf0ac5e

fix version numbers

ac8d9ef

review and rebased

9a14f8b

mjsax force-pushed the kafka-19173-kip-1071-feature-flag branch from ac715e8 to 9a14f8b Compare April 29, 2025 01:14

fix

7ea18d7

mjsax merged commit b0a26bc into apache:trunk Apr 30, 2025
21 checks passed

mjsax deleted the kafka-19173-kip-1071-feature-flag branch April 30, 2025 05:51

Yunyung mentioned this pull request Apr 30, 2025

HOTFIX: Fix IQv2EndpointToPartitionsIntegrationTest #19597

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-19173: Add `Feature` for "streams" group #19509

KAFKA-19173: Add `Feature` for "streams" group #19509

mjsax commented Apr 18, 2025 •

edited by github-actions bot

Loading

chia7712 left a comment

mjsax commented Apr 24, 2025

mjsax Apr 25, 2025

jolshan Apr 28, 2025

mjsax Apr 28, 2025

jolshan Apr 28, 2025

junrao Apr 28, 2025

mjsax Apr 28, 2025

jolshan Apr 29, 2025 •

edited

Loading

lucasbru left a comment

jolshan Apr 28, 2025

jolshan Apr 28, 2025

AndrewJSchofield Apr 28, 2025

jolshan Apr 28, 2025 •

edited

Loading

junrao Apr 28, 2025 •

edited

Loading

jolshan Apr 28, 2025

jolshan left a comment

junrao left a comment

junrao Apr 28, 2025

mjsax Apr 28, 2025

jolshan Apr 28, 2025

junrao Apr 28, 2025

junrao Apr 28, 2025

junrao Apr 28, 2025 •

edited

Loading

KAFKA-19173: Add Feature for "streams" group #19509

KAFKA-19173: Add Feature for "streams" group #19509

Conversation

mjsax commented Apr 18, 2025 • edited by github-actions bot Loading

chia7712 left a comment

Choose a reason for hiding this comment

mjsax commented Apr 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jolshan Apr 29, 2025 • edited Loading

Choose a reason for hiding this comment

lucasbru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jolshan Apr 28, 2025 • edited Loading

Choose a reason for hiding this comment

junrao Apr 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jolshan left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junrao Apr 28, 2025 • edited Loading

Choose a reason for hiding this comment

KAFKA-19173: Add `Feature` for "streams" group #19509

KAFKA-19173: Add `Feature` for "streams" group #19509

mjsax commented Apr 18, 2025 •

edited by github-actions bot

Loading

jolshan Apr 29, 2025 •

edited

Loading

jolshan Apr 28, 2025 •

edited

Loading

junrao Apr 28, 2025 •

edited

Loading

junrao Apr 28, 2025 •

edited

Loading