Skip to content

365.add support of custom codecs in topics #447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: release_v2.4.0
Choose a base branch
from

Conversation

ekuvardin
Copy link
Contributor

@ekuvardin ekuvardin commented Apr 27, 2025

Finally after all discussion try to implement feature
#365

What major changes:

  1. Replace Codec enum to number
  2. Add ability to write custom codec using CodecRegistry local to TopicClient
  3. Changed a lot of interface. Some interfaces can't be change in backward compability(they use enum)
    methods that can be used in backward compability mark as deprecated
    (it's all syn/async reader/writer)
  4. Pass CodecRegistry to any ReaderImpl/Writer/Impl
  5. Write 3 new test classes
    5.1 Test on custom codec + include test on all existed predefind codec
    5.2 Test on register
    5.3 Test on old deprecated methods

To use custom codec:

  1. Implement interface CustomTopicCodec
  2. Write your implementation.
  3. Register Codec
  4. For writer - specify codec in WritterSettings
  5. For readers - for TopicClient it have to be registered
    Summary
        CustomTopicCodec codec1 = new CustomCustomTopicCode(1);
        client1.registerCodec(10113, codec1);
          WriterSettings settings = WriterSettings.newBuilder()
                .setTopicPath(topicName)
                .setCodec(10113)
                .build();

Seems next days try to pass all tests for all jdks

ekuvardin added 15 commits April 6, 2025 22:26
…-in-topics' into 365.Add-support-of-custom-codecs-in-topics

# Conflicts:
#	topic/src/main/java/tech/ydb/topic/utils/Encoder.java
#	topic/src/main/java/tech/ydb/topic/write/impl/AsyncWriterImpl.java
#	topic/src/main/java/tech/ydb/topic/write/impl/SyncWriterImpl.java
#	topic/src/main/java/tech/ydb/topic/write/impl/WriterImpl.java
#	topic/src/test/java/tech/ydb/topic/impl/YdbTopicsCustomCodecIntegrationTest.java
#	topic/src/test/java/tech/ydb/topic/impl/YdbTopicsCustomCodecTest.java
…custom-codecs-in-topics

# Conflicts:
#	topic/src/main/java/tech/ydb/topic/utils/Encoder.java
@codecov-commenter
Copy link

codecov-commenter commented Apr 27, 2025

Codecov Report

Attention: Patch coverage is 81.25000% with 15 lines in your changes missing coverage. Please review.

Project coverage is 59.22%. Comparing base (4ece287) to head (d52df35).
Report is 3 commits behind head on release_v2.4.0.

Files with missing lines Patch % Lines
...main/java/tech/ydb/topic/impl/TopicClientImpl.java 50.00% 6 Missing ⚠️
...ic/src/main/java/tech/ydb/topic/impl/RawCodec.java 40.00% 3 Missing ⚠️
...va/tech/ydb/topic/description/SupportedCodecs.java 50.00% 1 Missing ⚠️
...c/src/main/java/tech/ydb/topic/impl/GzipCodec.java 80.00% 1 Missing ⚠️
...c/src/main/java/tech/ydb/topic/impl/LzopCodec.java 83.33% 1 Missing ⚠️
...c/src/main/java/tech/ydb/topic/impl/ZstdCodec.java 80.00% 1 Missing ⚠️
...ava/tech/ydb/topic/write/impl/AsyncWriterImpl.java 0.00% 1 Missing ⚠️
.../java/tech/ydb/topic/write/impl/MessageSender.java 50.00% 1 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                  @@
##             release_v2.4.0     #447      +/-   ##
====================================================
+ Coverage             58.75%   59.22%   +0.47%     
- Complexity             2171     2207      +36     
====================================================
  Files                   320      323       +3     
  Lines                 12938    12965      +27     
  Branches               1269     1271       +2     
====================================================
+ Hits                   7602     7679      +77     
+ Misses                 4720     4686      -34     
+ Partials                616      600      -16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@alex268
Copy link
Member

alex268 commented Apr 28, 2025

Tests YdbTopicsIntegrationTest.step03_readNextHalfWithoutCommit and YdbTopicsIntegrationTest.step04_readNextHalfWithCommit flaps, so you don't need to fix them. Just rerun

@ekuvardin
Copy link
Contributor Author

@alex268
What have changed, please take a look at code

  1. Codec now is an interface. All custom and predefined topics must override it's methods.
    Predefined codec now is a singleton.
  2. Remove unregister codec. CodecRegistry under the hood contains HashMap cause works in a single thread
  3. Remove old deprecated classes
  4. Mark one test with ignore YdbTopicsCodecIntegrationTest.writeWithCustomCodec10000 - that's a pity. We already discussed about it, that SyncWriter have some bugs with flush.
  5. CI hasn't passed cause
    java.util.List<java.lang.Integer> cannot be converted to java.util.List<tech.ydb.topic.description.Codec>
    in examples
    Seems we have to change examples in ydb-cookbook

Add tests for rewrite predefined codec.
Remove ProtoUtils class
@alex268 alex268 requested review from pnv1 and alex268 May 5, 2025 13:40
@alex268 alex268 changed the base branch from master to release_v2.4.0 May 6, 2025 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants