Add internal helpers for SMB fluent API experimentation #5814
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds internal bridge classes to enable experimental fluent API development for SMB operations. These helpers expose package-private methods needed for advanced SMB use cases without changing the public API. Changes: - Add SMBCollectionHelper: bridge to access FileOperations and metadata extraction from SortedBucketIO.Read and TransformOutput - Add SmbValidationHelper: bridge for source validation and metadata access - Fix AvroFileOperations codec serialization: use transient CodecFactory with custom serialization to properly handle Avro codec across serialization - Make SortedBucketTransform.BucketSource package-visible to enable custom readers - Add getSplitPointsConsumed/Remaining overrides to prevent unnecessary split attempts - Make SortedBucketIO.KeyFn public to support custom key extraction All new classes are marked @internal and subject to change without notice. These changes are non-breaking and enable experimentation with SMB-aware libraries and tooling.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5814 +/- ##
=======================================
Coverage 61.44% 61.45%
=======================================
Files 314 314
Lines 11429 11429
Branches 812 812
=======================================
+ Hits 7023 7024 +1
+ Misses 4406 4405 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds internal bridge classes to enable experimental fluent API development for SMB operations. These helpers expose package-private methods needed for advanced SMB use cases without changing the public API.
Changes
New Files
SMBCollectionHelper.java(272 lines) - Bridge to access FileOperations and metadata extraction fromSortedBucketIO.ReadandTransformOutputSmbValidationHelper.java(104 lines) - Bridge for source validation and metadata accessModified Files
AvroFileOperations.java- Fixed codec serialization using transientCodecFactorywith custom serialization to properly handle Avro codec across serialization boundariesSortedBucketTransform.java- MadeBucketSourcepackage-visible and addedgetSplitPointsConsumed/Remainingoverrides to prevent unnecessary split attemptsSortedBucketIO.java- MadeKeyFninterface public to support custom key extractionRationale
These changes enable experimentation with SMB-aware libraries and tooling (similar to how
SortedBucketIOUtilprovides testing helpers). All new classes are marked@Internaland explicitly documented as subject to change without notice.Impact
@InternalThis follows the precedent of existing internal helpers in Scio and enables users to build experimental SMB tooling without requiring changes to the main codebase.