Partition field source columns should be primitive typed #25144
+538
−118
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Logical DATE type should be bound to AVRO_INT only. Previously we used
AVRO_LONG, so the Avro lib would throw when exercising this code path.
Also adds a unit test for Iceberg types that are backed by Avro logicalTypes.
Note that timestamptz serialization is currently bugged, as it relies on an
extra non-standard field ('adjust-to-utc') in the Avro type description.
Technically this should appear in both micro precision timestamp types,
'false' for 'timestamp'; 'true' for 'timestamptz'. As written, both timestamp
types serialize to logical type timestamp-micros, with no way to distinguish
between the two in deser.
See https://iceberg.apache.org/spec/#avro for detail
Signed-off-by: Oren Leiman <[email protected]>
To support special promotions that affect partition transforms type_promoted: - no - yes - unless_partition Signed-off-by: Oren Leiman <[email protected]>
Signed-off-by: Oren Leiman <[email protected]>
Specify either an annotate error or a validate error. This simplifies some of the conditional logic around instantiating test suites. No functional changes. Signed-off-by: Oren Leiman <[email protected]>
Signed-off-by: Oren Leiman <[email protected]>
Signed-off-by: Oren Leiman <[email protected]>
…otion Signed-off-by: Oren Leiman <[email protected]>
Though this restriction does not appear in the spec, dropping a data column that also appears in the table's partition spec can cause validation errors in clients, downstream. It may be possible to avoid this by performing live partition spec reconciliation inline with the schema update, but for now we simply reject such an update out of hand. Also slightly refactors to consolidate error checks in validate_schema_transform. Signed-off-by: Oren Leiman <[email protected]>
Signed-off-by: Oren Leiman <[email protected]>
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is sequenced on #25114 - only commit of interest is 62c61f7
Backports Required
Release Notes