Iceberg Metadata for Variant Shredding

[aihuaxu]

Proposed Change

To support file pruning for shredded subcolumns, it is necessary to collect subcolumn metadata in the manifest files. The proposal includes the following key points:

• Lower and upper bounds for subcolumns of a Variant column are encoded as Variant values.

• Other metadata, such as value_counts and null_value_counts, will not be collected for Variant columns.

• Type promotion from primitive types to Variant will not be supported.

• Bounds will be stored only if the subcolumn values match the shredded types.

This approach ensures efficient pruning while minimizing changes to the metadata format.

Proposal document

https://docs.google.com/document/d/1gAvt0x_ez89O8y-YqkCdMnTEykb-583YslYOgzf5sPg/edit?tab=t.0#heading=h.escoiuuiw331

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

Part of #10392