Amazon S3 Tables now support all Apache Iceberg V3 data types

Amazon S3 Tables now support all data types in the Apache Iceberg V3 specification, allowing users to create new V3 tables or upgrade existing V2 tables in place. This update addresses common limitations in V2, such as slow query performance from positional delete files and the need to parse semi-structured JSON strings, by introducing native support for deletion vectors, row lineage, and new data types like variant, nanosecond timestamps, geometry, geography, and unknown.

Key V3 Capabilities

The new features aim to reduce storage costs, query latency, and pipeline complexity:

• Deletion vectors: Replace V2’s positional delete files with a compact binary format, significantly reducing compaction time and delete file overhead for operations like compliance deletions.
• Row lineage: Automatically adds _row_id and _last_updated_sequence_number to each record, enabling downstream pipelines to query changed rows without scanning the full table.
• New data types: Allow native storage of semi-structured, geospatial, and nanosecond-precision data. The variant type stores semi-structured data in columnar format, using statistics for file pruning to reduce I/O compared to parsing JSON strings.

Implementation and Compatibility

Users can upgrade existing tables atomically without rewriting data using ALTER TABLE ... SET TBLPROPERTIES ('format-version' = '3'). This is a one-way operation; the Apache Iceberg specification does not support downgrading from V3 to V2. Existing V2 readers continue to work on upgraded tables until full V3 adoption.

To use the new V3 data types, engines must be built on Apache Spark 4.0 or later, such as AWS Glue 6.0 or later, or Amazon EMR release 8.1 or later. These data types are supported only for tables using the Parquet file format. Additionally, columns of type variant, geometry, geography, or nanosecond timestamp cannot be included in a table’s sort order for compaction.

Availability

Amazon S3 Tables support for all Apache Iceberg V3 data types is now available in all AWS Regions where S3 Tables are supported. There is no additional charge for V3 support; standard S3 Tables pricing applies. Users can create V3 tables via the Amazon S3 console, AWS CLI, or any engine supporting the Iceberg REST Catalog API.