system.iceberg_metadata_log

Querying in ClickHouse Cloud

The data in this system table is held locally on each node in ClickHouse Cloud. Obtaining a complete view of all data, therefore, requires the clusterAllReplicas function. See here for further details.

Description

The system.iceberg_metadata_log table records metadata access and parsing events for Iceberg tables read by ClickHouse. It provides detailed information about each metadata file or entry processed, which is useful for debugging, auditing, and understanding Iceberg table structure evolution.

This table logs every metadata file and entry read from Iceberg tables, including root metadata files, manifest lists, and manifest entries. It helps users trace how ClickHouse interprets Iceberg table metadata and diagnose issues related to schema evolution, file resolution, or query planning.

Note

This table is primarily intended for debugging purposes.

Controlling log verbosity

You can control which metadata events are logged using the iceberg_metadata_log_level setting.

To log all metadata used in the current query:

SELECT * FROM my_iceberg_table SETTINGS iceberg_metadata_log_level = 'manifest_file_entry';

SYSTEM FLUSH LOGS iceberg_metadata_log;

SELECT content_type, file_path, row_in_file
FROM system.iceberg_metadata_log
WHERE query_id = '{previous_query_id}';

To log only the root metadata JSON file used in the current query:

SELECT * FROM my_iceberg_table SETTINGS iceberg_metadata_log_level = 'metadata';

SYSTEM FLUSH LOGS iceberg_metadata_log;

SELECT content_type, file_path, row_in_file
FROM system.iceberg_metadata_log
WHERE query_id = '{previous_query_id}';

See more information in the description of the iceberg_metadata_log_level setting.

Good To Know

• Use iceberg_metadata_log_level at the query level only when you need to investigate your Iceberg table in detail. Otherwise, you may populate the log table with excessive metadata and experience performance degradation.
• The table contains duplicate entries, as it is intended primarily for debugging and does not guarantee uniqueness per entity. Separate rows store content and pruning status because they are collected at different moments in a program. Content is collected when the metadata is read, pruning status is collected when the metadata is checked for pruning. Never rely on the table itself for deduplication.
• If you use a content_type more verbose than ManifestListMetadata, the Iceberg metadata cache is disabled for manifest lists.
• Similarly, if you use a content_type more verbose than ManifestFileMetadata, the Iceberg metadata cache is disabled for manifest files.
• If the SELECT query was cancelled or failed, the log table may still contain entries for metadata processed before the failure but will not contain information about metadata entities that were not processed.

Columns

• hostname (LowCardinality(String)) — Hostname of the server executing the query.
• event_date (Date) — Date of the entry.
• event_time (DateTime) — Event time.
• query_id (String) — Query id.
• content_type (Enum8('None' = 0, 'Metadata' = 1, 'ManifestListMetadata' = 2, 'ManifestListEntry' = 3, 'ManifestFileMetadata' = 4, 'ManifestFileEntry' = 5)) — Content type.
• table_path (String) — Table path.
• file_path (String) — File path.
• content (String) — Content in a JSON format (json file content, avro metadata or avro entry).
• row_in_file (Nullable(UInt64)) — Row in file.
• pruning_status (Nullable(Enum8('NotPruned' = 0, 'PartitionPruned' = 1, 'MinMaxIndexPruned' = 2))) — Status of partition pruning or min-max index pruning for the file.

content_type values

• None: No content.
• Metadata: Root metadata file.
• ManifestListMetadata: Manifest list metadata.
• ManifestListEntry: Entry in a manifest list.
• ManifestFileMetadata: Manifest file metadata.
• ManifestFileEntry: Entry in a manifest file.

See also

• Iceberg Table Engine
• Iceberg Table Function
• system.iceberg_history