Repository navigation
Add data-file byte totals to manifest_list entries, so table size can be read without scanning manifest bodies #17743
Description
Feature Request / Improvement
manifest_list entries summarize counts and rows, but not bytes:
added_files_count existing_files_count deleted_files_count
added_rows_count existing_rows_count deleted_rows_count
manifest_length <- the manifest file's own size, not the data it describes
Every data_file already carries file_size_in_bytes, so the number exists — it just isn't rolled up where it can be read cheaply. Getting a table's reachable bytes today means opening every manifest body and summing per entry, which is decompression-bound and almost entirely redundant: manifests are immutable and inherited largely unchanged between snapshots, so the same bytes are re-inflated every run.
Proposal
Three optional fields on manifest_file, mirroring the existing triplets:
added_files_size_in_bytes
existing_files_size_in_bytes
deleted_files_size_in_bytes
Zero write cost — the writer already holds every file_size_in_bytes. Backward compatible: absent means readers fall back to reading bodies.
Why snapshot summary total-files-size doesn't cover it
It gives one snapshot's total, and summing across live snapshots double counts files shared between them. Manifests are the natural dedup unit, since a manifest is inherited wholesale — dedup by manifest_path, then sum. That composes; per-snapshot totals don't.
Useful well beyond one use case: cost attribution, compaction and retention planning, catalog-wide dashboards. #14803 and #14820 added delete-file size totals to PartitionsTable; this is the same idea in the manifest list, readable without a scan.
I found no prior discussion (searched added_files_size, manifest_list + bytes, spec-labelled size proposals) — happy to be pointed at it. As a spec change this presumably wants a dev@ thread and spec PR; filing here to gauge support.
Activity
hemanthboyina commentedon Aug 21, 2026
thanks for this @HaoXuAI I do feel this is well needed with respective to v4 too. Started a mail chain in dev list.
RussellSpitzer commentedon Aug 21, 2026
I think we get this for free in V4? Since all content stats will roll up?
haoxu0 commentedon Aug 24, 2026
Thanks @RussellSpitzer — I checked against main (
f0e7de3c2f) and I don't think v4 gives us this, for a stronger reason than I first thought.V4Metadata.MANIFEST_LIST_SCHEMA(core/src/main/java/org/apache/iceberg/V4Metadata.java:32-49) is field-for-field the v3 set — three*_files_count, three*_rows_count, no byte total. So the v4 writer as it stands today doesn't summarize bytes.And
content_statscan't be rolled up into one:FieldStatsexposeslowerBound,upperBound,valueCount,nullValueCount,nanValueCountandavgValueSizeInBytes, and that last one is documented as the average in-memory, uncompressed size for memory estimation. There's no on-disk byte stat in there. On-disk size lives inDataFile.FILE_SIZE(104) on the entry, inside the manifest body — so answering "how big is this table" is still a full manifest scan, which is the IO this issue is about.#16025 is also widening the gap rather than closing it: it adds
replaced_files_count,modified_files_count,replaced_rows_countandmodified_rows_countto the summary, still with no*_files_size_in_bytes. Since that PR is already reshaping this struct, it seems like the natural place to add them — and the write-side cost is three longs the writer is already summing per entry.