Repository navigation
[Feature][Lakehouse] Support Iceberg REST Catalog read restrictions #68917
Description
Search before asking
- I had searched in the issues and found no similar issues.
Description
Motivation
Iceberg REST Catalog read restrictions provide a standard way for a catalog to return caller-specific row filters and column masking decisions with loadTable.
Doris should enforce these restrictions when querying external Iceberg tables. This would allow organizations to define access policies in their source catalog and use Doris for interactive analytics while preserving the catalog’s access decisions.
For example, the same table could expose only European customers to one user, while also masking sensitive columns according to that user’s privileges.
Proposed implementation
1. Parse and bind restrictions
Read and preserve the optional read-restrictions object from the loadTable response:
required-row-filter: translate the Iceberg predicate into Doris expressions.required-column-projections: bind each required action to the corresponding field.
Resolve references using Iceberg field IDs, with correct handling of schema evolution. Preserve the specified output types and field identities.
Implement the nine standardized projection actions:
| Category | Actions |
|---|---|
| Masking | mask-alphanum, mask-to-fixed-value, replace-with-null |
| Partial visibility | show-first-4, show-last-4 |
| Temporal truncation | truncate-to-year, truncate-to-month |
| Hashing | sha-256-global, sha-256-query-local |
Follow the specification’s type, NULL, Unicode, timestamp, and byte-encoding semantics. Query-local hashing requires one cryptographically random salt of at least 16 bytes per query, used consistently across all execution fragments.
2. Enforce restrictions throughout query execution
The logical evaluation order must be:
- Apply the mandatory row filter to original, unmasked values.
- Apply required column projections to surviving rows.
- Evaluate user query operations against the restricted relation.
In relational terms:
UserQuery(Mask(RequiredRowFilter(IcebergTable)))
Ensure predicates, joins, aggregates, derived expressions, and ordering observe this restricted relation. Optimizer rewrites, predicate pushdown, runtime filters, and metadata-only optimizations must preserve these semantics.
Catalog-mandated restrictions must remain effective when combined with Doris-local authorization.
3. Fail explicitly when enforcement is impossible
If an applicable restriction cannot be parsed, bound, or fully executed, fail the query. Do not silently omit the restriction or substitute raw, partially restricted, or empty results.
Validate restrictions according to the specification, including action/type compatibility, duplicate projection field IDs, map-key projections, overlapping parent/descendant projections, and null replacement on required fields.
Absent or empty restrictions impose no additional restrictions. Projections on columns that are not actually read do not apply.
4. Preserve identity and authorization context
Use the effective caller identity when loading the table, and retain the returned restrictions with the table handle used by the query.
Scope cached authorization decisions, plans, and results appropriately. Define refresh behavior for policy and role changes: an unchanged Iceberg snapshot does not imply unchanged authorization.
All supported reads using the load response must honor its restrictions, including any subsequent scan-planning operations.
Document the requirements for a trusted Doris deployment, including credential handling and prevention of alternate access paths that bypass enforcement.
5. Handle acceleration and materialization boundaries
The initial implementation should support direct external-table queries.
Materialized-view substitution should be allowed only when equivalent enforcement can be established. For CTAS, INSERT ... SELECT, and materialized-view refresh, enforce restrictions on the source read or explicitly reject unsupported cases.
Destination data requires its own authorization lifecycle; copied data must not be assumed to inherit future source-policy changes.
Acceptance criteria
- Queries correctly enforce row filters and all supported projection actions.
- Column renames retain correct field-ID binding.
- Predicates, joins, aggregates, and derived expressions cannot bypass filtering or masking.
- Unknown, invalid, or unsupported applicable restrictions cause an explicit query failure.
- Different users querying the same table cannot reuse incompatible authorization decisions or results.
- Policy changes are reflected through the documented refresh/invalidation behavior, including when the data snapshot is unchanged.
- Query-local hashing is consistent across fragments and changes between queries.
- Tests cover optimizer rewrites, supported historical reads, scan-planning paths, and materialization boundaries.
- Documentation identifies supported capabilities and deployment requirements.
References
- Iceberg read restrictions specification: apache/iceberg#13879
- Iceberg REST Catalog specification
- Snowflake: A Step Toward Interoperable Governance
- Spark enforcement POC: apache/iceberg#16082
Use case
No response
Related issues
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity