From 42c374ec480925417bdd3183ef44ab253a19189b Mon Sep 17 00:00:00 2001 From: Feng Ruohang Date: Mon, 24 Aug 2026 20:49:19 +0800 Subject: [PATCH] docs: record CopyObject checksum invariant Signed-off-by: Feng Ruohang --- .../blog/design/copyobject-checksum/index.md | 74 +++++++++++++++++++ .../design/copyobject-checksum/index.zh.md | 74 +++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 content/blog/design/copyobject-checksum/index.md create mode 100644 content/blog/design/copyobject-checksum/index.zh.md diff --git a/content/blog/design/copyobject-checksum/index.md b/content/blog/design/copyobject-checksum/index.md new file mode 100644 index 00000000..3bdb59f3 --- /dev/null +++ b/content/blog/design/copyobject-checksum/index.md @@ -0,0 +1,74 @@ +--- +title: "CopyObject Checksums Must Cover Logical Object Bytes" +date: 2026-08-24 +lastmod: 2026-08-24 +author: "Ruohang Feng" +summary: > + When destination compression was enabled, SILO could persist a CopyObject checksum of the S2 storage stream instead of the logical S3 object. This record explains the plaintext-reader invariant, verification boundary, related fixes, and remediation of older objects. +tags: [Design, S3, Compatibility, Checksum] +weight: 10 +draft: false +url: "/blog/design/copyobject-checksum/" +--- + +This is the design and verification record for [SILO #63](https://github.com/pgsty/silo/issues/63). + +**Status:** the checksum-domain fix was merged through [PR #66](https://github.com/pgsty/silo/pull/66); public release pending. +**Related fixes:** metadata-only transform state [#67](https://github.com/pgsty/silo/issues/67) through [PR #69](https://github.com/pgsty/silo/pull/69), and CopyObjectResult checksum fields [#68](https://github.com/pgsty/silo/issues/68) through [PR #70](https://github.com/pgsty/silo/pull/70); both merged, public release pending. +**Upstream client:** [minio-go #2295](https://github.com/minio/minio-go/pull/2295). +**Release boundary:** a merge does not prove that a release artifact, package, or container image already contains the fix. + +## The defect {#defect} + +CopyObject reads the source as logical object data, then may compress and encrypt the destination storage stream. The old handler installed a requested server-side checksum on a reader that already represented compressed bytes: + + logical object -> S2 compression -> checksum -> optional encryption -> storage + +The digest was valid but covered the wrong byte domain. A client downloading and independently hashing the object therefore obtained a different value. The API reproduction was deterministic: + + stored CRC32 before the fix: hN7ytg== + logical object CRC32: 1WxbLg== + +All five algorithms implemented by this SILO baseline were affected: CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. Compression combined with encryption made the wrong value nondeterministic because encrypted-stream S2 padding is randomized. + +## Accepted invariant {#invariant} + +The logical checksum reader is now separate from storage transformation readers: + + logical object + -> server-side checksum + -> optional S2 compression + -> storage hash + -> optional server-side encryption + -> erasure coding and commit + +The handler installs the hasher before starting the compression goroutine. PutObjReader retains the logical reader even when its active storage reader is replaced. At EOF, the object layer requires the checksum to exist, be valid, and match the expected base algorithm before committing metadata. + +This reuses the checksumReader contract introduced for multipart upload. It adds no second abstraction, no second object read, and no new on-disk representation. + +## Verification boundary {#verification} + +The permanent API suite covers all five algorithms and default CRC64NVME; uncompressed, compressed, encrypted-only, and compressed-plus-encrypted destinations; SSE-C and SSE-S3; encrypted and compressed sources; versioned buckets; full and multipart-composite source checksums; in-place copy; zero-length and threshold data; indexed S2 streams; ETag; body round trip; HEAD/GET checksum mode; and internal invariant failures. + +The regression is red on the unfixed baseline and green on the repaired tree. Focused race tests, shuffled repeated runs, full cmd tests, the CGO-disabled kqueue/dev CI shape, lint, vet, cross-compilation, compatibility guards, and remote CI were also required before merge. + +## Adjacent defects kept separate {#adjacent} + +Adversarial review found two inherited defects in nearby code: + +1. A metadata/reference-only self-copy could change compression markers without rewriting referenced data. Versioned SSE-C key rotation could also fall into an invalid rewrite. This is isolated in [#67](https://github.com/pgsty/silo/issues/67). +2. Successful CopyObject XML omitted checksum elements after the checksum was committed. The server response fix is [#68](https://github.com/pgsty/silo/issues/68); minio-go also discarded those fields and is followed in [#2295](https://github.com/minio/minio-go/pull/2295). + +Legacy federated UploadPartCopy checksum recovery is a different API and remains [#64](https://github.com/pgsty/silo/issues/64). + +The archived upstream minio/minio tree retains the original placement. silo-pkg does not own this reader chain. MCLI switches from server-side copy to download/upload when --checksum is requested, and SILO Console only passes CopyObject through minio-go, so neither required a duplicate server fix. + +## Existing objects {#existing-objects} + +The repair affects future CopyObject operations. It does not scan or rewrite checksum metadata already stored by an affected version. + +Objects are candidates for verification when they were created by CopyObject, destination compression matched their key or content type, and they carry an additional S3 checksum. Retrieve the object with checksum mode enabled, independently hash the downloaded logical bytes with the reported algorithm, and compare the Base64 values. + +To repair an object, copy it to a new key while explicitly selecting the checksum algorithm. An in-place copy is possible with x-amz-metadata-directive: REPLACE, but it rewrites the object and replaces the current value on an unversioned bucket; a versioned bucket receives a new version. Validate retention, legal hold, metadata, tags, encryption keys, free capacity, and rollback requirements before bulk remediation. + +SILO does not perform automatic online backfill because that would read and rewrite user data outside an explicit S3 operation. diff --git a/content/blog/design/copyobject-checksum/index.zh.md b/content/blog/design/copyobject-checksum/index.zh.md new file mode 100644 index 00000000..8bcd56c9 --- /dev/null +++ b/content/blog/design/copyobject-checksum/index.zh.md @@ -0,0 +1,74 @@ +--- +title: "CopyObject Checksum 必须覆盖逻辑对象字节" +date: 2026-08-24 +lastmod: 2026-08-24 +author: "冯若航" +summary: > + 目标端启用压缩时，SILO 可能把 S2 存储流的 checksum 当成 CopyObject 逻辑对象 checksum 持久化。本文记录明文 reader 不变量、验证边界、独立后续修复与旧对象处理方法。 +tags: [设计, S3, 兼容性, Checksum] +weight: 10 +draft: false +url: "/zh/blog/design/copyobject-checksum/" +--- + +本文是 [SILO #63](https://github.com/pgsty/silo/issues/63) 的设计与验证归档。 + +**状态：** checksum 数据域修复已通过 [PR #66](https://github.com/pgsty/silo/pull/66) 合并；公开发布待完成。 +**相关修复：** metadata-only transform state [#67](https://github.com/pgsty/silo/issues/67) 已通过 [PR #69](https://github.com/pgsty/silo/pull/69) 合并，CopyObjectResult checksum 字段 [#68](https://github.com/pgsty/silo/issues/68) 已通过 [PR #70](https://github.com/pgsty/silo/pull/70) 合并；公开发布仍待完成。 +**上游客户端：** [minio-go #2295](https://github.com/minio/minio-go/pull/2295)。 +**发布边界：** 合并不代表公开 release、软件包或容器镜像已经包含修复。 + +## 缺陷本质 {#defect} + +CopyObject 先把源对象读成逻辑数据，再按目标配置压缩并加密存储流。旧处理器把 server-side checksum 挂在了已经代表压缩字节的 reader 上： + + 逻辑对象 -> S2 压缩 -> checksum -> 可选加密 -> 存储 + +digest 在数学上有效，却覆盖了错误的数据域。客户端下载对象后对逻辑字节独立计算，结果自然不同。API 级复现是确定性的： + + 修复前持久化 CRC32：hN7ytg== + 逻辑对象 CRC32： 1WxbLg== + +当前基线实现的 CRC32、CRC32C、CRC64NVME、SHA1、SHA256 都会受影响。压缩叠加加密时，S2 加密流填充包含随机值，错误 checksum 甚至会变成非确定值。 + +## 采用的不变量 {#invariant} + +逻辑 checksum reader 现在与存储变换 reader 分离： + + 逻辑对象 + -> server-side checksum + -> 可选 S2 压缩 + -> 存储流 hash + -> 可选服务端加密 + -> 纠删码与提交 + +处理器必须在压缩 goroutine 启动前安装 hasher。即使压缩或加密替换活动存储 reader，PutObjReader 仍保存逻辑 reader。读到 EOF 后，对象层要求 checksum 存在、有效并与预期 base algorithm 一致，随后才允许提交 metadata。 + +该设计直接复用 multipart checksum 的 checksumReader 契约，不创建第二套抽象，不重读对象，也不改变盘上格式。 + +## 验证边界 {#verification} + +永久 API 测试覆盖五种算法与默认 CRC64NVME；不压缩、压缩、仅加密和压缩加密；SSE-C 与 SSE-S3；加密源和压缩源；版本化桶；full 与 multipart-composite 来源；原地复制；零长度、压缩阈值与带索引 S2 流；ETag、正文 round trip、HEAD/GET checksum mode，以及内部不变量失败。 + +同一条回归在未修复基线上失败，在修复树上通过。合并前还要求定向 race、随机顺序重复运行、全量 cmd、禁用 CGO 的 kqueue/dev CI 形态、lint、vet、交叉编译、兼容性 guard 与远端 CI 全部通过。 + +## 刻意拆开的邻接缺陷 {#adjacent} + +对抗审查又发现两个继承缺陷： + +1. metadata/reference-only self-copy 可能在没有重写引用数据时改变压缩标记；版本化 SSE-C 密钥轮换还可能落入非法重写。该问题独立收敛在 [#67](https://github.com/pgsty/silo/issues/67)。 +2. 目标 checksum 已提交后，成功的 CopyObject XML 仍不返回 checksum 元素。服务端由 [#68](https://github.com/pgsty/silo/issues/68) 跟踪；minio-go 还会丢弃字段，对应 [#2295](https://github.com/minio/minio-go/pull/2295)。 + +旧 federation 的 UploadPartCopy checksum 恢复属于另一个 API，继续由 [#64](https://github.com/pgsty/silo/issues/64) 跟踪。 + +归档的上游 minio/minio 仍保留原始 reader 放置方式。silo-pkg 不拥有该 reader 链。MCLI 在指定 --checksum 时会从 server-side copy 切换为下载再上传，SILO Console 只通过 minio-go 透传 CopyObject，因此两者不需要复制一份服务端修复。 + +## 已存在对象 {#existing-objects} + +修复只影响此后的 CopyObject，不会自动扫描或重写旧版本已经保存的 checksum metadata。 + +由 CopyObject 创建、当时目标 key 或内容类型命中压缩配置、并带有额外 S3 checksum 的对象值得核验。使用 checksum mode 取回对象与 checksum，再用同一算法独立计算下载后的逻辑字节并比较 Base64 值。 + +修复时可显式指定 checksum algorithm，把对象复制到新 key。也可以带 x-amz-metadata-directive: REPLACE 做原地复制，但这会重写对象：未版本化桶替换当前值，版本化桶创建新版本。批量处理前必须确认 retention、legal hold、metadata、tag、加密密钥、剩余空间和回滚要求。 + +SILO 不做自动在线回填，因为那意味着在没有显式 S3 操作的情况下读取并重写用户数据。