CopyObject server-side checksum can cover transformed data when compression is enabled

[Vonng]

Problem

Reviewing #46 exposed a separate checksum-placement problem in CopyObject.

When CopyObject needs a server-computed destination checksum and destination compression is enabled, the current handler starts newS2CompressReader and later attaches AddServerSideChecksumHasher to srcInfo.Reader. That reader represents the storage stream, not necessarily the logical S3 plaintext stream. The compression goroutine can also begin consuming input as soon as it is created.

Static data-flow therefore indicates that the stored checksum can cover compressed bytes, or miss bytes consumed before the hasher is attached, instead of covering the logical destination object. This needs an API-level reproduction before the final patch is accepted.

Relevant code:

• cmd/object-handlers.go: destination compression and AddServerSideChecksumHasher
• cmd/object-api-utils.go: newS2CompressReader and PutObjReader
• cmd/erasure-object.go: RawServerSideChecksumResult persistence

Relationship to #46

#46 will intentionally fix only UploadPart and UploadPartCopy. It will establish one shared internal invariant for later fixes:

1. PutObjReader retains a dedicated logical plaintext checksum reader, distinct from the storage reader.
2. A fallback server-side hasher is installed in the handler before compression or encryption can consume data.
3. The object layer validates the effective algorithm and persists the result.

The CopyObject fix should reuse that exact mechanism in a separate PR. It must not add a second checksum-reader abstraction or attach the hasher later in the object layer.

Proposed direction

• Reuse the unexported checksum reader introduced by UploadPart requires a per-part checksum header that AWS S3 computes server-side #46.
• Preserve the pre-compression CopyObject reader as the logical checksum source.
• Install the server-side hasher before newS2CompressReader starts.
• Keep supplied client checksum validation unchanged.
• Verify that the checksum returned by HEAD/GET covers logical object bytes for compressed and encrypted destinations.

Acceptance criteria

• API-level reproduction of the current compressed CopyObject behavior.
• Correct server-computed CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 where supported.
• Tests for default, compressed, encrypted, and compressed plus encrypted modes.
• Explicit SSE-C and SSE-S3 coverage where applicable.
• Targeted race test proving no hasher installation occurs after reader consumption begins.
• No public API or on-disk format change beyond the existing checksum metadata representation.

This issue is deliberately separate from #46 so the multipart P0 can be reviewed, released, and rolled back independently.

[Vonng]

Implementation and adversarial-review completion:

• fix: checksum CopyObject data before compression #66 merged the checksum-domain fix and closed this issue.
• The permanent API suite proves the unfixed baseline stores the transformed-stream CRC32 (hN7ytg==) instead of the logical-object CRC32 (1WxbLg==).
• Coverage includes all five supported algorithms, default CRC64NVME, compressed/plain, encrypted-only, compressed+encrypted, SSE-C, SSE-S3, encrypted/compressed sources, versioning, full/composite source checksums, in-place copy, thresholds, indexed streams, ETag, HEAD/GET, body round trip, failure cleanup, race, and repeated shuffled runs.
• Remote CI passed DCO, build/vet, lint/generated, race/S3 Select, cmd/internal tests, cross-compilation, and vulnerability analysis.

The original issue hypothesis contained two possible failure modes. Dynamic reproduction confirmed the wrong-data-domain failure. It did not find bytes missed before hasher attachment; the hasher observed the complete transformed stream. The fix still installs the logical hasher before the compression goroutine starts, which removes both ordering ambiguity and the demonstrated domain error.

The review also reproduced two inherited adjacent gaps and kept them out of this issue's rollback boundary:

• Metadata-only CopyObject can stamp compression metadata without rewriting data #67 / fix: preserve transform state on metadata-only copies #69: metadata/reference-only copies and versioned SSE-C key rotation could corrupt transform state. The fix is merged.
• CopyObjectResult omits checksum fields and minio-go drops them #68 / fix: return checksums from CopyObject #70: CopyObjectResult omitted committed checksum fields. The server response fix is merged after the complete release-pipeline validation passed.
• CopyObject drops checksum fields from CopyObjectResult minio/minio-go#2294 / PR Rate limit is working and supports limited waiting clients minio/minio#2295: active minio-go discarded CopyObjectResult checksums; local and upstream Linux, Windows, RDMA, VulnCheck, and automated review pass, pending maintainer review.
• docs: record CopyObject checksum invariant silo.pgsty.com#2: bilingual design, verification, release-boundary, and historical-object remediation guidance is merged, deployed, and live on both language routes.

No silo-pkg change is needed because it does not own the server reader chain. SILO Console only passes CopyObject through minio-go. MCLI deliberately switches away from server-side CopyObject when --checksum is requested, so neither needs a duplicate patch.

The archived minio/minio upstream retains the original defect. The merged SILO fix changes no public server API and no on-disk checksum representation. Existing wrong checksum metadata is not automatically backfilled; operators must verify and explicitly rewrite affected copied objects.

[Vonng]

Documentation follow-up submitted

The bilingual design expansion is complete:

• Commit: pgsty/silo.pgsty.com@b87146a1
• Diff from parent: ef7f30b2...b87146a1
• PR: silo.pgsty.com#3
• Production CI/deploy: run 32733839992
• English: CopyObject Checksums Must Cover Logical Object Bytes
• 中文：CopyObject Checksum 必须覆盖逻辑对象字节

The PR build, Cloudflare preview, post-merge Hugo build, and GitHub Pages deployment passed. Both live routes return HTTP 200.

Online adversarial review prompt for commit b87146a1

你是一名对 S3 兼容性、Go 流式 I/O、MinIO/SILO 内部实现、checksum、压缩、加密、版本控制与 Hugo 双语技术文档都非常熟悉的资深维护者。

请对下面这个文档 commit 做一次只读、证据驱动、对抗性的 Code/Design Review。不要修改仓库，不要因为 PR、CI 或文档已经合并就默认它正确。你的任务是尽可能证伪文档中的技术结论、因果关系、正确性证明、测试声明、跨仓库边界与运维建议。

审查目标

• Commit: pgsty/silo.pgsty.com@b87146a
• Parent: pgsty/silo.pgsty.com@ef7f30b
• Exact diff: pgsty/silo.pgsty.com@ef7f30b...b87146a
• English source: https://github.com/pgsty/silo.pgsty.com/blob/b87146a1f86501a85ea3e35b1be4d38997b9bde8/content/blog/design/copyobject-checksum/index.md
• Chinese source: https://github.com/pgsty/silo.pgsty.com/blob/b87146a1f86501a85ea3e35b1be4d38997b9bde8/content/blog/design/copyobject-checksum/index.zh.md
• English live page: https://silo.pgsty.com/blog/design/copyobject-checksum/
• Chinese live page: https://silo.pgsty.com/zh/blog/design/copyobject-checksum/

必须核对的实现证据

• Original issue CopyObject server-side checksum can cover transformed data when compression is enabled #63: CopyObject server-side checksum can cover transformed data when compression is enabled #63
• Core checksum-domain fix PR fix: checksum CopyObject data before compression #66: fix: checksum CopyObject data before compression #66
• Merged checksum fix commit: c0e7159
• Metadata/reference transform-state fix PR fix: preserve transform state on metadata-only copies #69: fix: preserve transform state on metadata-only copies #69
• Merged transform-state commit: 05df6e7
• CopyObjectResult response fix PR fix: return checksums from CopyObject #70: fix: return checksums from CopyObject #70
• Merged response commit: f2520f3
• Multipart checksum predecessor UploadPart requires a per-part checksum header that AWS S3 computes server-side #46: UploadPart requires a per-part checksum header that AWS S3 computes server-side #46
• Still-independent federation issue Federated UploadPartCopy cannot reliably return the remote computed part checksum #64: Federated UploadPartCopy cannot reliably return the remote computed part checksum #64
• minio-go downstream/upstream response PR Rate limit is working and supports limited waiting clients minio/minio#2295: Return CopyObject checksums in UploadInfo minio/minio-go#2295
• Archived MinIO upstream: https://github.com/minio/minio
• AWS CopyObject specification: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
• AWS checksum/copy guidance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/copy-object.html

所有动态状态都必须现场复核。尤其不要假设 minio-go PR minio#2295 已经合并，也不要把源码 merge、文档 deploy 与 SILO release/package/image publication 混为一谈。

审查程序

1. 读取 commit 与 parent 的精确 diff，确认只有两份双语设计文档被修改。
2. 逐段把文档中的实现声明映射到 SILO 真实代码和测试，不得只阅读 PR 描述。
3. 追踪 CopyObjectHandler 中 logical reader、checksumReader、rawReader、storage Reader、newS2CompressReader、etag.Wrap、WithEncryption、putObject 和 FileInfo.Checksum 的完整数据流。
4. 阅读 internal/hash.Reader 的 Read/EOF 行为，以及 newS2CompressReader 中 io.Copy、S2 Close/CloseIndex 和 io.Pipe 关闭顺序，独立验证文档的 happens-before 与 checksum 完成性论证。
5. 验证未修复基线是否确实计算完整 transformed stream，而不是漏掉 hasher 安装前的前缀；检查文档对原 Issue 两种假设的修正是否准确。
6. 验证显式 checksum algorithm、来源 full-object checksum、来源 multipart composite checksum、无来源 checksum 默认 CRC64NVME 四条策略是否与真实实现和 AWS 语义一致。
7. 验证 ETag、存储 reader 记账、纠删码 bitrot、加密认证、S3 additional checksum 与 compression index 是否被正确区分，没有互相冒充。
8. 验证对象层对 missing / invalid / mismatched checksum 的硬失败、临时 shard 清理和 metadata 发布顺序；确认不会留下半对象，也不会用 200 静默丢失 checksum。
9. 检查 SSE-C、SSE-S3、加密来源、压缩来源、版本化 self-copy、metadata-only/reference-only、空对象、阈值、8 MiB index、HEAD/GET、race 等测试声明是否真实存在且断言了文档声称的语义。
10. 对文档列出的每个备选方案进行反向挑战：是否真的必须否决，是否遗漏了更简单且同样正确的方案，选中方案是否确实是最小充分实现。
11. 审查 Metadata-only CopyObject can stamp compression metadata without rewriting data #67 与 CopyObjectResult omits checksum fields and minio-go drops them #68 的拆分是否合理；确认文档没有把相邻修复的功劳错误归到 CopyObject server-side checksum can cover transformed data when compression is enabled #63，也没有遗漏它们带来的新边界。
12. 审查 mixed-version、历史错误 metadata 识别、下载后独立计算、copy-to-new-key 与 in-place REPLACE 建议。重点寻找会丢 metadata、tag、retention、legal hold、version、encryption 或 replication 语义的危险建议。
13. 核对 silo-pkg、MCLI、Console、minio-go 和 archived MinIO 的责任归属；不要接受没有代码证据的“无需修改”。
14. 对中英文逐节比对：章节顺序、锚点、表格、决策、否决理由、正确性证明、测试矩阵、运维警告与动态状态必须语义等价。指出任何翻译弱化、遗漏或额外承诺。
15. 检查 Hugo front matter、标题层级、anchor、表格、代码块、外链、长行可读性、移动端渲染和 live page 内容；区分 source 正确、preview 正确、deploy 成功和 production route 正确。

重点攻击面

• checksum 的 byte domain 是否被严格定义，还是只用了“plaintext”这一模糊词。
• logical source bytes 是否必然等于 logical destination bytes，包括 source encryption、source compression 和 range/metadata paths。
• hasher 安装顺序能否严格推出完整覆盖，io.Pipe 是否真的提供文档声称的可见性。
• ServerSideChecksumResult 是否只在 EOF 产生；出现非 EOF 错误、短读或 writer 提前失败时结论是否仍成立。
• ETag delegate 链是否真的覆盖逻辑 MD5，文档是否对加密 ETag 过度承诺。
• “来源 full-object checksum 可以直接保留”是否与 AWS 所说的重新计算存在语义冲突，是否需要更精确表述。
• object-layer hard failure是否可能掩盖更早、更具体的客户端错误。
• metadata-only 与 version-only 修复是否证明了文档中的通用不变量，还是只覆盖两个例子。
• test 数量很多是否掩盖了生产路径中的冗余 reader、隐式 hard limit 或未覆盖分支。
• “最小复杂度”是否仅按行数判断，还是同时考虑了回滚、接口、盘上格式、I/O、内存与认知复杂度。
• 历史对象修复建议是否足够保守，是否应明确要求先复制到新 key、校验、再切换。
• 最新 release 是否真的仍未包含这些 commit；必须检查当前 GitHub Releases，不得沿用文档时间点。

输出格式

严格使用以下结构：

1. Verdict：GO / GO WITH FIXES / NO-GO，并给出置信度。
2. Blocking findings：按严重度排序。每条必须包含具体文件、行或 heading anchor、GitHub permalink、反例或代码证据、为什么会误导维护者/运维人员、最小修订建议。
3. Non-blocking findings：精确性、可读性、双语漂移、链接、布局或未来维护风险。
4. Claims verified：列出已经通过代码、测试、CI 或 live route 独立确认的关键声明。
5. Claims unverified or time-sensitive：列出无法证明或可能随时间变化的声明。
6. Alternative analysis：判断被否决方案是否完整，是否存在文档遗漏的更小正确方案。
7. Bilingual parity：逐节说明中英文是否等价。
8. Scope and minimality：判断文档是否充分且必要，是否混入 Federated UploadPartCopy cannot reliably return the remote computed part checksum #64、release 或客户端范围外内容。
9. Required patch：如果不是 GO，给出尽可能小的逐段修改清单；不要重写整篇。
10. Final consensus statement：用一段话说明该设计文档能否作为 SILO 后续维护与发布的可信决策记录。

如果没有 finding，也必须说明你具体检查了哪些失败模式。不要只复述文档，不要把已有 CI 绿灯当成技术正确性的替代品。