fix(cli): detect equal-size binary changes during publish

[atomicdjt]

Fixes #350.

Binary files currently contribute only their byte count to content_hash, so replacing an asset with different bytes of the same length silently skips publication. Use a SHA-256 digest of the raw binary bytes as input to the existing content-and-metadata hash. Publish filtering is kept separate from text classification, and a text-suffix file falls back to binary state when strict UTF-8 decoding fails. Valid UTF-8 text hashing and document identity are unchanged.

Regression coverage exercises equal-size replacements, stable/empty binary content, malformed text-suffix content, metadata changes, and valid text-hash compatibility. Publisher tests exercise both publish_file and publish_folder with only the remote RAGFlow client mocked: changed bytes upload, subsequent unchanged bytes skip, existing length-based binary hashes migrate, and a configured .pdf filter does not redefine that file as text. The architecture documentation and its website counterpart describe the one-time migration.

Validation on Windows, Python 3.12.14:

• Baseline CLI suite: 105 passed.
• Before the fix: six new regression cases failed (equal-size hashes, file/folder skipping, and legacy hash transition).
• Final CLI suite: python -m pytest src/rosetta-cli/tests -q — 118 passed.
• python -m mypy --config-file mypy.ini src/rosetta-cli — passed, 22 source files.
• python -m build in src/rosetta-cli — wheel and sdist built.
• python -m twine check src/rosetta-cli/dist/* — both passed.
• git diff --check — passed.

No live RAGFlow upload or parsing was performed. Existing binary documents and malformed text-suffix files with lossy stored hashes will republish once because their stored hashes used the old format. AI assistance: Codex assisted implementation, regression tests, and diff review.

[github-actions]

Rosetta Triage Review

Summary: This PR replaces the binary length surrogate with a SHA-256 digest of raw bytes before the existing metadata hash, adds regression coverage for file and folder publishing, and documents the one-time binary republish.

Findings:

• P2: ContentPublisher.file_extensions is both the publish filter and the DocumentData text-classification input. A configured binary suffix (for example, .pdf) is therefore decoded with errors="ignore" and hashed as lossy text instead of raw bytes; different invalid UTF-8 byte sequences can still produce the same hash and be skipped. Keep publish filtering separate from text classification, and use raw-byte hashing when UTF-8 decoding fails.

Caveats:

• The change intentionally republishes each existing default-classified binary once because its stored length-based hash differs. The new tests exercise this migration.

Suggestions:

• Add a regression case with ContentPublisher(file_extensions=[".pdf"]) and equal-size invalid UTF-8 replacements. Also cover an invalid UTF-8 file with a default text suffix.

Validation:

• The source and website architecture pages are synchronized.
• The new tests cover .bin/.pdf equal-size replacements, stable bytes, metadata-sensitive hashes, file/folder publishing, and legacy-hash migration.
• GitHub build-test, CodeQL, and analysis checks are passing at triage time.

Automated triage by Rosetta agent

[atomicdjt]

Addressed in ec349035.

• ContentPublisher.file_extensions now controls folder selection only; it is no longer passed into DocumentData as text classification.
• Text-suffix files use strict UTF-8 decoding. A decode failure produces consistent binary state (is_text=False, no text, line-count, or frontmatter metadata) and hashes the raw-byte digest.
• Regressions cover a configured .pdf filter with equal-size invalid UTF-8 replacements, the one-time lossy-hash migration, subsequent unchanged skipping, filter exclusion, and invalid UTF-8 under a default .md suffix. Existing binary migration and valid UTF-8 text-hash coverage remain in place.

Local validation: 118 CLI tests, mypy, package build, twine checks, and git diff --check passed. The updated upstream CLI workflow and all CodeQL analyses also passed.

[atomicdjt]

Hi @isolomatov-gd — following up now that the P2 from automated triage has been addressed in ec34903.
The fix separates publish filtering from text classification, uses strict UTF-8 decoding with binary fallback, and adds the requested regressions for configured .pdf filters and invalid UTF-8 under text suffixes.
Current validation is green: 118 CLI tests, mypy, package build/twine checks, git diff --check, the updated upstream CLI workflow, and CodeQL.
When you have a chance, could you take another look? Happy to address any remaining concerns. Thanks.