Skip to content

Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA - #2074

Open
tyrielv wants to merge 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/fix-invalid-sha-hydration
Open

Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA#2074
tyrielv wants to merge 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/fix-invalid-sha-hydration

Conversation

@tyrielv

@tyrielv tyrielv commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

When a user process reads a virtualized placeholder file whose stored content-id (SHA) is corrupt — specifically 40 NUL bytes (\u0000 × 40) — GVFS builds a loose-object path from it and Path.Combine throws System.ArgumentException: Illegal characters in path:

System.ArgumentException: Illegal characters in path.
   at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)
   at System.IO.Path.Combine(String path1, String path2, String path3)
   at GVFS.Common.Git.GitRepo.GetLooseBlobState(String blobSha, Action`2 writeAction, Int64& size)
   at GVFS.Common.Git.GitRepo.TryCopyBlobContentStream(String blobSha, Action`2 writeAction)
   at GVFS.Common.Git.GVFSGitObjects...TryCopyBlobContentStream...
   at GVFS.Platform.Windows.WindowsFileSystemVirtualizer.GetFileStreamHandlerAsyncHandler(...)

ArgumentException is not in RetryWrapper.IsHandlableException (only HttpRequestException, IOException, RetryableException), so it bypasses both the retry logic and the download fallback in GVFSGitObjects.TryCopyBlobContentStream and propagates to the virtualizer's outer catch (Exception), which returns FileNotAvailable to ProjFS. The placeholder can never hydrate, so the failing read repeats indefinitely — a retry storm.

Not AllZeroSha

This is not GVFSConstants.AllZeroSha. AllZeroSha is 40 ASCII '0' characters, which yields directory "00" and does not throw. The failing SHA is literally 40 NUL bytes — a corrupt placeholder content-id.

Where the malformed SHA comes from

A placeholder's content-id is the 40-char SHA encoded as UTF-16 (80 bytes) and stored in the ProjFS placeholder reparse point (FileSystemVirtualizer.ConvertShaToContentId = Encoding.Unicode.GetBytes(sha)). On read, GetShaFromContentId decodes those 80 bytes with Encoding.Unicode.GetString. If that on-disk content-id region is zeroed — a corrupt placeholder from an interrupted/partial placeholder write, a crash/power-loss during placeholder creation, or disk corruption — the decode yields 40 \u0000 characters.

Telemetry impact

Signature: VFS.Error, ErrorMessage contains TryCopyBlobContentStream, Exception contains Illegal characters in path and GetLooseBlobState.

  • Top blob-hydration failure cause on the LKG field build 1.0.26014.1 (38 machines).
  • ~61 machines / ~6.2K events across 30 days; the corruption is old and version-agnostic (observed across ≥4 GVFS builds: 1.0.25164, 25314, 26014, 26098), and the same 1–4 files per machine fail repeatedly for days — the placeholder never self-heals.
  • Produces permanent failures and retry storms — the worst single machine emitted ~2.49M error events in 30 days.

Fix

A minimal, surgical behavior fix (graceful-fail, defense-in-depth). Reject a malformed SHA before it is turned into a filesystem path:

  • GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean, non-retryable miss) when the SHA is not 40 hex characters, so Path.Combine can never throw here again.
  • GitRepo.LooseObjectExists guards the same Path.Combine.
  • GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA before the retry loop, so a bogus SHA never triggers a doomed 404 download or a retry storm.
  • SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString renders the bad value with non-hex characters escaped (\uXXXX) so telemetry stays greppable and free of control characters.
  • WindowsFileSystemVirtualizer.GetFileDataCallback routes the request's logged sha through ToLoggableShaString, so the terminal hydration-failure RelatedError (and every sink using that request metadata) can no longer record raw NUL/control bytes. No-op for a valid hex SHA.

All three guard sites emit the same greppable Warning event (*_MalformedBlobSha, with the offending value under the sha metadata key) at Warning level, with no unhandled exception (an expected, handled corrupt-placeholder read is a Warning, not an Error).

Telemetry category stays "Unexpected"

Per an existing decision, a malformed SHA gets no dedicated telemetry category. This branch was developed on top of the (now-merged) blob-hydration telemetry work (#2071), which added out BlobHydrationFailureCategory to TryCopyBlobContentStream; the malformed-SHA short-circuit sets failureCategory = Unexpected. The case previously reached the virtualizer's outer catch and was tagged Unexpected there; now that it no longer throws, it returns cleanly and the virtualizer tags its terminal telemetry Unexpected via the out-param — same bucket, no double-logging, no new category. The paired *_MalformedBlobSha warning lets an analyst separate this deterministic case from genuinely-unexpected exceptions.

Relationship to #2071 (merged)

Developed stacked on #2071 ("Split blob-hydration and directory-enumeration failure telemetry by cause"), which has since merged to master. This branch has been rebased onto master, so the diff below is now just this fix (the #2071 changes are in the base). No further coordination needed.

Not in scope (follow-up)

  • No placeholder repair. This PR fails gracefully. A read-time self-heal that re-derives the authoritative SHA from the git-index projection and rewrites the corrupt placeholder — with a detected → repaired / repair-failed telemetry funnel — is a separate follow-up PR targeting vnext (2.1): it is new, destabilizing behavior for an old, non-2.0-regressing problem, so it does not belong in the 2.0 stabilization line. This PR's graceful-fail guards remain the fallback for the repair-miss case.

Tests

  • GVFSGitObjectsTests.TryCopyBlobContentStreamFailsCleanlyForCorruptAllNullByteSha / ...ForOtherMalformedShas: a 40-NUL-byte SHA, a 40-char SHA with an embedded path-illegal character, and other malformed SHAs return false from both GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream with no ArgumentException (Assert.DoesNotThrow), no download/retry is attempted, the out category is Unexpected, and the *_MalformedBlobSha diagnostic events are actually emitted (verified via a small MockTracer enhancement that records RelatedEvent names).
  • SHA1UtilTests: null-safety and ToLoggableShaString escaping (mixed valid/NUL, control chars, high code points).
  • Full unit suite: 897 passed, 0 failed (11 pre-existing ignores).

Review

Reviewed with a multi-lens self-review (correctness / security / design / tests / async / risk-rollout) over two iterations; feedback applied (consistent *_MalformedBlobSha event names and sha metadata key across all guards; escaped telemetry at the single virtualizer chokepoint; explicit no-throw / no-download / emission assertions). No outstanding code findings; the only rollout note is to re-point any dashboard/alert keyed on the old Illegal characters in path exception signature at the new *_MalformedBlobSha events / the Unexpected category tag (the top-level hydration-failure RelatedError still fires, so affected machines remain visible).

@tyrielv
tyrielv force-pushed the tyrielv/fix-invalid-sha-hydration branch 3 times, most recently from 902a99b to 882b7d8 Compare August 7, 2026 21:57
When a user process reads a virtualized placeholder whose stored
content-id is corrupt - specifically 40 NUL bytes instead of a hex SHA -
GVFS builds a loose-object path from it and Path.Combine throws
System.ArgumentException ("Illegal characters in path").

ArgumentException is not in RetryWrapper.IsHandlableException, so it
bypasses both the retry logic and the download fallback in
GVFSGitObjects.TryCopyBlobContentStream and propagates to the
virtualizer's outer catch, which returns FileNotAvailable to ProjFS. The
placeholder can never hydrate, so the failing read repeats forever - a
retry storm. This is the #1 blob-hydration failure cause on the LKG field
build 1.0.26014.1 (38 machines; ~61 machines / ~6.2K events across 30d;
one machine emitted ~2.49M error events).

This is a corrupt content-id, NOT GVFSConstants.AllZeroSha: AllZeroSha is
40 ASCII '0' characters, which yields directory "00" and does not throw.

Reject a malformed SHA before it is turned into a path:

- GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean,
  non-retryable miss) for a SHA that is not 40 hex characters, so
  Path.Combine can never throw here again.
- GitRepo.LooseObjectExists guards the same Path.Combine.
- GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA
  before the retry loop, so a bogus SHA never triggers a doomed 404
  download or a retry storm.
- SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString
  renders the bad value with non-hex characters escaped so telemetry stays
  greppable and free of control characters.
- WindowsFileSystemVirtualizer routes the request's logged sha through
  ToLoggableShaString, so a malformed content-id can no longer enter
  telemetry with raw NUL/control bytes at the terminal hydration-failure
  error either (a no-op for a valid hex SHA).

All three guard sites emit the same greppable Warning event
(*_MalformedBlobSha) at Warning level with no unhandled exception. Per an
existing decision this case stays telemetry category "Unexpected"; no new
BlobHydrationFailureCategory is added.

Stacked on microsoft#2071 (tyrielv/split-hydration-enum-telemetry): this branch is
rebased onto it, so microsoft#2071's out BlobHydrationFailureCategory parameter is
honored - the malformed-SHA short-circuit sets failureCategory =
Unexpected, so the virtualizer's terminal telemetry tags the case exactly
as before (it no longer reaches the outer catch because it no longer
throws). This PR must NOT merge before microsoft#2071; after microsoft#2071 lands, rebase
onto master.

Unit tests assert that a 40-NUL-byte SHA, a 40-char SHA with an embedded
path-illegal character, and other malformed SHAs return false from both
GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream
with no ArgumentException (Assert.DoesNotThrow), that no download/retry is
attempted, and that the out category is Unexpected.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv force-pushed the tyrielv/fix-invalid-sha-hydration branch from 882b7d8 to 2d88803 Compare August 7, 2026 22:04
@tyrielv
tyrielv marked this pull request as ready for review August 7, 2026 22:05
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 10, 2026
When a user process reads a virtualized placeholder whose stored content-id
is corrupt - 40 NUL bytes instead of a hex blob SHA - GVFS cannot hydrate it
from the corrupt content-id. microsoft#2074 makes that read fail cleanly (no crash, no
retry storm). This change goes one step further and repairs the underlying
data so the file works again.

Telemetry shows this corruption is durable and localized: the same 1-4 files
per machine fail repeatedly over multiple days until something rewrites the
placeholder (~61 machines / ~6.2K events over 30 days). It is old and
version-agnostic (spans >=4 GVFS builds), not a 2.0 regression. The triggering
processes are readers (git.exe, copilot.exe, Code.exe); the placeholder was
already corrupt on disk. The authoritative path->SHA still exists, because the
path is still projected and the git index projection can return the correct
SHA for it.

Read-time self-heal (WindowsFileSystemVirtualizer.GetFileStreamHandlerAsyncHandler):

- Plumb virtualPath into the handler and, when the placeholder's decoded SHA is
  not a valid hex SHA, recover the authoritative SHA for virtualPath from
  GitIndexProjection.GetProjectedFileInfo and hydrate the blob from that instead
  of the corrupt content-id.
- The recovery passes a null BlobSizesConnection: repair needs only the SHA, not
  the blob size, so size resolution (which can throw SizesUnavailableException)
  is skipped and a size-lookup fault cannot deny a SHA-only self-heal.
- A successful hydration writes the whole file, which converts the placeholder
  into a full file on disk. The corrupt content-id is superseded and future
  reads never call back, so the file is repaired for good.
- If the path is no longer projected (deleted/renamed), the projection lookup
  throws, or the recovered SHA cannot be hydrated, fall back to the same clean,
  non-crashing FileNotAvailable failure as microsoft#2074.

We deliberately do NOT rewrite the placeholder's content-id in place via
UpdateFileIfNeeded. Confirmed empirically against real inbox ProjFS with a
throwaway probe: (1) serving the full content converts the placeholder to a
full file, so a second read issues no GetFileData callback (hydration alone is
the repair); and (2) UpdateFileIfNeeded on the file mid-read returns
0x80070020 (ERROR_SHARING_VIOLATION) because the reader holds the file open.
The same probe showed a corrupt placeholder is only injectable from the owning
virtualization instance (WritePlaceholderInfo accepts an all-NUL content-id)
and that an external FSCTL_SET_REPARSE_POINT rewrite is blocked (ERROR 1359),
so this behavior is covered by unit tests rather than a functional test.

Telemetry funnel (paired with microsoft#2074's *_MalformedBlobSha detection):

- Repaired: *_MalformedBlobShaRepaired (Warning) with the recovered SHA, so we
  can watch the corrupt-placeholder population drain.
- Repair miss: *_MalformedBlobShaRepairFailed (Warning) tagged with a
  MalformedShaRepairFailureReason (ProjectionMiss / ProjectionException /
  HydrateFailed / HydrateException). The failed event is emitted on every
  repair-failure exit - including hydration failures that throw after a SHA is
  recovered (size mismatch, local IO, ProjFS write failure) - so that
  repaired + repair-failed accounts for every repair attempt.

Coordinates with microsoft#2071: a repair miss stays telemetry category Unexpected; a
successful repair simply succeeds. No new BlobHydrationFailureCategory value.

Two known, accepted behaviors are documented in code: the projection is read
live, so a concurrent checkout can change the projected SHA between placeholder
open and repair (serving the currently-projected SHA is the best answer for an
already-corrupt file and matches the placeholder-creation path); and an
unrepairable-but-projected file whose blob is unavailable pays the normal
download + retry budget per read (the same cost any valid-but-unavailable
placeholder pays), bounded and never re-crashing.

Stacked on microsoft#2074 (tyrielv/fix-invalid-sha-hydration), which is stacked on
microsoft#2071. Targets vnext: this is a new behavioral change on the read path for an
old, rare, pre-existing corruption, so it does not belong on the 2.0
stabilization line. microsoft#2074 already removes the crash and retry storm on master.

Unit tests (WindowsFileSystemVirtualizerTests) cover: repair success (asserting
hydration uses the RECOVERED SHA, not the corrupt one) emits
*_MalformedBlobShaRepaired and completes Ok; a non-projected path, a throwing
projection lookup, an unhydratable recovered SHA, and a hydration that throws
after recovery each emit *_MalformedBlobShaRepairFailed with the expected
reason and fail cleanly; mid-repair cancellation emits neither repair event;
and a valid content-id still hydrates with no repair telemetry.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant