Skip to content

Repair a corrupt (NUL-byte) placeholder SHA at read time - #2081

Draft
tyrielv wants to merge 16 commits into
microsoft:vnextfrom
tyrielv:tyrielv/repair-malformed-sha-placeholder
Draft

Repair a corrupt (NUL-byte) placeholder SHA at read time#2081
tyrielv wants to merge 16 commits into
microsoft:vnextfrom
tyrielv:tyrielv/repair-malformed-sha-placeholder

Conversation

@tyrielv

@tyrielv tyrielv commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Warning

Do NOT publish / mark ready until BOTH: (1) #2074 is completed (approved + finalized), AND (2) #2074 has been merged into vnext. This PR is stacked on #2074 (which is stacked on #2071) and targets vnext. Until #2074 lands in vnext, the diff below includes #2071 + #2074 as well as this repair. Once #2074 is in vnext, rebase this branch so the diff reduces to just the repair. Keep as DRAFT until then.

What this does

Repairs a corrupt (NUL-byte) placeholder content-id at read time instead of only failing cleanly.

When a user process reads a virtualized placeholder whose stored content-id is corrupt — 40 NUL bytes instead of a hex blob SHA — GVFS cannot hydrate it from that content-id. #2074 makes the read fail cleanly (no crash, no retry storm). This PR goes one step further and repairs the underlying data so the file works again.

Root cause (NUL-byte, not AllZeroSha)

A placeholder stores its content-id (the 40-char blob SHA) as UTF‑16 in the ProjFS reparse point. A corrupt placeholder decodes back to a content-id of 40 NUL characters (\u0000 ×40) — literal on-disk corruption. This is not GVFSConstants.AllZeroSha (40 ASCII '0', which is valid hex and yields directory "00").

Telemetry evidence

  • Durable, localized corruption: the same 1–4 files per machine fail repeatedly over multiple days until something rewrites the placeholder (~61 machines / ~6.2K events over 30 days; ~643 events / 16 machines over 7 days). This recurring per-file failure is the real user harm — and it is exactly what repair fixes.
  • Not a 2.0 regression: spans ≥4 GVFS builds (1.0.25164, 25314, 26014, 26098) — old, version-agnostic, rare.
  • Triggering processes are readers (git.exe, copilot.exe, Code.exe) — they don't cause the corruption; the placeholder was already corrupt.
  • The authoritative path→SHA still exists: the path is still projected, and GitIndexProjection.GetProjectedFileInfo returns the correct SHA for it, so the corrupt content-id is recoverable from the path.

Design (read-time self-heal)

In WindowsFileSystemVirtualizer.GetFileStreamHandlerAsyncHandler:

  1. Plumb virtualPath and the request's BlobSizesConnection into the handler.
  2. When the decoded SHA is not a valid hex SHA, recover the authoritative SHA for virtualPath from the git index projection and hydrate from that instead of the corrupt content-id.
  3. A successful hydration writes the whole file, which converts the placeholder into a full file on disk — the corrupt content-id is superseded and future reads never call back, so the file is repaired for good.
  4. If the path is no longer projected (deleted/renamed), the projected SHA is itself invalid, or the recovered SHA cannot be hydrated → fall back to the same clean, non-crashing FileNotAvailable as Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA #2074.

Why not rewrite the content-id in place?

We deliberately do not call UpdateFileIfNeeded to rewrite the content-id. Confirmed empirically against real inbox ProjFS with a throwaway probe:

  • Serving the full content converts the placeholder to a full file, so a second read issues no GetFileData callback — hydration alone is the repair.
  • UpdateFileIfNeeded on the file mid-read returns 0x80070020 (ERROR_SHARING_VIOLATION) because the reader holds the file open.
  • A corrupt placeholder is only injectable from the owning virtualization instance (WritePlaceholderInfo accepts an all-NUL content-id); an external FSCTL_SET_REPARSE_POINT rewrite is blocked (ERROR 1359).

That last point is why this behavior is covered by unit tests rather than a functional test — the corruption cannot be injected into a real mount from outside the provider.

Telemetry funnel

Paired with #2074's *_MalformedBlobSha detection:

  • Repaired: GetFileStreamHandlerAsyncHandler_MalformedBlobShaRepaired (Warning) with the recovered SHA — lets us watch the corrupt-placeholder population drain.
  • Repair miss: GetFileStreamHandlerAsyncHandler_MalformedBlobShaRepairFailed (Warning) with a reason (ProjectionMiss / InvalidProjectedSha / ProjectionException / HydrateFailed), then a clean fail.

Relationship to the stack

Why vnext, not master (2.0)

Per the branch model: master is the shippable/stabilization line; vnext is next-train behavioral work. This repair is a new behavioral change on the read path (projection lookup, a possible cache-server download, placeholder repair, edge cases like deleted/renamed paths) for an old, rare, pre-existing corruption. Adding it to the 2.0 stabilization line would spend stabilization risk on a rare problem that is not worse in 2.0. #2074 (the crash / graceful-fail fix) already removes the crash and retry storm and correctly targets master.

Tests

WindowsFileSystemVirtualizerTests (unit):

  • Repair success from the projection → emits *_MalformedBlobShaRepaired, completes Ok.
  • Non-projected path → emits *_MalformedBlobShaRepairFailed, fails cleanly.
  • Recovered SHA cannot hydrate → emits *_MalformedBlobShaRepairFailed, fails cleanly.
  • Valid content-id → hydrates normally with no repair telemetry.

Full unit suite: 906 passed, 0 failed (11 pre-existing ignored).

tyrielv and others added 16 commits July 20, 2026 07:57
Add `vnext` to build.yaml's pull_request/push branch filters so the
feature-integration branch gets the same CI (build + unit + functional
tests) as master. This lets PRs targeting vnext produce the required
status checks.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
…ause

The GVFS telemetry "Blob hydration failure" and "Directory enumeration
failure" buckets conflate causes outside gvfs.exe's control (network,
local disk/IO, ProjFS) with actionable ones (a missing object on the
server, a size mismatch, or GVFS's own stale-enumeration eviction).

Stamp a cause tag on the failure telemetry so the release-readiness
dashboard can bucket them apart. No behavior changes:

- BlobHydrationFailureCategory (nested in GVFSGitObjects) is now returned
  from TryCopyBlobContentStream via an out parameter as well as stamped on
  the terminal telemetry, so the virtualizer's own terminal event is tagged
  with the same cause rather than left uncategorized. Categories:
  NetworkUnavailable / DownloadFailed / LocalIO / ProjFSWriteFailed (not
  gvfs-fixable) vs ObjectNotOnServer / LocalCopyFailed / SizeMismatch /
  Unexpected (actionable). The size-mismatch, IOException, and WriteFileData
  failure sites were previously logged with a message the dashboard did not
  match; they now carry the tag so they are counted.

- EnumerationFailureReason (nested enum) on "Failed to find active
  enumeration ID": Evicted (GVFS eviction removed a live enumeration;
  self-inflicted) vs Unknown (ProjFS delivered an id GVFS never held or
  already ended). The eviction-tracking map is populated before the entry
  is removed from the active set (closing a mislabel race) and pruned on
  every sweep so it cannot outlive its window.

Unit tests assert each cause value deterministically.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5 to 6.
- [Release notes](https://github.com/actions/setup-dotnet/releases)
- [Commits](actions/setup-dotnet@v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-dotnet
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
…tions/actions/setup-dotnet-6

Bump actions/setup-dotnet from 5 to 6
…e-v2.55.0.vfs.0.3

Update default Microsoft Git version to v2.55.0.vfs.0.3
The health verb always labeled its final line 'Repository status: ...' even when the calculation was scoped to a subdirectory (via -d or when run from a subdirectory of the enlistment). That could report a highly hydrated subtree as 'Highly Hydrated' at the repository level, which is misleading.

When TargetDirectory is non-empty, print 'Directory status (<path>): ...' instead and add a hint suggesting the user re-run from the repo root for the full-repo status. Update the functional test regex to accept either label.
…n-enum-telemetry

Split blob-hydration and directory-enumeration failure telemetry by cause
RetryableException wraps its real cause in InnerException. The blob-
hydration failure categorization checked the RetryableException type
itself, so every RetryableException - the largest hydration failure
bucket in the field - was tagged NetworkUnavailable, even when the real
cause was local disk/IO.

On this branch the RetryableException reaches OnFailure from
Context.Repository.TryCopyBlobContentStream - typically StreamUtil
wrapping an IOException while reading a corrupt or truncated local loose
object. Unwrap RetryableException.InnerException before categorizing, and
map IOException / UnauthorizedAccessException / Win32Exception (the local
disk/IO family) to LocalIO. A RetryableException whose inner cause is not
local (e.g. HttpRequestException), or that has no inner cause, stays
NetworkUnavailable. Telemetry metadata only; no behavior change.

Add unit tests for each inner-cause mapping (IOException, Unauthorized-
AccessException, Win32Exception -> LocalIO; HttpRequestException and no
inner -> NetworkUnavailable), and reset the process-global
RetryCircuitBreaker in the fixture SetUp and TearDown so these failure-
driving tests cannot open the circuit for one another or for a later
fixture.

Stacked follow-up to PR microsoft#2071; do not publish until microsoft#2071 merges.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…yable-unwrap

Attribute wrapped blob-hydration failures to their inner cause
…e-v2.55.0.vfs.0.6

Update the default Microsoft Git version used by VFS for Git
to the newly promoted [`v2.55.0.vfs.0.6`](https://github.com/microsoft/git/releases/tag/v2.55.0.vfs.0.6) release.
…rectory-clarification

gvfs health: distinguish directory-scoped status from repository status
When a user process reads a virtualized placeholder whose stored
content-id is corrupt - specifically 40 NUL bytes instead of a hex SHA -
GVFS builds a loose-object path from it and Path.Combine throws
System.ArgumentException ("Illegal characters in path").

ArgumentException is not in RetryWrapper.IsHandlableException, so it
bypasses both the retry logic and the download fallback in
GVFSGitObjects.TryCopyBlobContentStream and propagates to the
virtualizer's outer catch, which returns FileNotAvailable to ProjFS. The
placeholder can never hydrate, so the failing read repeats forever - a
retry storm. This is the #1 blob-hydration failure cause on the LKG field
build 1.0.26014.1 (38 machines; ~61 machines / ~6.2K events across 30d;
one machine emitted ~2.49M error events).

This is a corrupt content-id, NOT GVFSConstants.AllZeroSha: AllZeroSha is
40 ASCII '0' characters, which yields directory "00" and does not throw.

Reject a malformed SHA before it is turned into a path:

- GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean,
  non-retryable miss) for a SHA that is not 40 hex characters, so
  Path.Combine can never throw here again.
- GitRepo.LooseObjectExists guards the same Path.Combine.
- GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA
  before the retry loop, so a bogus SHA never triggers a doomed 404
  download or a retry storm.
- SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString
  renders the bad value with non-hex characters escaped so telemetry stays
  greppable and free of control characters.
- WindowsFileSystemVirtualizer routes the request's logged sha through
  ToLoggableShaString, so a malformed content-id can no longer enter
  telemetry with raw NUL/control bytes at the terminal hydration-failure
  error either (a no-op for a valid hex SHA).

All three guard sites emit the same greppable Warning event
(*_MalformedBlobSha) at Warning level with no unhandled exception. Per an
existing decision this case stays telemetry category "Unexpected"; no new
BlobHydrationFailureCategory is added.

Stacked on microsoft#2071 (tyrielv/split-hydration-enum-telemetry): this branch is
rebased onto it, so microsoft#2071's out BlobHydrationFailureCategory parameter is
honored - the malformed-SHA short-circuit sets failureCategory =
Unexpected, so the virtualizer's terminal telemetry tags the case exactly
as before (it no longer reaches the outer catch because it no longer
throws). This PR must NOT merge before microsoft#2071; after microsoft#2071 lands, rebase
onto master.

Unit tests assert that a 40-NUL-byte SHA, a 40-char SHA with an embedded
path-illegal character, and other malformed SHAs return false from both
GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream
with no ArgumentException (Assert.DoesNotThrow), that no download/retry is
attempted, and that the out category is Unexpected.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
When a user process reads a virtualized placeholder whose stored content-id
is corrupt - 40 NUL bytes instead of a hex blob SHA - GVFS cannot hydrate it
from the corrupt content-id. microsoft#2074 makes that read fail cleanly (no crash, no
retry storm). This change goes one step further and repairs the underlying
data so the file works again.

Telemetry shows this corruption is durable and localized: the same 1-4 files
per machine fail repeatedly over multiple days until something rewrites the
placeholder (~61 machines / ~6.2K events over 30 days). It is old and
version-agnostic (spans >=4 GVFS builds), not a 2.0 regression. The triggering
processes are readers (git.exe, copilot.exe, Code.exe); the placeholder was
already corrupt on disk. The authoritative path->SHA still exists, because the
path is still projected and the git index projection can return the correct
SHA for it.

Read-time self-heal (WindowsFileSystemVirtualizer.GetFileStreamHandlerAsyncHandler):

- Plumb virtualPath into the handler and, when the placeholder's decoded SHA is
  not a valid hex SHA, recover the authoritative SHA for virtualPath from
  GitIndexProjection.GetProjectedFileInfo and hydrate the blob from that instead
  of the corrupt content-id.
- The recovery passes a null BlobSizesConnection: repair needs only the SHA, not
  the blob size, so size resolution (which can throw SizesUnavailableException)
  is skipped and a size-lookup fault cannot deny a SHA-only self-heal.
- A successful hydration writes the whole file, which converts the placeholder
  into a full file on disk. The corrupt content-id is superseded and future
  reads never call back, so the file is repaired for good.
- If the path is no longer projected (deleted/renamed), the projection lookup
  throws, or the recovered SHA cannot be hydrated, fall back to the same clean,
  non-crashing FileNotAvailable failure as microsoft#2074.

We deliberately do NOT rewrite the placeholder's content-id in place via
UpdateFileIfNeeded. Confirmed empirically against real inbox ProjFS with a
throwaway probe: (1) serving the full content converts the placeholder to a
full file, so a second read issues no GetFileData callback (hydration alone is
the repair); and (2) UpdateFileIfNeeded on the file mid-read returns
0x80070020 (ERROR_SHARING_VIOLATION) because the reader holds the file open.
The same probe showed a corrupt placeholder is only injectable from the owning
virtualization instance (WritePlaceholderInfo accepts an all-NUL content-id)
and that an external FSCTL_SET_REPARSE_POINT rewrite is blocked (ERROR 1359),
so this behavior is covered by unit tests rather than a functional test.

Telemetry funnel (paired with microsoft#2074's *_MalformedBlobSha detection):

- Repaired: *_MalformedBlobShaRepaired (Warning) with the recovered SHA, so we
  can watch the corrupt-placeholder population drain.
- Repair miss: *_MalformedBlobShaRepairFailed (Warning) tagged with a
  MalformedShaRepairFailureReason (ProjectionMiss / ProjectionException /
  HydrateFailed / HydrateException). The failed event is emitted on every
  repair-failure exit - including hydration failures that throw after a SHA is
  recovered (size mismatch, local IO, ProjFS write failure) - so that
  repaired + repair-failed accounts for every repair attempt.

Coordinates with microsoft#2071: a repair miss stays telemetry category Unexpected; a
successful repair simply succeeds. No new BlobHydrationFailureCategory value.

Two known, accepted behaviors are documented in code: the projection is read
live, so a concurrent checkout can change the projected SHA between placeholder
open and repair (serving the currently-projected SHA is the best answer for an
already-corrupt file and matches the placeholder-creation path); and an
unrepairable-but-projected file whose blob is unavailable pays the normal
download + retry budget per read (the same cost any valid-but-unavailable
placeholder pays), bounded and never re-crashing.

Stacked on microsoft#2074 (tyrielv/fix-invalid-sha-hydration), which is stacked on
microsoft#2071. Targets vnext: this is a new behavioral change on the read path for an
old, rare, pre-existing corruption, so it does not belong on the 2.0
stabilization line. microsoft#2074 already removes the crash and retry storm on master.

Unit tests (WindowsFileSystemVirtualizerTests) cover: repair success (asserting
hydration uses the RECOVERED SHA, not the corrupt one) emits
*_MalformedBlobShaRepaired and completes Ok; a non-projected path, a throwing
projection lookup, an unhydratable recovered SHA, and a hydration that throws
after recovery each emit *_MalformedBlobShaRepairFailed with the expected
reason and fail cleanly; mid-repair cancellation emits neither repair event;
and a valid content-id still hydrates with no repair telemetry.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv force-pushed the tyrielv/repair-malformed-sha-placeholder branch from 84a5825 to 49b77d1 Compare August 10, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants