Scale, observability & fast file-backed provisioning (stack A–I)#67
Scale, observability & fast file-backed provisioning (stack A–I)#67dfsweden wants to merge 28 commits into
Conversation
acquireVolumeLock/acquireSnapshotLock now return codes.Aborted instead of killing the whole controller process on a lock-acquire timeout (which under concurrent load cascaded into a crash loop). Cherry-picked ace0cc6 to skip the per-file objective + file-visibility poll so CreateVolume returns after local mkfs (~0.5s).
deleteFileBacked, CleanupLoopDevice The 2000-pod scale test surfaced pods stuck in Terminating because DeleteVolume was silently looping. The controller-side spans were present (DeleteVolume, DeleteSnapshot, UnmountBackingShareIfUnused) but the node-side destroy RPCs, the file-backed unpublish helper, the Anvil-side backing-file delete, and the loop-device teardown all had no spans. That means from a trace we could see "DeleteVolume was called" but not why any given call took 30 seconds or hung — the actual work happens in these five functions. Add tracer.Start on each: * Node/NodeUnstageVolume (node.go) * Node/NodePublishVolume (already had span; kept) * Node/NodeUnpublishVolume (node.go) NEW * unpublishFileBackedVolume (node_helper.go) NEW * deleteFileBackedVolume (controller.go) NEW * CleanupLoopDevice (utils.go) NEW CleanupLoopDevice previously took no context. Added ctx as first parameter; three call sites in node_helper.go updated to pass ctx (they all had it in scope already). No behavior change beyond the spans; local `make compile` clean.
Adds pkg/common/metrics.go (MeasureOp helper emitting hs_csi_operation_duration_seconds, hs_csi_operation_errors_total, hs_csi_operation_inflight keyed by operation) and the 13 call sites mirroring the tracing spans, plus env-driven trace+metric exporter wiring in main.go (OTEL_TRACES_EXPORTER / OTEL_METRICS_EXPORTER / OTEL_METRICS_PROMETHEUS_LISTEN). Matches the metric names the Grafana hs-csi-driver dashboard queries. On top of the xfs-freeze2 hardening + DeleteSnapshot/DeleteVolume fixes.
MeasureOp now takes *error; CreateVolume/DeleteVolume/CreateSnapshot and NodeStage/Unstage/Publish/Unpublish get a named err return + (&err) so hs_csi_operation_errors_total increments on real RPC failures. Sub-step call sites keep (nil) (duration+inflight only).
…tion_duration_seconds
Add hs_csi_anvil_requests_total, recorded after each doRequest completes so
it carries the real HTTP status code (0 on transport error). The existing
doRequest latency histogram is armed in a deferred closure *before* the
response is known, so it can only label method+path; this counter captures
every verb (GET/POST/PUT/DELETE) plus the 404 type-probes that dominate
file-backed traffic.
Also collapse per-resource IDs in the path to a low-cardinality route
template (AnvilRoute: shares/files/tasks/file-snapshots ids -> {id}) and use
it for the histogram's http.path label too, which previously exploded into
one series per share (file--pvc-<uuid>).
MeasureOp/span coverage was file-backed-biased: MakeEmptyRawFile, FormatDevice
and the file-visibility poll each had their own op, but the share path's
distinctive steps were invisible on the dashboard, hidden inside the top-level
Controller/CreateVolume timing. Add hs_csi_operation coverage for:
- HammerspaceClient.WaitForTaskCompletion (+ span w/ attempt count) - the
CreateShare task poll, the share-backed analog of the file-visibility poll
and the likeliest share-create bottleneck
- HammerspaceClient.CreateShare / CreateShareFromSnapshot
- SetMetadataTags (now takes ctx) - the post-create share tag step
- ensureShareBackedVolumeExists / ensureBackingShareExists (+ spans)
Now the per-operation panels localize share-based latency the same way they
already do for file-based. Pure observability; no behavior change.
Reference for the OTel metrics/tracing subsystem: wiring (env-driven, no-op when off), the hs_csi_operation_* family + MeasureOp contract, the hs_csi_anvil_requests_total counter + AnvilRoute normalization, the file/share instrumentation coverage map, the Grafana dashboard rows/queries, cardinality rules, and how to extend it. Behavioral fixes/optimizations stay documented inline + in commit messages.
The per-volume / per-backing-share / per-snapshot in-memory locks were invisible
to metrics: a lock acquired but never released (holder stuck in an uninterruptible
mount syscall -> deferred unlock never runs) could only be inferred indirectly
(CreateVolume inflight high while every sub-op gauge reads 0).
Instrument acquireVolumeLock/acquireSnapshotLock via common.LockProbe:
hs_csi_locks_held{lock_type} up/down gauge - LEAK = stuck > 0
hs_csi_lock_wait_seconds{lock_type} histogram - acquire contention
hs_csi_lock_hold_seconds{lock_type} histogram - hold duration
hs_csi_lock_acquire_failures_total{lock_type} counter - 30s timeouts (-> Aborted)
The held gauge is the reliable leak signal; wait/hold quantify contention and slow
holders. Pure observability; lock behavior unchanged.
Design reference and runbook for the CSI keyed locks and the LockProbe instrumentation: the keyLock primitive and its 30s codes.Aborted timeout, which RPCs take which keys, why held pins at the provisioner sidecar's --worker-threads ceiling (not a driver limit), the four lock metrics and their exact counter identity, the release-rate-gated leak verdict, and the stale-mount root cause with manual remediation and the code fix.
…ng the lock When the driver is re-pointed to a new Anvil (or a data portal/Anvil is terminated), the backing-share NFS mount left at <ShareStagingDir><exportPath> still points at the old, now-unreachable server. Two unbounded operations then hang uninterruptibly while holding the volume + backing-share locks, so the deferred unlocks never run: the locks leak and every serialized file-backed CreateVolume behind them times out (Aborted) forever. Observed as hs_csi_locks_held stuck non-zero with no release. Fixes: - IsShareMounted now uses the timeout-bounded SafeIsMountPoint, so stat()ing a hung mount returns 'not cleanly mounted' instead of blocking. - MountShare bounds the mount syscall with a timeout (defaultMountCheckTimeout); a hard NFS mount to a dead server returns DeadlineExceeded so the lock is released and the op retried. - EnsureBackingShareMounted force-unmounts (umount -f -l) a lingering stale mount before re-mounting, so it re-establishes against the current data portal - this is what lets file-backed provisioning survive an Anvil swap. Pairs with the CSI lock-observability metrics that surfaced the leak. Groups with the os.Exit lock-crash fix on the scale/reliability lane.
mkfs.xfs 6.4+ prints three warnings when asked to format a filesystem
smaller than 300 MiB ("Filesystem should be larger than 300MB",
"Log size should be at least 64MB", and "Support for filesystems like
this one is deprecated and they will not be supported in future
releases") but still returns exit 0. The driver's existing err check
never fires, so a deprecated-format XFS gets silently created and may
fail to mount on future kernels.
Reject the request early with codes.InvalidArgument and a message that
tells the user what threshold applies and what they requested. This
lands before any share mount, qemu-img create, or mkfs, so no state is
left behind on Anvil.
mkfs.ext4 accepts filesystems as small as ~8 MiB, but the journal + reserved-block overhead means anything smaller than ~20 MiB has almost no usable space and typically hits ENOSPC on the first meaningful write. Reject in CreateVolume with codes.InvalidArgument and a message that tells the user the floor and what they requested. Restructured the fileBacked-only fsType switch so future fsType floors (f2fs, btrfs, etc.) drop in cleanly.
Anvil's file-snapshot endpoint captures the on-disk bytes of the backing
file as they exist on the storage server at snapshot time. For file-backed
XFS volumes that includes XFS's in-flight log — any journal transaction
that Anvil catches mid-commit will be rolled back by XFS log recovery on
the next mount, deleting the associated user data. The galaxy demo
consistently reproduced this: snapshot succeeded, restored file was
loop-mountable as XFS, but the filesystem root was empty.
Fix: before calling Anvil's snapshot, locate every running pod that has
the source volume mounted and issue `fsfreeze --freeze` on the mount.
XFS quiesces (log flushed, no in-flight transactions) before the snapshot
fires. Unfreeze after — always, even if the snapshot itself failed, so
the app pod isn't left blocked on writes.
Design notes:
- Freeze is executed from the csi-node DaemonSet pod on the source
volume's node, not from the user's app pod. The user's image may be
nginx-alpine / distroless / scratch and lack fsfreeze; csi-node ships
util-linux and has kubelet mount propagation, so it can freeze
/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount
directly.
- The Freezer is nil (no-op) when the driver isn't running in-cluster,
keeping local `make sanity` runs functional.
- Failure to freeze is logged and best-effort — matching Velero pre-hook
semantics; the snapshot still fires. Unfreeze failure is escalated to
ERROR because a stuck freeze permanently blocks writes.
Requires new RBAC — the csi-provisioner ClusterRole needs 'get' on pods
and 'create' on pods/exec. Deployment side (plugin.yaml) is updated
separately.
Verified end-to-end with the galaxy demo on fsType=xfs, 1 GiB PVC.
Before: rollback pod showed empty filesystem. After: rollback pod
shows the galaxy SVG that was on disk at snapshot time.
Also updates go.mod / go.sum to pull in k8s.io/{api,apimachinery,client-go}
v0.33.6, matching the k8s.io/kubernetes v1.33.6 already in use.
…GetShare probe
The driver treated the volume ID as opaque and issued GET /shares/{name} to
learn a volume's type, using the 404 as the 'file-backed' signal. For
file-backed volumes (a file inside a shared backing share) that probe 404s on
every Delete/Expand/CreateSnapshot/DeleteSnapshot -- thousands of wasted Anvil
round-trips at scale, and it dominated REST traffic.
The volume ID already encodes the type structurally: CreateVolume writes
file-backed IDs as <prefix><backingShare>/<file> (an extra path segment) and
share-backed IDs as <prefix><share>. isFileBackedVolumeID() reads that off the
ID with zero REST calls. GetShare is retained only as a fallback on the
non-file-backed branch, so legacy/unexpected handles still resolve correctly.
Routing is unchanged; only the redundant probe is removed.
…ial backoff Share-create tasks always take >2s and almost always finish under ~15s. The old jpillora exponential backoff (Min 100ms, Factor 1.5, capped at 30s, sleep-first) grew the inter-poll gap to 6-30s by the time a share was ready, so detection lag could add up to a full 30s AFTER the task had already completed - which is what inflated the observed WaitForTaskCompletion latency. Replace it with a fixed cadence: poll every 2s for the first 30s (covers the normal completion window with ~2s detection latency), then relax to every 4s for the long tail, up to the unchanged 3600s deadline. Drops the jpillora/backoff dependency from this path.
…ng-share lock The per-backing-share lock was held across the entire file-backed CreateVolume path - both ensureBackingShareExists (share create-if-not-exists, which genuinely needs serialization) AND ensureDeviceFileExists (the per-file mkfs, which does not). That serialized ALL file creation on a backing share to one-at-a-time (~2-3/s) even though each create is ~0.3s of work and the Anvil (~5000/s capable) and controller node CPU sit idle. Narrow the lock to only the share create-if-not-exists, then release it, and keep the backing share mounted for the duration of concurrent creates via a refcount (acquire/releaseBackingMount) instead of mount/unmount under the lock per file. Per-file mkfs now runs concurrently across the provisioner worker threads; the share is unmounted only when the last in-flight create releases. File-backed only; share-backed and the NFS-directory-in-a-base-share path are untouched.
…e load mkfs.ext4 without flags eagerly zeroes the inode table and journal (~tens of MB per 1GB volume). For file-backed volumes those writes go over NFS to the backing share, saturating the DSX disk (measured ~442 MB/s / 100% util, mkfs p95 ~9.6s, capping CreateVolume at ~12/s). Passing -E lazy_itable_init=1,lazy_journal_init=1 defers that zeroing to lazy kernel background init, cutting create-time writes to ~KB. (XFS is not a workaround: mkfs.xfs is ~2x heavier over NFS, p95 ~19s.)
mkfs.xfs discards (TRIMs) the whole file range by default; over an NFS-backed file that pass is slow overhead (measured mkfs.xfs p95 ~19s vs ext4 lazy ~9.6s). -K skips it; the backing share manages its own space reclaim.
The bottleneck ladder (backing-share lock -> provisioner API throttle -> mkfs write volume/DSX saturation -> kube-controller-manager binding), each fix and how it was measured, the ext4/xfs/btrfs comparison, deployment tuning knobs, and the remaining ~15/s open item.
…reateVolume Adds objectiveTarget (share|file|both, default share). With the default, file-backed CreateVolume skips the per-file objective-set and the Anvil file-visibility poll that exists only to gate it, returning as soon as the local mkfs completes. The backing share already carries the objectives, so per-file objectives are redundant in the common single-site shape. This eliminates the GET /files 500-storm that poll generates under concurrency (measured: 240 500s for 10 concurrent volumes). Per-file objectives (and the poll) are still available via objectiveTarget=file|both for per-volume/multi-site policy. Metadata tags still apply on the fast path - they are local mount operations needing no Anvil round-trip. Config-gated version of the earlier unconditional experiment (8bde551). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add objectiveTarget to the example file-backed StorageClass, and an Unreleased CHANGELOG section covering the observability, perf, hardening, and fast-create work in this stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ifest
Tests (all green):
- Fix TestParseParams for the objectiveTarget default ("share").
- TestParseObjectiveTarget (share default / share|file|both / invalid->InvalidArgument).
- TestCheckFileBackedMinSize (xfs<300MiB, ext4<20MiB gates) — extracted the size
gate from CreateVolume into checkFileBackedMinSize for testability.
- TestIsFileBackedVolumeID (file- vs share-backed volume-ID discriminator).
- TestAnvilRoute + TestMeasureOp (metric route normalization + MeasureOp contract).
- Assert lock-acquire timeout returns codes.Aborted (not os.Exit) in
TestAcquireVolumeLockTimeout.
Monitoring (new deploy/monitoring/):
- grafana/hs-csi-driver-dashboard.json — the "Hammerspace CSI Driver" dashboard.
- victoriametrics/scrape.yml — example scrape config for the :9090/:9091 endpoints.
- README.md — enable-metrics -> scrape -> import walkthrough.
Kubernetes 1.36:
- deploy/kubernetes/kubernetes-1.36/plugin.yaml — validated on 1.36 (1.29 base +
host-networked metrics port + OTel env vars); documented in deploy/kubernetes/README.md.
Docs: objectiveTarget already in the example SC + CHANGELOG; observability.md now
points at deploy/monitoring/. Also gofmt-clean the stack files.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ravi100k
left a comment
ravi100k
left a comment
There was a problem hiding this comment.
Ran this branch end-to-end (built the driver image, redeployed csi-provisioner/csi-node in a live cluster against a real Anvil, and passed all 4 provisioning scenarios: file-backed, nfs, nfs-mount-backing, block). Functionally solid. While reviewing the diff for units B/C/F/G I found a few concurrency/observability issues worth a look before merge — details inline. None of these showed up in the happy-path test above; they're about edge cases (timeouts, concurrent create/delete, cancelled RPCs).
| // best-effort so the mount below re-establishes against the CURRENT | ||
| // data portal rather than stacking onto - or reusing - a dead mount. | ||
| // This is what makes file-backed provisioning survive an Anvil swap. | ||
| common.ForceUnmountStale(backingDir) |
There was a problem hiding this comment.
Force-unmount can fire on a healthy-but-slow mount, not just a dead one.
IsShareMounted (via SafeIsMountPoint, pkg/common/host_utils.go:591-608,701-722) treats any error the same way, including its own 50s timeout: on context.DeadlineExceeded it returns false — identical to "definitely not mounted." That false is what triggers ForceUnmountStale (umount -f -l) here.
backingDir is the shared staging mount for the whole backing share, used by every concurrent file-backed volume on this node. A single slow stat() on a congested/busy NFS server (arguably more likely under the higher concurrency this PR enables via unit G) reads as "stale" and force-lazy-unmounts the mount out from under any other in-flight pod using it — the same class of outage this fix is meant to prevent, just reachable via a false positive instead of a real dead mount.
Would it make sense to gate the force-unmount on a more positive signal of staleness (e.g. a quick reachability probe of the export/portal, or requiring N consecutive timeouts) rather than a single SafeIsMountPoint timeout?
| @@ -263,7 +333,7 @@ func (d *CSIDriver) UnmountBackingShareIfUnused(ctx context.Context, backingShar | |||
| } | |||
|
|
|||
| log.Infof("unmounting backing share %s", mountPath) | |||
There was a problem hiding this comment.
UnmountBackingShareIfUnused's "in use" check doesn't know about mountRefs, and can race with unit G's concurrent mkfs.
This function is called directly (bypassing the new mountRefs refcount entirely) from controller.go:244, controller.go:930, and several sites in node_helper.go. Its only "still in use" signal is whether losetup -a shows a loop device backed by a file under mountPath.
But in ensureDeviceFileExists (controller.go:462-486), mkfs.<fstype> (FormatDevice) runs directly on the plain backing file over the NFS mount — no loop device exists for a file-backed volume until it's later staged on a node. So during the entire CreateVolume window (including the mkfs that unit G intentionally lets run without holding the per-backing-share lock), losetup -a shows nothing for that file, and this function will happily report "unused."
Concretely: volume A's CreateVolume calls acquireBackingMount (mountRefs[share]=1) and starts mkfs on its backing file over the shared mount, lock released per unit G. A concurrent DeleteVolume for volume B on the same backing share calls this function directly via controller.go:244's defer, sees no loop device anywhere under mountPath, and unmounts the share — out from under volume A's in-flight mkfs, even though mountRefs[share] is 1.
Suggest also checking d.mountRefs[mountPath] > 0 here before unmounting, so the two mechanisms agree on what "in use" means.
| // Always unfreeze, even if snapshot failed — otherwise the app pod | ||
| // stays blocked on writes indefinitely. | ||
| if d.freezer != nil && fileBackedSource { | ||
| d.freezer.Unfreeze(ctx, frozen) |
There was a problem hiding this comment.
Freeze/unfreeze run on the cancelable request ctx; a cancelled/expired RPC between them can leave the pod's filesystem frozen.
FreezeForVolumeHandle (line 1449) and this Unfreeze call both use the plain gRPC ctx. execFsfreeze (freezer.go:266) does exe.StreamWithContext(ctx, ...) — if ctx is already cancelled/expired (client-side deadline, or the snapshotter sidecar giving up while SnapshotFile/SnapshotShare at lines 1454-1456 is still running), the unfreeze exec never actually reaches the pod; it just fails fast with a context error. freezer.go:122's log ("filesystem may remain frozen; investigate") acknowledges this can happen but there's no fallback — the app pod stays frozen (all writes blocked) until a manual fsfreeze -u or pod restart.
This PR already has the fix for this exact class of problem a few hundred lines away: context.WithoutCancel(ctx) at controller.go:494, used specifically so the objective/metadata step survives gRPC cancellation. Worth applying the same pattern to Unfreeze (or at minimum wrapping freeze/unfreeze in a defer against a detached context) so a cancelled snapshot RPC can't leave a workload's filesystem stuck frozen.
| segs := strings.Split(urlPath, "/") | ||
| for i := 1; i < len(segs); i++ { | ||
| switch segs[i-1] { | ||
| case "shares", "tasks", "files", "objectives": |
There was a problem hiding this comment.
Missing share-snapshots case — the exact cardinality blowup this function exists to prevent.
hsclient.go hits /share-snapshots/snapshot-create/{shareName}, /share-snapshots/snapshot-list/{shareName}, and /share-snapshots/snapshot-delete/{shareName}/{snapshotName} (lines 1036, 1060, 1090). None of those match shares/tasks/files/objectives/file-snapshots here, so for those paths AnvilRoute returns the URL untouched — arbitrary share names (and for the delete path, also per-snapshot timestamps) flow straight into the http.route label on hs_csi_anvil_requests_total. That's an unbounded series per share (and per snapshot, for deletes), which is the cardinality problem this function is meant to guard against. TestAnvilRoute doesn't have a case for this family either, so it read as covered.
Suggest adding a "share-snapshots" case that collapses the share-name (and snapshot-name, for delete) segments to {id}, same as the existing cases.
| common.MinXfsSizeBytes, common.MinXfsSizeBytes/mib, | ||
| requestedSize, requestedSize/mib) | ||
| } | ||
| case "ext4": |
There was a problem hiding this comment.
ext3 has no size floor, but is formatted identically to ext4.
This switch only guards xfs and ext4. FormatDevice (pkg/common/host_utils.go:270) explicitly treats ext3 the same as ext4 for the lazy-init mkfs tuning (fsType == "ext4" || fsType == "ext3"), which implies the same "almost no usable space below ~20MiB" concern applies — but a tiny ext3 file-backed volume sails through this check with no floor at all.
Suggest adding "ext3" alongside "ext4" here (or case "ext4", "ext3":).
Add deploy/kubernetes/kubernetes-1.34/ and kubernetes-1.35/ (parity with 1.36: 1.29 base + host-networked metrics port + OTel env). 1.34/1.35/1.36 are now the supported+validated set. Default OTEL_TRACES_EXPORTER=none across all three (console spans are verbose; metrics=prometheus stays on); fix the now-stale inline comment. Update deploy/kubernetes/README.md compat range and CHANGELOG. Validated end-to-end on live AWS k8s 1.34.10 and 1.35.7 clusters (deployed in parallel, pointed at a shared Anvil with unique per-cluster backing shares): file-backed PVC create + pod mount + delete, 0 provisioner restarts, metrics scraped into VictoriaMetrics labeled by k8s_version. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- VERSION -> v1.3.0 (ldflag-injected into pkg/common.Version at build). - Supported manifests (kubernetes-1.34/1.35/1.36) image tag -> v1.3.0; historical 1.25-1.29 keep their pinned tags. - CHANGELOG: [Unreleased] -> [1.3.0]. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
e8a9f6e to
576c0a3
Compare
…nt deadlock fix
Review fixes:
1. Don't force-unmount a backing share on a single mount-check timeout
(classifyMount requires mountStaleProbes consecutive timeouts).
2. UnmountBackingShareIfUnused honors the mountRefs refcount.
3. Snapshot Unfreeze runs on a context detached from gRPC cancellation.
4. AnvilRoute collapses share-snapshots ids to {id} (metric cardinality).
5. Drop ext3 as a supported file-backed fsType (reject with InvalidArgument).
Plus, found during live ext3/ext4/xfs validation on a 2-node AWS cluster
against a lab Anvil: releaseBackingMount self-deadlocked when a file-backed
volume dropped the LAST backing-share reference. It held mountRefsMu while
calling UnmountBackingShareIfUnused, which re-locks the same non-reentrant
mutex via backingMountInUse -> the goroutine wedged forever, never releasing
mountRefsMu or the volume lock, so every later create failed to acquire the
volume lock. ext4 hid it (xfs still held a ref, so refcount != 0 skipped the
unmount); a lone xfs PVC (refcount 1 -> 0) reproduced it every time. Fixed by
dropping the refcount under the lock (dropBackingRef) and unmounting only
after releasing it. Added TestDropBackingRefNoDeadlock (watchdog-guarded).
Validated live: ext3 rejected, ext4 + xfs bound and pods running (fs=xfs
confirmed), ext4 snapshot readyToUse=true, all within the default 60s timeout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013r1qXHBJyPeVXWMvVS7iwp
Follow-up hardening from the deadlock audit. acquireBackingMount held the global mountRefsMu across EnsureBackingShareMounted, which performs a real NFS mount that can block up to the ~5 min command-exec timeout on a slow or dead portal. While held, every other file-backed create/delete blocked on mountRefsMu — including refcount reads (backingMountInUse) and releases on unrelated backing shares. Not a lock cycle, but the same "raw mutex held too long" liveness hazard as the self-deadlock just fixed, and it re-wedges the serialized-provisioning path the PR set out to remove. Fix: serialize the actual mount/unmount of a given backing share with a per-directory lock (mountLockFor); mountRefsMu now guards only the map updates and is held for microseconds. The reference is reserved before the mount (bumpBackingRef returns the 0->1 transition), so a concurrent delete's UnmountBackingShareIfUnused can't observe refcount 0 mid-mount and unmount the share out from under an in-flight create — a window the old global-lock design closed incidentally. releaseBackingMount takes the same per-dir lock so an unmount can't race a concurrent remount. Tests: TestBumpBackingRefFirst (0->1 transition), and TestBackingMountLockDoesNotBlockRefcount — holds a per-dir mount lock (simulating a hung mount) and asserts refcount ops on the same and other dirs still complete under a watchdog. Full driver suite passes under -race. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013r1qXHBJyPeVXWMvVS7iwp
3 participants
Stacked on #66 (
fix/fqdn-no-dataportals), so this diff is exactly the units below — no fqdn duplication. Built as a linear commit stack; can be split into per-unit PRs on request (branches available).What's here (9 units)
acquireVolumeLock/acquireSnapshotLockreturncodes.Abortedon lock-timeout instead ofos.Exit(1)(which crashed the whole controller under concurrent load).OTEL_TRACES_EXPORTER).hs_csi_operation_*+hs_csi_anvil_requests_total(Prometheus/OTLP). Seedocs/observability.md.docs/node-unmount-recovery.md.fsfreezesource volume before snapshot.GetShareprobe that 404s; also routes file-backed snapshot deletes correctly.docs/tunable-retry-parameters.md.docs/file-backed-performance.md.objectiveTargetStorageClass param (sharedefault |file|both): default skips the per-file objective-set + the Anvil visibility poll that gates it.Validation (real Anvil + DSX + 3-node k8s in AWS)
GET /files500-poll-storm dropped 240 → 10 for 10 concurrent volumes; per-fileobjective-set10 → 0; 10-concurrent bind 26s → 8s.Notes for review
objectiveTargetis documented indeploy/kubernetes/example_storage_class_file_backed.yamlandCHANGELOG.md.go.mod(grpc 1.79→1.80, protobuf, x/net etc.) — flagged for a deliberate look.🤖 Generated with Claude Code
https://claude.ai/code/session_013r1qXHBJyPeVXWMvVS7iwp