Preserve API watch event ordering during mutations

[David Negstad (danegsta)]

Concurrent mutations in Tilt's filepath storage can commit in resource-version order but publish their watch events in the opposite order. During shutdown, this can regress a controller cache from a deleting object to an older non-deleting object, preventing finalizer cleanup and causing resource cleanup to time out.

This change serializes mutating requests per API group/resource around the complete storage handler operation. Parent-resource and subresource mutations share the same lock because they publish to the same watch stream, while reads and mutations of different resource types remain concurrent.

The tests include a deterministic reproduction using Tilt's actual filepath storage. It commits a status update, delays its notification, and then commits deletion, proving that the unprotected storage can publish the higher-resource-version deletion before the stale status event. The same experiment verifies that the middleware preserves storage and watch ordering.

This is a local containment measure. The underlying ordering guarantee should ultimately be enforced by Tilt so persistence and watch publication cannot be reordered.

Validation

• make lint
• make test
• go test -count 1 -race ./internal/apiserver -timeout 180s

[Copilot]

Copilot review overview

🔵 Needs a closer look

The concurrency change affects mutation throughput and ordering across every API resource and warrants final human review.

Review effort: Balanced
Findings: None

What changed in this PR

Serializes API mutations per group/resource to keep Tilt storage commits and watch events consistently ordered.

Changes:

• Adds mutation-ordering middleware to the API server handler chain.
• Shares locks across parent resources, versions, and subresources.
• Adds middleware and real filepath-storage concurrency tests.

File	Description
internal/​apiserver/​apiserver.go	Registers the ordering middleware.
internal/​apiserver/​ordered_resource_mutation_handler.go	Implements per-resource mutation serialization.
internal/​apiserver/​ordered_resource_mutation_handler_test.go	Tests lock scope and mutation detection.
internal/​apiserver/​ordered_resource_mutation_storage_test.go	Reproduces and verifies watch-event ordering.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[David Negstad (danegsta)]

/backport to release/0.26

[github-actions]

Started backporting to release/0.26: https://github.com/microsoft/dcp/actions/runs/36035788308

[github-actions]

David Negstad (@danegsta) backport PR couldn't be created automatically, please create the backport PR manually!

Open backport PR into release/0.26.