Repository navigation
fix(accounts): racing ensures of one derived role serialize on the role row - #1261
Merged
Merged
Conversation
…le row #1259 made EnsureDerivedRole read the role and its permissions and write only what differs. The read took no lock under READ COMMITTED, so two ensures of the same cause with different declarations both read before either committed and the role ended at their union, keeping a permission the later declaration dropped; the upsert it replaced serialized them. The role read is now SELECT ... FOR UPDATE: a row lock, which fires no trigger and moves no revision, and which app_tenant may take through its UPDATE grant on the description column. The second ensure waits, reads what the first committed, and ends at its own declaration. The permission insert gets back ON CONFLICT DO NOTHING so an add a racing writer already made cannot fail on the primary key. Tests: a racing pair (the first adds doc:write and stays open, the second declares only solution:use and must wait, ending at solution:use), and the revision test now also covers a pure removal moving the revision and an idle re-ensure after it moving none. The race test fails with the lock removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rant of the role waits on FOR UPDATE conflicts with the FOR KEY SHARE every row referencing the role takes through its foreign key. installSolution moves the organization's revision (its agent principal) and then writes the standing grant of the solution role; an ensure of that role with a changed declaration held the role row under FOR UPDATE and then needed the revision row, and the two deadlocked (SQLSTATE 40P01). FOR NO KEY UPDATE is the lock the replaced upsert took: it still serializes two ensures of one role and does not conflict with a reference to it. The permission insert drops ON CONFLICT DO NOTHING again: under the role lock no other ensure can add a permission between the read and the insert, and UpdateRole refuses a derived role, so the clause could only hide a broken lock. MODULE_INSTALLATION.md states that a re-derivation writes only what differs and that two of one cause serialize, last commit wins. Test: an ensure must not deadlock a writer that moved the revision and then references the role; it fails with 40P01 under FOR UPDATE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
antoinetoussaint-byte
removed this pull request from the merge queue due to a manual request
Oct 27, 2026
Contributor
Author
|
lazybox has finished working on this. claim detail |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1259, from its review.
What was wrong
#1259 made
EnsureDerivedRoleread the role and its permissions and write only what differs, so idle reconcile passes stop moving the authorization revision. But the role was read with no lock under READ COMMITTED. Two ensures of the same cause with different declarations both read before either committed, so the role ended at their union, which kept a permission the later declaration had dropped until the next pass. The upsert it replaced serialized them on the row.Fix (classified: fix, at the owner of the behaviour)
SELECT … FOR NO KEY UPDATE. That is a row lock, which fires no trigger and moves no revision.app_tenantmay take it through its column UPDATE grant ondescription(migration 5), androles_polymorphicadmits the row. The second ensure waits, reads what the first committed, and ends at its own declaration, so the last writer wins, as before fix(accounts): ensuring a derived role that is already as declared writes nothing #1259.FOR UPDATE, which conflicts with theFOR KEY SHAREthat every row referencing the role (scope_grants,role_assignments,role_permissions,record_shares) takes through its foreign key.installSolutionmoves the organization's revision (its agent principal insert) and then writes the standing grant of the solution role. An ensure of that role with a changed declaration held the role row and then needed the revision row, and the two deadlocked (SQLSTATE 40P01, reproduced by the new test).FOR NO KEY UPDATEis the lock the replaced upsert took. It serializes ensures with each other and does not conflict with a reference.ON CONFLICT DO NOTHING. Under the role lock, no other ensure can add a permission between the read and the insert, andUpdateRolerefuses a derived role, so the clause could only hide a broken lock. No constraint violation is caught anywhere. The lock is the only serialization.Lock ordering, checked across every writer of a derived role row:
solutionRoleIDruns the ensure in its own transaction holding nothing before it.carryModuleAccessTxruns it after the cause grant's write, which takes the installation row and the org'ssource_read_revisionsrow, so two carries of one cause serialize there first.UpdateRoletakesFOR UPDATEand refuses a derived role, holding nothing before.DeleteRolereads unlocked and refuses. The catalog importer excludes derived roles. Every path takes the role row before the organization revision row, the order the replaced upsert had.Not the authority core's: #1257 moved authority, composition and installation decisions into the shared core. This is row locking on accounts' own
rolestable, which the core neither stores nor decides.Reused: the existing row, index, grant and policy, and the lock mode the replaced upsert took. No new mechanism.
Docs
module/MODULE_INSTALLATION.md, "Reconciliation has no background pass": a re-derivation writes only what differs (an idle one leaves the authorization revision where it is), and two re-derivations of one cause serialize on the role row, ending at the declaration of the one that commits last.Tests
TestRacingEnsuresOfADerivedRoleEndAtTheLastDeclaration(new): tx1 adds {solution:use, doc:write} and stays open. tx2 declares only {solution:use}, with the same description so it has no write that could wait on the revision row, and must still be waiting after 500 ms. Once tx1 commits, the role is {solution:use}. Mutation: with the lock removed (main's production code), it fails with "the second ensure did not wait for the first to commit (err=)".TestAnEnsureDoesNotDeadlockAWriterThatReferencesTheRole(new): a writer moves the revision, then takes the FK'sFOR KEY SHAREon the role while an ensure with a changed declaration holds the role row. Both must commit. Mutation: withFOR UPDATEit fails withdeadlock detected (SQLSTATE 40P01).TestEnsuringADerivedRoleAsDeclaredMovesNoRevision(extended): idle re-ensures move no revision; permission and description changes do; a pure removal moves it and deletes the row; an idle re-ensure after the removal moves nothing.go test -p 1 ./pkg/infra/...passes;go vet ./pkg/infrais clean; the naming gate andcheck-boundaries.share clean.Not verified locally:
./pkg/businessand./pkg/adapters(left to CI), and a real concurrentinstallSolutionagainst a changed solution declaration. The deadlock test reproduces the lock order with an explicitFOR KEY SHARE, not throughinstallSolutionitself.🤖 Generated with Claude Code