Skip to content

fix(accounts): racing ensures of one derived role serialize on the role row - #1261

Merged
antoinetoussaint-byte merged 2 commits into
mainfrom
accounts/derived-role-lock
Oct 27, 2026
Merged

antoinetoussaint-byte merged 2 commits into
mainfrom
accounts/derived-role-lock

Conversation

@antoinetoussaint-byte

@antoinetoussaint-byte antoinetoussaint-byte commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #1259, from its review.

What was wrong

#1259 made EnsureDerivedRole read the role and its permissions and write only what differs, so idle reconcile passes stop moving the authorization revision. But the role was read with no lock under READ COMMITTED. Two ensures of the same cause with different declarations both read before either committed, so the role ended at their union, which kept a permission the later declaration had dropped until the next pass. The upsert it replaced serialized them on the row.

Fix (classified: fix, at the owner of the behaviour)

  • The role read is SELECT … FOR NO KEY UPDATE. That is a row lock, which fires no trigger and moves no revision. app_tenant may take it through its column UPDATE grant on description (migration 5), and roles_polymorphic admits the row. The second ensure waits, reads what the first committed, and ends at its own declaration, so the last writer wins, as before fix(accounts): ensuring a derived role that is already as declared writes nothing #1259.
  • It is NO KEY UPDATE, not FOR UPDATE. This was a review finding: the first revision of this PR used FOR UPDATE, which conflicts with the FOR KEY SHARE that every row referencing the role (scope_grants, role_assignments, role_permissions, record_shares) takes through its foreign key. installSolution moves the organization's revision (its agent principal insert) and then writes the standing grant of the solution role. An ensure of that role with a changed declaration held the role row and then needed the revision row, and the two deadlocked (SQLSTATE 40P01, reproduced by the new test). FOR NO KEY UPDATE is the lock the replaced upsert took. It serializes ensures with each other and does not conflict with a reference.
  • The permission insert has no ON CONFLICT DO NOTHING. Under the role lock, no other ensure can add a permission between the read and the insert, and UpdateRole refuses a derived role, so the clause could only hide a broken lock. No constraint violation is caught anywhere. The lock is the only serialization.

Lock ordering, checked across every writer of a derived role row: solutionRoleID runs the ensure in its own transaction holding nothing before it. carryModuleAccessTx runs it after the cause grant's write, which takes the installation row and the org's source_read_revisions row, so two carries of one cause serialize there first. UpdateRole takes FOR UPDATE and refuses a derived role, holding nothing before. DeleteRole reads unlocked and refuses. The catalog importer excludes derived roles. Every path takes the role row before the organization revision row, the order the replaced upsert had.

Not the authority core's: #1257 moved authority, composition and installation decisions into the shared core. This is row locking on accounts' own roles table, which the core neither stores nor decides.

Reused: the existing row, index, grant and policy, and the lock mode the replaced upsert took. No new mechanism.

Docs

  • module/MODULE_INSTALLATION.md, "Reconciliation has no background pass": a re-derivation writes only what differs (an idle one leaves the authorization revision where it is), and two re-derivations of one cause serialize on the role row, ending at the declaration of the one that commits last.

Tests

  • TestRacingEnsuresOfADerivedRoleEndAtTheLastDeclaration (new): tx1 adds {solution:use, doc:write} and stays open. tx2 declares only {solution:use}, with the same description so it has no write that could wait on the revision row, and must still be waiting after 500 ms. Once tx1 commits, the role is {solution:use}. Mutation: with the lock removed (main's production code), it fails with "the second ensure did not wait for the first to commit (err=)".
  • TestAnEnsureDoesNotDeadlockAWriterThatReferencesTheRole (new): a writer moves the revision, then takes the FK's FOR KEY SHARE on the role while an ensure with a changed declaration holds the role row. Both must commit. Mutation: with FOR UPDATE it fails with deadlock detected (SQLSTATE 40P01).
  • TestEnsuringADerivedRoleAsDeclaredMovesNoRevision (extended): idle re-ensures move no revision; permission and description changes do; a pure removal moves it and deletes the row; an idle re-ensure after the removal moves nothing.
  • Locally, on the repo's test Postgres at aaf998b + this branch: all three pass three runs in a row; go test -p 1 ./pkg/infra/... passes; go vet ./pkg/infra is clean; the naming gate and check-boundaries.sh are clean.

Not verified locally: ./pkg/business and ./pkg/adapters (left to CI), and a real concurrent installSolution against a changed solution declaration. The deadlock test reproduces the lock order with an explicit FOR KEY SHARE, not through installSolution itself.

🤖 Generated with Claude Code

…le row

#1259 made EnsureDerivedRole read the role and its permissions and write
only what differs. The read took no lock under READ COMMITTED, so two
ensures of the same cause with different declarations both read before
either committed and the role ended at their union, keeping a permission
the later declaration dropped; the upsert it replaced serialized them.

The role read is now SELECT ... FOR UPDATE: a row lock, which fires no
trigger and moves no revision, and which app_tenant may take through its
UPDATE grant on the description column. The second ensure waits, reads
what the first committed, and ends at its own declaration. The permission
insert gets back ON CONFLICT DO NOTHING so an add a racing writer already
made cannot fail on the primary key.

Tests: a racing pair (the first adds doc:write and stays open, the second
declares only solution:use and must wait, ending at solution:use), and the
revision test now also covers a pure removal moving the revision and an
idle re-ensure after it moving none. The race test fails with the lock
removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antoinetoussaint-byte antoinetoussaint-byte added the working Claimed by a lazybox agent; cleared when the session ends. label Oct 27, 2026
@antoinetoussaint-byte
antoinetoussaint-byte added this pull request to the merge queue Oct 27, 2026
…rant of the role waits on

FOR UPDATE conflicts with the FOR KEY SHARE every row referencing the
role takes through its foreign key. installSolution moves the
organization's revision (its agent principal) and then writes the
standing grant of the solution role; an ensure of that role with a
changed declaration held the role row under FOR UPDATE and then needed
the revision row, and the two deadlocked (SQLSTATE 40P01). FOR NO KEY
UPDATE is the lock the replaced upsert took: it still serializes two
ensures of one role and does not conflict with a reference to it.

The permission insert drops ON CONFLICT DO NOTHING again: under the role
lock no other ensure can add a permission between the read and the
insert, and UpdateRole refuses a derived role, so the clause could only
hide a broken lock.

MODULE_INSTALLATION.md states that a re-derivation writes only what
differs and that two of one cause serialize, last commit wins.

Test: an ensure must not deadlock a writer that moved the revision and
then references the role; it fails with 40P01 under FOR UPDATE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antoinetoussaint-byte
antoinetoussaint-byte removed this pull request from the merge queue due to a manual request Oct 27, 2026
@antoinetoussaint-byte
antoinetoussaint-byte added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 74e35f0 Oct 27, 2026
26 checks passed
@antoinetoussaint-byte

antoinetoussaint-byte commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

lazybox has finished working on this. claude (Opus 5.5) held it from 2026-10-27 13:12 UTC until 2026-11-02 21:22 UTC. The working label has been removed; this comment is the record.

claim detail
Lazybox-Claim-Device: effacd542b611010656e
Lazybox-Claim-Session: b9efe85c38
Lazybox-Claim-Workspace: github-codefly-dev-module-saas-starter-1261
Lazybox-Claim-Agent: claude
Lazybox-Claim-Model: Opus 5.5
Lazybox-Claim-Started: 2026-10-27T13:12:44Z
Lazybox-Claim-Heartbeat: 2026-11-02T21:22:59Z
Lazybox-Claim-Expires: 2026-11-02T16:08:11Z
Lazybox-Claim-Released: 2026-11-02T21:22:59Z

@antoinetoussaint-byte antoinetoussaint-byte removed the working Claimed by a lazybox agent; cleared when the session ends. label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant