Skip to content

fix(cleanup): remove old steps that are part of the saved state - #9582

Open
mejo- wants to merge 2 commits into
mainfrom
fix/cleanup_steps
Open

mejo- wants to merge 2 commits into
mainfrom
fix/cleanup_steps

Conversation

@mejo-

@mejo- mejo- commented Nov 10, 2026

Copy link
Copy Markdown
Member

Steps were only removed once a document had no sessions at all. Sessions that pushed steps but were not closed cleanly are kept for 90 days, so documents in regular use never got there and oc_text_steps kept growing.

Remove steps that are older than 7 days and part of the last saved version, except for the 200 steps before it that recovering clients receive along with the document state. A client that missed removed steps sends a sync query and recovers from the document state. Documents without a state file are skipped, as recovering clients get all of their steps.

Adds a playwright test for recovering missing steps via document state.

Fixes: #9080

Assisted-by: ClaudeCode:claude-opus-5-5

🏁 Checklist

  • Code is properly formatted (npm run lint / npm run stylelint / composer run cs:check)
  • Sign-off message is added to all commits
  • Tests (unit, integration and/or end-to-end) passing and the changes are covered with tests

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI tools
  • The AI-generated content was reviewed, comprehended and tested by a human

@mejo- mejo- self-assigned this Nov 10, 2026
@mejo- mejo- added bug Something isn't working 3. to review labels Nov 10, 2026
@github-project-automation github-project-automation Bot moved this to 🧭 Planning evaluation (don't pick) in 📝 Productivity team Nov 10, 2026
@nextcloud-command nextcloud-command added the AI assisted This PR contains AI-assisted commits label Nov 10, 2026
@mejo- mejo- moved this from 🧭 Planning evaluation (don't pick) to 🏗️ In progress in 📝 Productivity team Nov 10, 2026
@mejo-
mejo- force-pushed the fix/cleanup_steps branch from 27fb4bc to c14224f Compare November 10, 2026 09:30
@mejo-

mejo- commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

/backport to stable35

Steps were only removed once a document had no sessions at all.
Sessions that pushed steps but were not closed cleanly are kept for
90 days, so documents in regular use never got there and
oc_text_steps kept growing.

Remove steps that are older than 7 days and part of the last saved
version, except for the 200 steps before it that recovering clients
receive along with the document state. A client that missed removed
steps sends a sync query and recovers from the document state.
Documents without a state file are skipped, as recovering clients
get all of their steps.

Adds a playwright test for recovering missing steps via document state.

Fixes: #9080

Signed-off-by: Jonas <jonas@freesources.org>
Assisted-by: ClaudeCode:claude-opus-5-5
Should speed up playwright tests in CI a bit.

Signed-off-by: Jonas <jonas@freesources.org>
@mejo-
mejo- force-pushed the fix/cleanup_steps branch from c14224f to b97c836 Compare November 10, 2026 16:49

@blizzz blizzz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two cases were found, the first one might be a blocker.

a stale client misses deleted steps without noticing, then its save overwrites other people's edits

Why the client doesn't notice:

  • The client only asks for recovery when Yjs is left with pendingStructs. The check is at src/services/y-websocket.js:53-63. When that happens, Yjs sends a SyncStep1 message while steps are being applied, and WebSocketPolyfill.ts:104-120 turns that into sendRecoveryStep.
  • The 30-second resync timer (y-websocket.js:347-351) also only fires when there are pendingStructs.
  • Nothing else triggers recovery, and the server never checks for gaps. Both sync and push simply return find(id > version) (ApiService.php:168, DocumentService.php:292).

Why there are no pending structs in this case:

  • Each step row holds one client's push: encodeStateAsUpdate(doc, sv(remote)) at y-websocket.js:383-384.
  • B's insertions into paragraph 1 point to items in paragraph 1, so the stale client can apply them even without A's paragraph-2 items.
  • If B deleted something A had written, that delete goes to pendingDs. Nothing checks pendingDs.

Why the bad save is accepted:

  • The stale client's version jumps to the newest step id (WebSocketPolyfill.ts:91).
  • So the outdated check in autosave (DocumentService.php:420) passes, and the client writes both the .md file and the .yjs state file without A's edit.

Why a client can be this stale:

  • A session that pushed steps is never removed by deleteInactiveWithoutSteps. Only removeOldSessions removes it, after 90 days.
  • getValidSession (SessionService.php:165-188) only checks that the session row exists. So a laptop that resumes after 8 days is still accepted.

Reproduction (manual): the same steps as before.

  1. S edits a.md, then blocks /session/* requests in devtools.
  2. A edits paragraph 2 and closes the tab.
  3. B makes more than 200 pushes in paragraph 1, and autosave runs.
  4. Run UPDATE oc_text_steps SET timestamp = timestamp - 700000, then occ background-job:execute <Cleanup id> --force-execute.
  5. Unblock S. No request carries recoveryAttempt, and A's text is missing. S types something, waits 10 seconds for autosave, then reload: A's edit is gone.

Reproduction (unit, Vitest, in the style of src/tests/yjs.spec.js; not run yet):

const base = new Y.Doc(); /* fragment with <p>one</p><p>two</p> */
const [s, a, b] = [0,0,0].map(() => { const d = new Y.Doc(); Y.applyUpdate(d, Y.encodeStateAsUpdate(base)); return d })
// A edits p2
const svA = Y.encodeStateVector(a); /* insert into p2 */ const updA = Y.encodeStateAsUpdate(a, svA)
Y.applyUpdate(b, updA)
// B edits p1 only
const svB = Y.encodeStateVector(b); /* insert into p1 */ const updB = Y.encodeStateAsUpdate(b, svB)
Y.applyUpdate(s, updB)           // updA was "deleted" on the server
expect(s.store.pendingStructs).toBeNull()   // → no recovery triggered

to consider: a costly query every 5 minutes, and unbounded deletes on the first run

A full table read every 5 minutes isn't new. The PR adds a second pass with a join and GROUP BY, and that pass can't stop early the way the orphan query's LIMIT 1000 can.

  • First run: for each document, deleteBeforeVersion deletes every old saved step in one statement (StepMapper.php:119-126), with no batch limit and no time limit. A busy document with millions of rows means one long transaction, lock waits, and lag on replicas or Galera nodes.
  • New detail: documents without a state file still match the query on every run, and DocumentService.php:527-531 skips them. So they are fetched, and their app data looked up, again every 5 minutes forever.
  • Steady state: because of lte (finding #4), once 2 more old steps age past the cutoff, a document matches again (HAVING > 200). Documents in active use are reprocessed every 5 minutes, a few rows at a time.

Reproduction:

  1. Seed about 10M steps across about 10k documents, with timestamp older than 7 days and last_saved_version set to the newest step.
  2. Run EXPLAIN on the SQL from findDocumentsWithSavedSteps. Expect a range or full read of text_steps, a row lookup for timestamp, and a temporary table for the GROUP BY.
  3. Run Cleanup with --force-execute and watch the slow log for single DELETE statements that remove 100k+ rows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review AI assisted This PR contains AI-assisted commits backport-request bug Something isn't working

Projects

Status: 🏗️ In progress

Development

Successfully merging this pull request may close these issues.

oc_text_steps seems to not be cleanupped

3 participants