Repository navigation
fix(worker): three production errors from transfers - #4
Merged
Merged
Conversation
Spotify's paged library lists can return the same playlist twice. The duplicate id reached JobDO.start, where each id is the item primary key, and job creation failed with a UNIQUE constraint error.
Reading a playlist a second after creating it, or its next page, can answer 404 playlistNotFound until Google's replicas catch up. It was a plain error, so every verify pass that hit it was reported and counted towards the eight retries before provider_error. It is now a throttle: back off and read again.
YouTube caps a playlist at 5,000 songs; the next add answers FAILED_PRECONDITION "Maximum playlist size exceeded". It was a plain error, so the job retried it eight times and failed with provider_error, losing the rest of the transfer. writeBatch now fills a playlist up to the cap and continues in "<name> (2)", "(3)" and so on, the Liked Songs mirror included (likes themselves are uncapped). Each track records the playlist it went into, so verify reads every part and the re-drive puts a missing song back into its own part. Jobs already running get the column on load, backfilled from their single playlist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the three unresolved sideload errors in crashyard.
Playlists past 5,000 songs (`Maximum playlist size exceeded`, 18 events, two transfers failed)
YouTube caps a playlist at 5,000 songs. The next add was a plain error, so the job retried eight times and failed with `provider_error`. Now a playlist fills to the cap and continues in " (2)", "(3)" and so on, the Liked Songs mirror included; likes stay uncapped. Each track records the part it went into (`track.yt_pl`), so verify reads every part and the re-drive puts a missing song back into its own part. Running jobs get the column on load, backfilled from their single playlist. Each extra part costs 50 Data API units.
Read-back 404 `playlistNotFound` (2 events)
Reading a playlist a second after creating it, or its next page, can 404 until Google catches up. The job recovered on retry, but each hit was reported and counted towards the eight retries. It is now a throttle: back off and read again.
`UNIQUE constraint failed: item.id` (2 events)
Spotify's paged library lists can return the same entry twice. The duplicate id reached `JobDO.start`, where ids are the item primary key, and creating the transfer failed. `validateSelection` now drops duplicate ids.
Known limits
Checks
`pnpm test` (72 worker, 4 web) and `pnpm --filter worker typecheck` pass. New tests: duplicate ids, `playlistNotFound` as a throttle, and a full playlist continuing into "(2)" with both parts verified.