marctorrelles · Nov 4, 2026 · Nov 4, 2026 · Nov 4, 2026 · Nov 4, 2026
diff --git a/worker/CLAUDE.md b/worker/CLAUDE.md
@@ -16,12 +16,12 @@ Cloudflare Worker: Hono routes + OAuth + provider clients + the job engine. Test
 | `spotify-demo.ts` | The built-in demo library (two playlists, five likes, an album, an artist; SVG data-URI covers). `Spotify.get` serves it instead of the API when the access token is `DEMO_ACCESS`; `/auth/spotify/start` hands out those tokens when the client id equals the `REVIEW_CODE` secret. Reviewers and demos only; the YouTube side stays real. |
 | `spotify.ts` | PKCE URL/exchange/refresh (no client secret), `Spotify` client with transparent refresh + `onRefresh` persistence, 30 s timeout → `SpotifyError(429,'timeout')`, `toTrack` (handles `item` and legacy `track` wrappers, local files, episodes, null entries). |
 | `google.ts` | Device-code flow (`deviceCode`, `pollDevice` → pending/slow_down/denied/expired/ok) and `refreshGoogle`. |
-| `innertube.ts` | One `InnerTube` class, three transports: `music()` anonymous on music.youtube.com, walking `MUSIC_PLAN` (`ANDROID_MUSIC`, `IOS_MUSIC`, Android, iOS, `WEB_REMIX`) with growing pauses (`retryDelayMs`, 250 ms × attempt; tests pass 0) and a `music_retry` log line per failed attempt; only the last failure surfaces as a throttle. From Cloudflare's egress the web client gets the abuse page or a 404 on every call and the app clients on a request here and there (measured 2026-09-03: Android 8/10, iOS 5/5, web 0/5), search songs/videos/albums/artists with parsers for both item shapes (`musicResponsiveListItemRenderer` and the Android `musicTwoColumnItemRenderer`, which carries no album and no artist links); the album's `OLAK5uy_` id comes from the search result's play overlay on both shapes, `albumPlaylistId()` (web-client browse, `microformat.urlCanonical`) only when it is missing; `tv()` `TVHTML5` on www.youtube.com with the bearer token (`addPlaylistItems`, `like`, `likePlaylist` = save album, `subscribe`); `data()` Data API v3 (`createPlaylist` = `playlists.insert`, `playlistVideoIds`/`likedVideoIds` = `playlistItems.list`, `LL` = liked videos). Errors: 401 → `AuthError`; 429/5xx/HTML-200/403-HTML-abuse-page/timeout → `ThrottleError`; Data API `quotaExceeded` → `ThrottleError` with `retryAfterMs` until midnight Pacific; other Data API 403 → `AuthError`. |
+| `innertube.ts` | One `InnerTube` class, three transports: `music()` anonymous on music.youtube.com, walking `MUSIC_PLAN` (`ANDROID_MUSIC`, `IOS_MUSIC`, Android, iOS, `WEB_REMIX`) with growing pauses (`retryDelayMs`, 250 ms × attempt; tests pass 0) and a `music_retry` log line per failed attempt; only the last failure surfaces as a throttle. From Cloudflare's egress the web client gets the abuse page or a 404 on every call and the app clients on a request here and there (measured 2026-09-03: Android 8/10, iOS 5/5, web 0/5), search songs/videos/albums/artists with parsers for both item shapes (`musicResponsiveListItemRenderer` and the Android `musicTwoColumnItemRenderer`, which carries no album and no artist links); the album's `OLAK5uy_` id comes from the search result's play overlay on both shapes, `albumPlaylistId()` (web-client browse, `microformat.urlCanonical`) only when it is missing; `tv()` `TVHTML5` on www.youtube.com with the bearer token (`addPlaylistItems`, `like`, `likePlaylist` = save album, `subscribe`); `data()` Data API v3 (`createPlaylist` = `playlists.insert`, `playlistVideoIds`/`likedVideoIds` = `playlistItems.list`, `LL` = liked videos). Errors: 401 → `AuthError`; 429/5xx/HTML-200/403-HTML-abuse-page/timeout → `ThrottleError`; Data API `quotaExceeded` → `ThrottleError` with `retryAfterMs` until midnight Pacific; Data API 404 `playlistNotFound` → `ThrottleError` (a playlist created a second earlier, or its next page, reads as 404 until Google catches up); other Data API 403 → `AuthError`. |
 | `match.ts` | `stripFeat`, `buildQuery` (primary artist + title, no `&`), `cacheKey` (`artist|title`, NFKD, lowercase, punctuation collapsed), Dice-bigram `similarity`, `score` (title, artist, duration ×5, album ×0.3: the album term only breaks ties between same-title candidates), `pickBest` with the confidence gate (title ≥ 0.6, artist ≥ 0.5, duration Δ ≤ 15 s) and a plausibility gate for the review list's "closest" suggestion (title ≥ 0.5 and artist ≥ 0.3, or title ≥ 0.8; below that the row is `no_match` with search only, because "Use closest" on a 0.2-similar title adds a wrong song). `stripFeat` also drops Beatport's "Original Mix" suffix. Calibrated 2026-09-03 with `pnpm calibrate` (`worker/scripts/calibrate-match.ts`, live search against the author's accepted Python-tool picks, `CALIB_CACHE=file` replays offline): 100 tracks, 89 same id, 93 same song, confident 89 of 100 at 96.6% same-song precision; the misses are variant recordings (a "(Small Mix)"), never a different song. |
 | `csv.ts` | `toCsv` with BOM, CRLF, RFC quoting. |
 | `job-do.ts` | `JobDO`: the transfer engine (below). |
 | `stats-do.ts` | `StatsDO`: global counters for the landing page (`add`, `get` → tracksMoved, jobs, matchRate, medianMinutes over runs ≥ 100 tracks). |
-| `routes-validate.ts` | `validateSelection` for `POST /api/jobs` (Spotify id regex, length caps, 500/2000/2000 item caps). |
+| `routes-validate.ts` | `validateSelection` for `POST /api/jobs` (drops duplicate ids, which Spotify's paged lists can return and which would collide as JobDO item keys; Spotify id regex, length caps, 500/2000/2000 item caps). |
 
 `scripts/` holds one-off Node scripts run from the repo root with `pnpm spike:innertube` / `pnpm spike:spotify` (tsx is a root devDependency; outside the worker typecheck): `spike-innertube.ts`, `spike-spotify.ts` record fixtures with real credentials.
 
@@ -38,6 +38,7 @@ Invariants:
 - Verify (D15): after writes, read the playlist (and `LL` for `liked`) back through the Data API; mark missing tracks `redo` and re-drive; up to `MAX_VERIFY_PASSES = 3`, then `write_failed` review items.
 - Collapsed matches (D16): if a track's best video is already used by another track in the same item, it becomes a `duplicate_match` review item with a suggestion, never silently merged.
 - Liked songs (D18): a private "Liked Songs" mirror playlist plus individual likes (oldest first).
+- Full playlists: YouTube caps a playlist at 5,000 songs (`PLAYLIST_MAX`; one more add answers `FAILED_PRECONDITION Maximum playlist size exceeded`). `writeBatch` fills the current part up to the cap, then creates `<name> (2)`, `(3)`… (50 Data API units each; the Liked Songs mirror too, likes are uncapped). `track.yt_pl` records the part a song went into; verify reads every part and the re-drive adds a song back to its own part. `item.yt_id` is the newest part. Jobs older than the column get it in the constructor, backfilled from `item.yt_id`.
 - Tokens: sealed with `TOKEN_SECRET`; Spotify tokens wiped at finish, YouTube token kept 24 h for review actions (`disconnect()` wipes early), storage deleted 7 days after finish.
 - Match cache: KV `m1:<cacheKey>` → videoId, 180-day TTL, confident matches only.
 - RPC methods return values (`start()` returns `{ok:false,error}`) instead of throwing: a throw inside an RPC is also logged by workerd as an uncaught exception.

diff --git a/worker/src/innertube.ts b/worker/src/innertube.ts
@@ -326,6 +326,9 @@ export class InnerTube {
     if (r.status === 403 && /rateLimit/i.test(reason)) throw new ThrottleError(403, reason);
     if (r.status === 403) throw new AuthError(reason);
     if (r.status === 429 || r.status >= 500) throw new ThrottleError(r.status, reason);
+    // Measured: reading a playlist created a second earlier, or its second page, can 404 until Google's replicas catch up.
+    // ponytail: a playlist the user deleted mid-transfer also backs off until JOB_TIMEOUT_MS; fail the item if that shows up.
+    if (r.status === 404 && reason === 'playlistNotFound') throw new ThrottleError(404, reason);
     throw new Error(
       `data api ${path.split('?')[0]}: ${r.status} ${reason || String(j?.error?.message ?? '').slice(0, 120)}`,
     );

diff --git a/worker/src/job-do.ts b/worker/src/job-do.ts
@@ -36,6 +36,7 @@ const TICK_BUDGET_MS = 50_000; // wall-clock per alarm; CPU use is tiny, network
 const RECENT = 12; // live feed depth
 const CONCURRENCY = 2; // ponytail: per job. If logs show shared-IP throttling across jobs, add a global limiter DO.
 const WRITE_BATCH = 100;
+const PLAYLIST_MAX = 5_000; // YouTube's cap: one more add answers FAILED_PRECONDITION "Maximum playlist size exceeded"
 const MAX_VERIFY_PASSES = 3; // measured: one pass fixed all 346 silent no-op likes; three is generous
 const MAX_TRACKS = 25_000;
 const JOB_TIMEOUT_MS = 24 * 3600_000;
@@ -60,7 +61,7 @@ CREATE TABLE IF NOT EXISTS track (
   id INTEGER PRIMARY KEY AUTOINCREMENT, item_id TEXT NOT NULL, pos INTEGER NOT NULL, spotify_id TEXT,
   name TEXT NOT NULL, artists TEXT NOT NULL, album TEXT, duration_ms INTEGER, is_local INTEGER NOT NULL DEFAULT 0,
   status TEXT NOT NULL DEFAULT 'pending', video_id TEXT, reason TEXT, suggestion TEXT, resolution TEXT,
-  liked INTEGER NOT NULL DEFAULT 0, redo INTEGER NOT NULL DEFAULT 0, collides_with TEXT
+  liked INTEGER NOT NULL DEFAULT 0, redo INTEGER NOT NULL DEFAULT 0, collides_with TEXT, yt_pl TEXT
 );
 CREATE INDEX IF NOT EXISTS track_item_status ON track(item_id, status, pos);
 CREATE INDEX IF NOT EXISTS track_status ON track(status);`;
@@ -115,6 +116,7 @@ interface TrackRow {
   liked: number;
   redo: number;
   collides_with: string | null;
+  yt_pl: string | null;
 }
 const ids = (rows: { id: number }[]) => rows.map(() => '?').join(',');
 const REASON_TEXT: Record<string, string> = {
@@ -136,6 +138,13 @@ export class JobDO extends DurableObject<Env> {
   constructor(ctx: DurableObjectState, env: Env) {
     super(ctx, env);
     for (const stmt of SCHEMA.split(';')) if (stmt.trim()) this.sql.exec(stmt);
+    // jobs created before playlists could continue past PLAYLIST_MAX: everything moved so far went into the item's one playlist
+    if (!this.sql.exec("SELECT 1 FROM pragma_table_info('track') WHERE name = 'yt_pl'").toArray().length) {
+      this.sql.exec('ALTER TABLE track ADD COLUMN yt_pl TEXT');
+      this.sql.exec(
+        "UPDATE track SET yt_pl = (SELECT yt_id FROM item WHERE item.id = track.item_id) WHERE status = 'moved'",
+      );
+    }
   }
   private job(): JobRow | null {
     try {
@@ -423,9 +432,10 @@ export class JobDO extends DurableObject<Env> {
       if (item.kind === 'liked') await yt.like(videoId);
       else await yt.addPlaylistItems(item.yt_id!, [videoId]);
       this.sql.exec(
-        "UPDATE track SET status = 'moved', video_id = ?, resolution = ? WHERE id = ?",
+        "UPDATE track SET status = 'moved', video_id = ?, resolution = ?, yt_pl = ? WHERE id = ?",
         videoId,
         a.action,
+        item.yt_id, // a liked song resolved mid-write is missing from the mirror; verify adds it there, as before
         trackId,
       );
     } else {
@@ -687,9 +697,23 @@ export class JobDO extends DurableObject<Env> {
     };
   }
   private async writeBatch(item: ItemRow, yt: InnerTube): Promise<void> {
-    if (!item.yt_id) {
+    const room = item.yt_id
+      ? PLAYLIST_MAX -
+        (this.sql.exec('SELECT COUNT(*) AS n FROM track WHERE yt_pl = ?', item.yt_id).one() as { n: number }).n
+      : 0;
+    const waiting = this.sql
+      .exec("SELECT 1 FROM track WHERE item_id = ? AND status = 'matched' LIMIT 1", item.id)
+      .toArray().length;
+    if (!item.yt_id || (room <= 0 && waiting)) {
       // D18: liked songs get a private "Liked Songs" mirror playlist (order preserved, shareable, verifiable) plus the individual likes below.
-      const title = item.kind === 'liked' ? 'Liked Songs' : item.name;
+      // Past PLAYLIST_MAX the rest continues in "<name> (2)", "(3)"…; every part costs another 50 Data API units.
+      const parts = (
+        this.sql
+          .exec('SELECT COUNT(DISTINCT yt_pl) AS n FROM track WHERE item_id = ? AND yt_pl IS NOT NULL', item.id)
+          .one() as { n: number }
+      ).n;
+      const name = item.kind === 'liked' ? 'Liked Songs' : item.name;
+      const title = item.yt_id ? `${name} (${parts + 1})` : name;
       const description =
         item.kind === 'liked' ? 'Your Spotify liked songs, moved with Sideload.' : (item.description ?? '');
       const ytId = await yt.createPlaylist(title, description, item.is_public ? 'PUBLIC' : 'PRIVATE');
@@ -705,15 +729,19 @@ export class JobDO extends DurableObject<Env> {
       .exec(
         "SELECT id, video_id FROM track WHERE item_id = ? AND status = 'matched' ORDER BY pos ASC LIMIT ?",
         item.id,
-        WRITE_BATCH,
+        Math.min(WRITE_BATCH, room),
       )
       .toArray() as { id: number; video_id: string }[];
     if (batch.length) {
       await yt.addPlaylistItems(
         item.yt_id,
         batch.map((b) => b.video_id),
       );
-      this.sql.exec(`UPDATE track SET status = 'moved' WHERE id IN (${ids(batch)})`, ...batch.map((b) => b.id));
+      this.sql.exec(
+        `UPDATE track SET status = 'moved', yt_pl = ? WHERE id IN (${ids(batch)})`,
+        item.yt_id,
+        ...batch.map((b) => b.id),
+      );
       await this.note(
         'add',
         `Added ${batch.length} song${batch.length === 1 ? '' : 's'} to "${item.kind === 'liked' ? 'Liked Songs' : item.name}"`,
@@ -743,12 +771,13 @@ export class JobDO extends DurableObject<Env> {
    */
   private async verifyItem(item: ItemRow, yt: InnerTube): Promise<void> {
     const moved = this.sql
-      .exec("SELECT id, video_id, liked FROM track WHERE item_id = ? AND status = 'moved'", item.id)
-      .toArray() as { id: number; video_id: string; liked: number }[];
+      .exec("SELECT id, video_id, liked, yt_pl FROM track WHERE item_id = ? AND status = 'moved'", item.id)
+      .toArray() as { id: number; video_id: string; liked: number; yt_pl: string }[];
     const finishItem = () => this.sql.exec("UPDATE item SET status = 'done' WHERE id = ?", item.id);
     if (!moved.length) return void finishItem();
-    const inPlaylist = await yt.playlistVideoIds(item.yt_id!);
-    const missingAdds = moved.filter((t) => !inPlaylist.has(t.video_id));
+    const inPlaylist = new Map<string, Set<string>>(); // one read per part
+    for (const pl of new Set(moved.map((t) => t.yt_pl))) inPlaylist.set(pl, await yt.playlistVideoIds(pl));
+    const missingAdds = moved.filter((t) => !inPlaylist.get(t.yt_pl)!.has(t.video_id));
     const likedSet = item.kind === 'liked' ? await yt.likedVideoIds() : null;
     const missingLikes = likedSet ? moved.filter((t) => t.liked && !likedSet.has(t.video_id)) : [];
     const passes = item.verify_passes + 1;
@@ -793,16 +822,17 @@ export class JobDO extends DurableObject<Env> {
     });
   }
   private async redrive(item: ItemRow, yt: InnerTube): Promise<void> {
-    const adds = this.sql
+    const firstPart = this.sql
       .exec(
-        'SELECT id, video_id FROM track WHERE item_id = ? AND redo = 1 ORDER BY pos ASC LIMIT ?',
+        'SELECT id, video_id, yt_pl FROM track WHERE item_id = ? AND redo = 1 ORDER BY yt_pl, pos LIMIT ?',
         item.id,
         WRITE_BATCH,
       )
-      .toArray() as { id: number; video_id: string }[];
+      .toArray() as { id: number; video_id: string; yt_pl: string }[];
+    const adds = firstPart.filter((a) => a.yt_pl === firstPart[0]!.yt_pl); // each song goes back into the part it was missing from
     if (adds.length) {
       await yt.addPlaylistItems(
-        item.yt_id!,
+        adds[0]!.yt_pl,
         adds.map((a) => a.video_id),
       );
       this.sql.exec(`UPDATE track SET redo = 0 WHERE id IN (${ids(adds)})`, ...adds.map((a) => a.id));

diff --git a/worker/src/routes-validate.ts b/worker/src/routes-validate.ts
@@ -3,11 +3,13 @@ import type { Selection } from '@shared/types';
 import { HttpError } from './http';
 const SPOTIFY_ID = /^[A-Za-z0-9]{22}$/;
 const str = (v: unknown, max: number) => typeof v === 'string' && v.length <= max;
+// Spotify's paged lists can return the same entry twice (seen in production: a duplicate playlist id), and an id is a JobDO item key.
+const uniq = (v: unknown): any[] => (Array.isArray(v) ? [...new Map(v.map((x) => [x?.id, x])).values()] : []);
 export function validateSelection(b: unknown): Selection {
   const o = (b ?? {}) as Record<string, unknown>;
-  const playlists = Array.isArray(o.playlists) ? o.playlists : [];
-  const albums = Array.isArray(o.albums) ? o.albums : [];
-  const artists = Array.isArray(o.artists) ? o.artists : [];
+  const playlists = uniq(o.playlists);
+  const albums = uniq(o.albums);
+  const artists = uniq(o.artists);
   // Each new playlist costs 50 Data API units against a 10,000/day project quota, so ~200 a day is the real ceiling;
   // past it the job parks until midnight Pacific and then dies on JOB_TIMEOUT_MS. Fail here instead, hours earlier.
   // ponytail: one shared project quota, so two big transfers on one day still collide. Raise to 500 once the quota

diff --git a/worker/test/innertube.test.ts b/worker/test/innertube.test.ts
@@ -273,4 +273,10 @@ describe('client', () => {
       .reply(403, { error: { code: 403, errors: [{ reason: 'forbidden' }] } });
     await expect(new InnerTube('T').playlistVideoIds('PLx')).rejects.toBeInstanceOf(AuthError);
   });
+  it('maps playlistNotFound on a read → ThrottleError (a fresh playlist reads as 404 for a moment)', async () => {
+    data()
+      .intercept({ path: (p) => p.startsWith('/youtube/v3/playlistItems') })
+      .reply(404, { error: { code: 404, errors: [{ reason: 'playlistNotFound' }] } });
+    await expect(new InnerTube('T').playlistVideoIds('PLx')).rejects.toBeInstanceOf(ThrottleError);
+  });
 });
diff --git a/worker/test/job-do.test.ts b/worker/test/job-do.test.ts
@@ -342,6 +342,47 @@ describe('JobDO', () => {
     });
   });
 
+  it('continues a full playlist in "<name> (2)" and verifies each part', async () => {
+    // No Spotify interceptor: the first native tick backs off on the un-mocked fetch; seed a playlist at YouTube's cap meanwhile.
+    await stub.start(payload(id));
+    await vi.waitFor(() =>
+      runInDurableObject(stub, (_, state) => {
+        if (!state.storage.sql.exec('SELECT attempt FROM job').one().attempt) throw new Error('first tick not done');
+      }),
+    );
+    await runInDurableObject(stub, (_, state) => {
+      const sql = state.storage.sql;
+      sql.exec("UPDATE item SET fetched = 1, status = 'writing', yt_id = 'PL1'");
+      sql.exec(
+        `WITH RECURSIVE n(i) AS (SELECT 0 UNION ALL SELECT i + 1 FROM n WHERE i < 4999)
+         INSERT INTO track (item_id, pos, name, artists, status, video_id, yt_pl)
+         SELECT (SELECT id FROM item), i, 'x', 'y', 'moved', printf('v%010d', i), 'PL1' FROM n`,
+      );
+      sql.exec(
+        "INSERT INTO track (item_id, pos, name, artists, status, video_id) SELECT id, 5000, 'Xtal', 'Aphex Twin', 'matched', 'abcdefghijk' FROM item",
+      );
+      sql.exec('UPDATE job SET attempt = 0, throttled_until = 0');
+    });
+    data()
+      .intercept({ ...CREATE, body: (b) => JSON.parse(String(b)).snippet.title === 'Deep Focus (2)' })
+      .reply(200, { id: 'PL2' });
+    tv()
+      .intercept({ ...EDIT, body: (b) => JSON.parse(String(b)).playlistId === 'PL2' })
+      .reply(200, captureAdds);
+    data()
+      .intercept({ ...READBACK, path: (p) => p.includes('playlistId=PL1') })
+      .reply(200, pageOf(Array.from({ length: 5000 }, (_, i) => `v${String(i).padStart(10, '0')}`)));
+    data()
+      .intercept({ ...READBACK, path: (p) => p.includes('playlistId=PL2') })
+      .reply(200, () => pageOf(added));
+    expect(await runDurableObjectAlarm(stub)).toBe(true);
+    await settle(stub);
+    const v = (await stub.view())!;
+    expect(v.status).toBe('done');
+    expect(v.totals).toMatchObject({ moved: 5001, review: 0 });
+    expect(added).toEqual(['abcdefghijk']);
+  });
+
   it('reads the built-in demo library when the job carries demo tokens', async () => {
     const p = payload(id);
     p.spotify = demoTokens('deadbeef'.repeat(4));