[twk3]

Owner: @twk3
Linear: ENG-1881

The compose half of ENG-1881. The chart's version is currents-dev/helm-charts#72; the CLI both rely on is currents-dev/currents#4558.

What and why

A release that declares a new index builds it the first time the new scheduler starts, in its startup tasks. docker compose up -d returns straight away, so the services look up while the API answers against a database that is still building. On a large instances collection that has been about an hour, and nothing tells the operator beforehand.

Between docker compose pull and docker compose up -d the new images are on disk and the old containers are still serving. That window is where the question can be asked, and where the build can be done instead:

docker compose run --rm scheduler \
  node /app/packages/mongo/dist/scripts/migrate.js index-plan --gate

Exit 0 means carry on, exit 2 means the estimated build is over ten minutes and the output names the collections and indexes, anything else means the check itself failed. The escape hatch is the same command with sync-indexes, which builds the indexes while the old containers keep serving.

Compose gets this almost for free. The hard part in the chart was reaching the next image before the upgrade — Helm needs helm template --show-only piped into kubectl apply. Here docker compose run uses DC_CURRENTS_IMAGE_TAG from .env, which the operator has already pointed at the new version by that point in every upgrade path in this guide. No new service, no compose change, no new variable.

This is weaker than the chart's version, deliberately

The chart runs this as a pre-upgrade hook, so a long build stops the upgrade. Compose has no equivalent — the operator drives each step, so this only tells them. Making it enforcing would mean wrapping the upgrade in a script and changing how people upgrade, which is a much larger change than the problem warrants.

I also kept it inline rather than adding on-prem/scripts/check-indexes.sh. The upgrade guide is inline docker compose commands throughout; check-env.sh exists because diffing env files is genuinely scripted work, and two docker compose run lines are not. Happy to add the script if you'd rather it matched the scripts/ pattern.

What the recipe depends on

Three things about the scheduler image, checked against packages/scheduler/Dockerfile and packages/mongo/package.json in the currents repo, and since confirmed by @twk3 running it against a real 2M-document instances:

• The command override works. ENTRYPOINT ["/usr/bin/tini", "--"] with the pm2 runtime as CMD means docker compose run scheduler <cmd> replaces the command only, and tini execs it. No --entrypoint needed.
• The exit code survives. tini forwards its child's exit status, so the gate's exit 2 reaches the shell.
• The path exists. @currents/mongo is a scheduler dependency, the image copies the whole /app with packages/*/dist retained, and migrate:run in that package is node ./dist/scripts/migrate.js — so /app/packages/mongo/dist/scripts/migrate.js is right. The chart uses the same path against the same image.

Correction: an earlier version of this description said the scheduler service has no depends_on. That is wrong, and @twk3 caught it — I read templates/compose.currents.yml, where it is absent, but generate-compose.sh adds it, so all three generated files have mongodb (service_healthy) and redis (service_started), both required: false. Leaving --no-deps off is still right, but for a different reason than I gave: both dependencies are already running at this point in an upgrade, so there is nothing for compose to start.

Release dependency

--gate needs currents-dev/currents#4558 in a released on-prem image. The sync-indexes half of the section works against the images shipping today — that is the part that actually saves the downtime, so this is not blocked on a release in the way the chart PR is. Draft anyway so the three land in a sensible order.

I also linked the new section from the "Database migration issues" troubleshooting entry: a long silent gap before Startup tasks complete is usually an index build rather than a stuck service, and that entry did not say so.

Checklist

• PR title is Changelog-friendly: (fix|feat|chore): title [CSR-xxx]
• User-friendly and meaningful description of the changes
• Documentation was updated

Docs-only — no compose templates, no .env.example variables, no scripts changed, so nothing in docs/configuration.md or on-prem/README.md needed updating alongside it.

🤖 Generated with Claude Code

https://claude.ai/code/session_016uVeGDCC9GMmYpp8gZQSYt

[baz-reviewer]

Baz Summary

Document how operators can estimate pending index-build costs before upgrading, then pre-build indexes while the existing services remain available. Update the upgrade and troubleshooting guidance to use the scheduler’s migrate.js index-plan --gate and sync-indexes commands and explain their outcomes.

Topics

Topic	Details
Pre-build indexes	Explain how to run sync-indexes before upgrading to avoid startup downtime, and link index-build diagnosis from the startup-task troubleshooting guidance. Modified files (1) docs/upgrading.md Latest Contributors(2) User Commit Date noreply@anthropic.com docs: check what an up... October 30, 2026 dj@currents.dev docs: restart the sche... October 14, 2026
Check index costs	Document a pre-upgrade check that estimates index build duration using the new scheduler image, explains exit codes and rate assumptions, and directs operators to continue or investigate accordingly. Modified files (1) docs/upgrading.md Latest Contributors(2) User Commit Date noreply@anthropic.com docs: check what an up... October 30, 2026 dj@currents.dev docs: restart the sche... October 14, 2026

---

  _{Activate Skill Maintainer to keep your skills up to date}

  _{This PR would have been improved with Baz Planner - Try it now}

[twk3]

Local evidence on db3246d that the new section's commands work as written.

Setup

• on-prem from this branch with the database profile (docker-compose.database.yml as docker-compose.yml), .env from ./scripts/setup.sh --env-only, and local ports moved off the defaults.
• DC_CURRENTS_IMAGE_TAG=2026-10-14-002, docker compose up -d.
• 2M instances (1.2 GB) loaded, with createdAt_1 dropped to stand in for a release that adds an index.
• The upgrade target: a scheduler built locally from currents-dev/currents#4558 at 7db5b36 (adds index-plan --gate), tagged as …/currents/on-prem/scheduler:pr4558-7db5b36. The other services were re-tagged from 2026-10-14-002. .env was then changed to DC_CURRENTS_IMAGE_TAG=pr4558-7db5b36, standing in for the pull step.

The check, as documented, while the old containers kept running:

$ docker compose run --rm scheduler node /app/packages/mongo/dist/scripts/migrate.js index-plan --gate
Index build estimated at 42 seconds. The limit is 10 minutes.
Estimated at 30.0 MB/s, the built-in default — this deployment has recorded no index builds of its own, so the real build may be several times faster.
exit=0

$ … index-plan --gate --gate-seconds 30
Index build estimated at 42 seconds, which is over the limit of 30 seconds.

  instances: 1.2 GB, about 42 seconds (createdAt_1)
  ...
exit=2

That confirms the three things the description reasoned about:

• run replaces only the command under tini;
• exit 2 reaches the shell through compose run;
• /app/packages/mongo/dist/scripts/migrate.js exists.

docker compose ps still showed scheduler and api on 2026-10-14-002, so the check ran on the new image and nothing restarted.

Building first

• docker compose run --rm scheduler … sync-indexes exited 0 in 3 s, with the old stack still up.
• The re-check printed No index builds pending. and exited 0.
• After docker compose up -d, the new scheduler's log went migrations: syncing indexes → migrations: 1 pending in 36 ms, and on to Startup tasks complete. The index already existed.

Two notes

• The description says the scheduler service has no depends_on. It has one in docker-compose.yml and in the database and full files: mongodb (service_healthy) and redis, both required: false. Leaving out --no-deps is still right, because both are running at this point in an upgrade.
• "Pass --gate-seconds <n>": with no value, or a negative value after a space, the CLI currently treats it as a 1-second limit and exits 2. That's noted on currents#4558 with a one-line fix, and the docs don't need to change.

🤖 Generated with Claude Code