fix(H-11): /health carries the deployed commit, and both verifiers check it - #454
Open
vjvarada wants to merge 4 commits into
Open
fix(H-11): /health carries the deployed commit, and both verifiers check it#454vjvarada wants to merge 4 commits into
vjvarada wants to merge 4 commits into
Conversation
…CommandCenter The owner reported "none of the apps are there, I can't reach my email". It is not a grant, feature-catalog or RLS failure: commandcenter.fracktal.in and api.commandcenter.fracktal.in both resolve to 187.127.179.143 (srv1747539, the original CommandCenter box) and that box serves Metorite commit 0b6a0c6f (Hathi-Labs/Metorite PR #103) under the title "Metorite Control Plane". Metorite already has its own box — srv1914284 / 187.127.172.200, created 2026-08-19, serving api.metorite.com and app.metorite.com healthily — so the old box was never needed by it. Both repos declare the same compose project `acb` and database `acb`, so Metorite has been running against CommandCenter's own Postgres. Its ladder is identical through 176 and adds 177-188; all twelve were scanned and none carry DROP TABLE / DROP COLUMN / TRUNCATE / DELETE FROM, so expected data loss is zero — inferred from the migration text, not yet verified on the box. Recovery is OWNER-GATE throughout (work_plan.md §6: VPS reach, deploy, cutover), so this records the diagnosis, the evidence that re-derives it, and the ordered steps rather than performing any of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erifiers to ask
A forked product (Metorite) was deployed onto the CommandCenter box and served
`commandcenter.fracktal.in` for two days. The owner's symptom was "none of my
apps are there". Nothing was wrong with the apps — every verifier we own asked a
LIVENESS question, and a different product answers those perfectly:
* vps-health.yml — "any HTTP response means the stack is serving" (its own
comment). Green hourly, throughout.
* deploy.yml verify() — `/health` 200 + workbench 2xx/3xx. The foreign stack
satisfied both, so a deploy that shipped nothing would have gone green too.
So: give `/health` an identity and make both verifiers check it.
- `/health` now carries `sha` — the commit the process is serving, from
`ACB_GIT_SHA` or the checkout, cached, and never fatal (the on-box watchdog
restarts services off this endpoint). A SHA rather than a product name on
purpose: a rebranded fork inherits our string constants but cannot inherit
our commit ids.
- deploy.yml asks "did MY commit ship?" — served sha == GITHUB_SHA. A foreign
stack, a gateway that never restarted, and a mid-apply SSH timeout all fail
that; all three passed the old check.
- vps-health.yml asks "is that OUR history at all?" — `git cat-file -e` on the
served sha, plus the workbench <title> as a second signal that works on boxes
predating the endpoint. Verified against the live host: the new logic returns
HEALTHY=0 / brand FOREIGN "Metorite Control Plane" where the old returned
healthy. A genuine box, new or old, still passes.
- Alert wording generalised — "VPS unreachable" misdescribes a takeover, which
is not an outage you restart your way out of.
H-11 re-measured and its Check repaired: it named `/version`, which
CommandCenter does not have (`tasks_lens` appears nowhere in this repo) — it is
a Metorite route, so the Check would have read "still pending" forever after a
perfect recovery. Also recorded: the bleed has stopped (Metorite moved to its
own box; .143 frozen since 2026-08-26), a NEW risk that deploying CommandCenter
now could clobber Metorite's live box (our HOSTINGER_HOST was changed
2026-08-25 and its value is unreadable), the R1 collision grew 188 → 192, and
Metorite's migration 190 does DROP the tasks tables but is triple-guarded and
inert unless deliberately armed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, not used Root credentials for 187.127.179.143 were pasted into a session again on 2026-08-28, while trying to unblock the H-11 recovery. Refused and never used, same as 2026-08-14 — but a refusal protects the box, it does not un-disclose the secret. The recurrence is the finding: it happened twice for the same reason, an owner-gated repair that felt urgent enough to hand over the password. That recurs while root-password auth is possible at all, so H-3 now names key-only auth as the fix rather than "rotate and be careful" — a pasted password is harmless once it cannot authenticate. Adds a Check that needs no secret (does the box still offer a password prompt), and the lockout warning on the sshd change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cord what bit `commandcenter.fracktal.in` serves `CommandCenter Control Plane` again, the Metorite-only `/version` route 404s, all three services are active, and the data is intact (816 tasks, 14,938 emails). Verified by evidence, not by the deploy log's "Deployment complete" (CLAUDE.md §3.8). What was actually wrong, and what it cost: - The box's `origin` was `Hathi-Labs/Metorite`, and `acb-pull.timer` — a unit titled "Metorite pull-based delivery (apply origin/release)" — was applying it every 5 minutes. Re-running OUR deploy would have redeployed THEIRS. Timer stopped and disabled first, then the remote repointed. - The restore then failed on `144_crm.sql`, which is the R1 cross-repo hazard arriving in person: `apply_migrations.sh` re-applies any migration whose sha256 drifted, Metorite's rebrand had changed comment text, and our seed INSERT hit a NOT NULL that their tenancy migrations added. The three drifted files (144, 146, 157) differ ONLY in the words "CommandCenter"/"Metorite" — zero DDL, zero data — so the ledger was reconciled to our checksums rather than re-running DDL that was already correct. H-11 and H-1 deleted (both Checks now pass — no file/ledger gap remains). H-2 marked LOST rather than pending: 171 has applied, so its baseline count is unrecoverable, and this records that instead of inviting a different number. H-5's dangling "requires H-1" reference repaired. New, both measured today: - H-12: our next migration must be 193+. Metorite's 177-192 are in OUR database. - H-13: 🔴 re-apply-on-checksum-drift assumes every migration is idempotent. `56_purge_synced_done_backlog.sql` is a bare DELETE and it re-ran, purging two completed synced tasks (818 -> 816, re-importable per its own header). A one-character comment edit is enough to fire that. Needs a decision: guard the migrations, or make drift refuse instead of re-apply. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CommandCenter is not down — it was replaced in place. A forked product
(Metorite) was deployed onto the CommandCenter VPS and has been serving
commandcenter.fracktal.insince 2026-08-25, under the title Metorite ControlPlane. The owner's symptom was "none of the apps are there, I can't reach my
email".
Nothing is wrong with the apps, and nothing in this repo needs reverting —
app visibility resolves from grants ×
feature_catalog(unioned with a codemirror) in
/admin/me, with no subscription, entitlement or tenancy gateanywhere on that path. The multi-tenancy work did not hide anything.
What failed is that every verifier we own asked a liveness question, and a
different product answers those perfectly:
vps-health.ymldeploy.ymlverify()/health200 and workbench 2xx/3xxLiveness is not identity. This PR adds the second question.
What changed
/healthcarriessha— the commit the process is serving, fromACB_GIT_SHAor the checkout. Cached, and never fatal: the on-box watchdogrestarts services off this endpoint, so an identity lookup that could raise
would turn "I can't tell you which commit" into "the box is dead".
A SHA rather than a product name, deliberately — a rebranded fork
inherits every string constant we could put there, but cannot inherit our
commit ids.
deploy.ymlasks "did MY commit ship?" (served sha ==GITHUB_SHA).A foreign stack, a gateway that never restarted, and a mid-apply SSH timeout
all fail that question; all three passed the old one.
vps-health.ymlasks "is that OUR history at all?" —git cat-file -eagainst a full checkout, plus the workbench
<title>as a second signal thatstill works on boxes predating the endpoint. Either reading foreign is an
outage.
which is emphatically not an outage you restart your way out of, since
restarting would only restart the other product.
Verification
Run against the live host, not a fixture:
The old logic returned healthy for that same host. And the converse — a check
that always alarms would be no better:
5 passedon the new suite,25 passedincluding the gateway auth suites,both workflow YAMLs parse, and ruff findings on
main.pyare unchanged at 18(no new lint debt).
H-11 repaired
Its Check named
/version— which CommandCenter does not have.tasks_lens, a field in the response the box returns, appears nowhere in thisrepository:
/versionis a Metorite route. The Check would have read "stillpending" forever after a perfect recovery, when the route simply 404s. Now
keyed on the workbench title and
/health's sha.Also re-measured today and recorded:
(
187.127.172.200, serving today's HEAD);.143has been frozen since2026-08-26T08:05Z.
HOSTINGER_HOSTwas changed2026-08-25T14:22Z and secret values are unreadable — so which box we
deploy to is unknown. If it was repointed at
.200, re-running our deployclobbers Metorite's live box: the same accident in the other direction.
Reading both repos' values is now step 1 of the runbook.
190_gtd_retirement_drop.sqldoesDROP TABLE gtd_items/
gtd_waiting(the tasks tables) and merged inside the window when.143was deploying Metorite — but it is triple-guarded and inert unlessdeliberately armed. Data loss expected zero; the confirming queries are in
the runbook, because that is read from the migration text and not yet from
the box.
Not in this PR — owner-gated (
work_plan.md§6)Taking the box back: VPS reach, deploy, cutover. The runbook is H-11.
This PR is the part that makes the recovery verifiable by evidence rather
than by a green job (CLAUDE.md §3.8) — which the old
/version-based Checkcould not do.
🤖 Generated with Claude Code