Skip to content

fix(H-11): /health carries the deployed commit, and both verifiers check it - #454

Open
vjvarada wants to merge 4 commits into
mainfrom
fix/cc-metorite-host-collision
Open

fix(H-11): /health carries the deployed commit, and both verifiers check it#454
vjvarada wants to merge 4 commits into
mainfrom
fix/cc-metorite-host-collision

Conversation

@vjvarada

Copy link
Copy Markdown
Contributor

Why

CommandCenter is not down — it was replaced in place. A forked product
(Metorite) was deployed onto the CommandCenter VPS and has been serving
commandcenter.fracktal.in since 2026-08-25, under the title Metorite Control
Plane
. The owner's symptom was "none of the apps are there, I can't reach my
email"
.

Nothing is wrong with the apps, and nothing in this repo needs reverting
app visibility resolves from grants × feature_catalog (unioned with a code
mirror) in /admin/me, with no subscription, entitlement or tenancy gate
anywhere on that path. The multi-tenancy work did not hide anything.

What failed is that every verifier we own asked a liveness question, and a
different product answers those perfectly:

verifier what it asked verdict during the takeover
vps-health.yml "any HTTP response means the stack is serving" (its own comment) 🟢 green, hourly, for two days
deploy.yml verify() /health 200 and workbench 2xx/3xx 🟢 would bless a deploy that shipped nothing

Liveness is not identity. This PR adds the second question.

What changed

  • /health carries sha — the commit the process is serving, from
    ACB_GIT_SHA or the checkout. Cached, and never fatal: the on-box watchdog
    restarts services off this endpoint, so an identity lookup that could raise
    would turn "I can't tell you which commit" into "the box is dead".
    A SHA rather than a product name, deliberately — a rebranded fork
    inherits every string constant we could put there, but cannot inherit our
    commit ids.
  • deploy.yml asks "did MY commit ship?" (served sha == GITHUB_SHA).
    A foreign stack, a gateway that never restarted, and a mid-apply SSH timeout
    all fail that question; all three passed the old one.
  • vps-health.yml asks "is that OUR history at all?"git cat-file -e
    against a full checkout, plus the workbench <title> as a second signal that
    still works on boxes predating the endpoint. Either reading foreign is an
    outage.
  • Alert wording generalised. "VPS unreachable" misdescribes a takeover —
    which is emphatically not an outage you restart your way out of, since
    restarting would only restart the other product.

Verification

Run against the live host, not a fixture:

IDENTITY=unverified BRAND=foreign HEALTHY=0
- brand FOREIGN "Metorite Control Plane"
- identity UNVERIFIED and brand not ours

The old logic returned healthy for that same host. And the converse — a check
that always alarms would be no better:

case result
genuine box on new code (sha resolves) ✅ healthy
genuine box predating the endpoint (no sha, correct title) ✅ healthy + nudge to redeploy
the actual takeover (foreign sha / foreign title) 🔴 unhealthy

5 passed on the new suite, 25 passed including the gateway auth suites,
both workflow YAMLs parse, and ruff findings on main.py are unchanged at 18
(no new lint debt).

H-11 repaired

Its Check named /versionwhich CommandCenter does not have.
tasks_lens, a field in the response the box returns, appears nowhere in this
repository: /version is a Metorite route. The Check would have read "still
pending" forever after a perfect recovery, when the route simply 404s. Now
keyed on the workbench title and /health's sha.

Also re-measured today and recorded:

  1. The bleed has stopped — Metorite deploys to its own box now
    (187.127.172.200, serving today's HEAD); .143 has been frozen since
    2026-08-26T08:05Z.
  2. 🔴 A new risk: CommandCenter's HOSTINGER_HOST was changed
    2026-08-25T14:22Z and secret values are unreadable — so which box we
    deploy to is unknown
    . If it was repointed at .200, re-running our deploy
    clobbers Metorite's live box: the same accident in the other direction.
    Reading both repos' values is now step 1 of the runbook.
  3. ⚠️ R1 collision grew 188 → 192 while we sit at 176.
  4. ⚠️ Metorite's 190_gtd_retirement_drop.sql does DROP TABLE gtd_items
    / gtd_waiting (the tasks tables) and merged inside the window when
    .143 was deploying Metorite — but it is triple-guarded and inert unless
    deliberately armed. Data loss expected zero; the confirming queries are in
    the runbook, because that is read from the migration text and not yet from
    the box.

Not in this PR — owner-gated (work_plan.md §6)

Taking the box back: VPS reach, deploy, cutover. The runbook is H-11.
This PR is the part that makes the recovery verifiable by evidence rather
than by a green job
(CLAUDE.md §3.8) — which the old /version-based Check
could not do.

🤖 Generated with Claude Code

vjvarada and others added 4 commits August 26, 2026 01:03
…CommandCenter

The owner reported "none of the apps are there, I can't reach my email".
It is not a grant, feature-catalog or RLS failure: commandcenter.fracktal.in
and api.commandcenter.fracktal.in both resolve to 187.127.179.143 (srv1747539,
the original CommandCenter box) and that box serves Metorite commit 0b6a0c6f
(Hathi-Labs/Metorite PR #103) under the title "Metorite Control Plane".

Metorite already has its own box — srv1914284 / 187.127.172.200, created
2026-08-19, serving api.metorite.com and app.metorite.com healthily — so the
old box was never needed by it.

Both repos declare the same compose project `acb` and database `acb`, so
Metorite has been running against CommandCenter's own Postgres. Its ladder is
identical through 176 and adds 177-188; all twelve were scanned and none carry
DROP TABLE / DROP COLUMN / TRUNCATE / DELETE FROM, so expected data loss is
zero — inferred from the migration text, not yet verified on the box.

Recovery is OWNER-GATE throughout (work_plan.md §6: VPS reach, deploy, cutover),
so this records the diagnosis, the evidence that re-derives it, and the ordered
steps rather than performing any of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erifiers to ask

A forked product (Metorite) was deployed onto the CommandCenter box and served
`commandcenter.fracktal.in` for two days. The owner's symptom was "none of my
apps are there". Nothing was wrong with the apps — every verifier we own asked a
LIVENESS question, and a different product answers those perfectly:

  * vps-health.yml — "any HTTP response means the stack is serving" (its own
    comment). Green hourly, throughout.
  * deploy.yml verify() — `/health` 200 + workbench 2xx/3xx. The foreign stack
    satisfied both, so a deploy that shipped nothing would have gone green too.

So: give `/health` an identity and make both verifiers check it.

- `/health` now carries `sha` — the commit the process is serving, from
  `ACB_GIT_SHA` or the checkout, cached, and never fatal (the on-box watchdog
  restarts services off this endpoint). A SHA rather than a product name on
  purpose: a rebranded fork inherits our string constants but cannot inherit
  our commit ids.
- deploy.yml asks "did MY commit ship?" — served sha == GITHUB_SHA. A foreign
  stack, a gateway that never restarted, and a mid-apply SSH timeout all fail
  that; all three passed the old check.
- vps-health.yml asks "is that OUR history at all?" — `git cat-file -e` on the
  served sha, plus the workbench <title> as a second signal that works on boxes
  predating the endpoint. Verified against the live host: the new logic returns
  HEALTHY=0 / brand FOREIGN "Metorite Control Plane" where the old returned
  healthy. A genuine box, new or old, still passes.
- Alert wording generalised — "VPS unreachable" misdescribes a takeover, which
  is not an outage you restart your way out of.

H-11 re-measured and its Check repaired: it named `/version`, which
CommandCenter does not have (`tasks_lens` appears nowhere in this repo) — it is
a Metorite route, so the Check would have read "still pending" forever after a
perfect recovery. Also recorded: the bleed has stopped (Metorite moved to its
own box; .143 frozen since 2026-08-26), a NEW risk that deploying CommandCenter
now could clobber Metorite's live box (our HOSTINGER_HOST was changed
2026-08-25 and its value is unreadable), the R1 collision grew 188 → 192, and
Metorite's migration 190 does DROP the tasks tables but is triple-guarded and
inert unless deliberately armed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, not used

Root credentials for 187.127.179.143 were pasted into a session again on
2026-08-28, while trying to unblock the H-11 recovery. Refused and never used,
same as 2026-08-14 — but a refusal protects the box, it does not un-disclose
the secret.

The recurrence is the finding: it happened twice for the same reason, an
owner-gated repair that felt urgent enough to hand over the password. That
recurs while root-password auth is possible at all, so H-3 now names key-only
auth as the fix rather than "rotate and be careful" — a pasted password is
harmless once it cannot authenticate.

Adds a Check that needs no secret (does the box still offer a password prompt),
and the lockout warning on the sshd change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cord what bit

`commandcenter.fracktal.in` serves `CommandCenter Control Plane` again, the
Metorite-only `/version` route 404s, all three services are active, and the data
is intact (816 tasks, 14,938 emails). Verified by evidence, not by the deploy
log's "Deployment complete" (CLAUDE.md §3.8).

What was actually wrong, and what it cost:

- The box's `origin` was `Hathi-Labs/Metorite`, and `acb-pull.timer` — a unit
  titled "Metorite pull-based delivery (apply origin/release)" — was applying it
  every 5 minutes. Re-running OUR deploy would have redeployed THEIRS. Timer
  stopped and disabled first, then the remote repointed.
- The restore then failed on `144_crm.sql`, which is the R1 cross-repo hazard
  arriving in person: `apply_migrations.sh` re-applies any migration whose
  sha256 drifted, Metorite's rebrand had changed comment text, and our seed
  INSERT hit a NOT NULL that their tenancy migrations added. The three drifted
  files (144, 146, 157) differ ONLY in the words "CommandCenter"/"Metorite" —
  zero DDL, zero data — so the ledger was reconciled to our checksums rather
  than re-running DDL that was already correct.

H-11 and H-1 deleted (both Checks now pass — no file/ledger gap remains).
H-2 marked LOST rather than pending: 171 has applied, so its baseline count is
unrecoverable, and this records that instead of inviting a different number.
H-5's dangling "requires H-1" reference repaired.

New, both measured today:
- H-12: our next migration must be 193+. Metorite's 177-192 are in OUR database.
- H-13: 🔴 re-apply-on-checksum-drift assumes every migration is idempotent.
  `56_purge_synced_done_backlog.sql` is a bare DELETE and it re-ran, purging two
  completed synced tasks (818 -> 816, re-importable per its own header). A
  one-character comment edit is enough to fire that. Needs a decision: guard the
  migrations, or make drift refuse instead of re-apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant