diff --git a/.claude/skills/audit-user-stories/SKILL.md b/.claude/skills/audit-user-stories/SKILL.md index 04cae3d6b..c2592d4c6 100644 --- a/.claude/skills/audit-user-stories/SKILL.md +++ b/.claude/skills/audit-user-stories/SKILL.md @@ -227,12 +227,12 @@ Execute the operator's chosen scope in this wave order. Each wave can run in par If the operator's chosen scope creates any new sub-issues (e.g., a shared-types consolidation story, a verification-conversion that splits one story into two), run the cap check before any writes — the same pre-flight `draft-user-stories` runs. Hitting the cap mid-repair leaves the work tree half-fixed and forces an out-of-band consolidation rollup before you can resume. - `needed` = number of new sub-issues the repair will create. If the repair only rewrites + cancels existing issues (the common case), `needed = 0` and you can skip — `save_issue(id=...)` updates don't consume cap. -- For `needed > 0`, run `uv run python scripts/check_linear_cap.py --needed --json` (one JSON line: `active`, `cap`, `buffer`, `needed`, `margin`, `required`, `ok`; 2-issue safety margin applied internally; exit code mirrors `ok`: `0` clear, `1` cap risk, `2` error). +- For `needed > 0`, run `uv run python scripts/linear/check_linear_cap.py --needed --json` (one JSON line: `active`, `cap`, `buffer`, `needed`, `margin`, `required`, `ok`; 2-issue safety margin applied internally; exit code mirrors `ok`: `0` clear, `1` cap risk, `2` error). Decision: - **`ok == true`** — proceed to Wave 1. -- **`ok == false`** — surface to the operator *before any writes*. Report the numbers from the JSON. Recommend `/linear-consolidate` (per the `project_linear_consolidation` memory) or `scripts/linear_consolidation_candidates.py` to free slots. Ask whether to (a) pause for rollup, (b) proceed accepting cap-risk and inline the un-created stories in the hand-off, or (c) trim repair scope to drop the new creations. +- **`ok == false`** — surface to the operator *before any writes*. Report the numbers from the JSON. Recommend `/linear-consolidate` (per the `project_linear_consolidation` memory) or `scripts/linear/linear_consolidation_candidates.py` to free slots. Ask whether to (a) pause for rollup, (b) proceed accepting cap-risk and inline the un-created stories in the hand-off, or (c) trim repair scope to drop the new creations. - **exit 2** — script failure (missing `LINEAR_API_KEY` in `.env`, network/API error). Surface and resolve before proceeding; don't fall back to manual MCP counting. #### Wave 1 — Duplicate cleanup diff --git a/.claude/skills/comb-collector-logs/SKILL.md b/.claude/skills/comb-collector-logs/SKILL.md index 177525973..ac9b0241b 100644 --- a/.claude/skills/comb-collector-logs/SKILL.md +++ b/.claude/skills/comb-collector-logs/SKILL.md @@ -60,7 +60,7 @@ After identifying a candidate bug in today's log, grep older rotated logs for th A traceback shows *what raised*, not *what should have caught it*. Always read the relevant file: -- For retry-related errors: read `src/alphamind/data_sources/_common.py` `_is_retryable` to confirm the exception type is covered. If `_is_retryable` returns False, the retry wrapper silently gave up — that's the bug, not the vendor error. +- For retry-related errors: read `_is_retryable` in `src/alphamind/data_sources/_common/retry.py` to confirm the exception type is covered. If `_is_retryable` returns False, the retry wrapper silently gave up — that's the bug, not the vendor error. - For per-ticker collectors: check whether the loop has try/except per ticker. No isolation means one ticker's exhausted retries kills the cycle. - For wrapped exceptions: check `__context__` and `__cause__` — fredapi wraps `urllib.HTTPError` as `ValueError(None)`, the message is lost but the context isn't. @@ -94,7 +94,7 @@ For each event group: 1. **Read the surrounding traceback** in `collector.log`. Note: full exception type, the immediate raising frame, and `__context__` if present. 2. **Cross-check operator memory** at `collector-service-ops.md` for known-transient patterns. If listed there with "should retry": - Verify retries actually engaged by checking elapsed time between cycle start (`collector=X start`) and error. Sub-second to a few seconds = retry didn't fire; tens of seconds = retry fired but exhausted. - - If retry didn't fire, the bug is in `_is_retryable` not classifying that exception as retryable. Read `src/alphamind/data_sources/_common.py:276-296` to confirm. + - If retry didn't fire, the bug is in `_is_retryable` not classifying that exception as retryable. Read `_is_retryable` in `src/alphamind/data_sources/_common/retry.py` to confirm. 3. **Read the source file from the traceback.** Confirm root cause — don't trust the traceback alone (Rule 4). 4. **Check rotated logs for recurrence** — grep the same error signature across `collector.log.YYYY-MM-DD` files. Record cadence (one-off, daily, multi-day pattern). 5. **Check git log for recent fixes** — `git log --oneline -5 -- `. A commit in the last day that matches the symptom means it was already fixed. diff --git a/.claude/skills/draft-user-stories/SKILL.md b/.claude/skills/draft-user-stories/SKILL.md index 84b19d8fc..3bd897fb4 100644 --- a/.claude/skills/draft-user-stories/SKILL.md +++ b/.claude/skills/draft-user-stories/SKILL.md @@ -106,7 +106,7 @@ Don't worry about ordering or naming conventions yet — those happen in Phase 5 Anchor the Scope sections of subsequent stories to the brief's concrete names — a story's "produces `PositionRecord` with these fields" is far more useful to a dispatched subagent than "introduces a position record". When the brief surfaces a hardest-to-reverse decision the design doc hasn't settled, add it to your Phase 6 open-decisions list. Skill invocation runs in your thread (no Agent dispatch), compatible with the no-delegation rule. Skip design mode when the feature is small, purely additive to an existing module, or already fully specified at the type level by its design doc. -**Operational runbook story.** Assess whether the feature changes production operational behavior — a service, the schedule, a port, an env var, a migration / bootstrap step, a CLI flag, a monitoring surface, or a new failure mode / gotcha. If so, add a final story that updates `scripts/RUNBOOK_production.md` (its Living-document rule keeps the runbook moving with the behavior), `blockedBy` every story whose operational change it documents so it lands last. Omit it for features with no operator-visible prod-runtime effect (pure internal logic, analysis / decision-layer changes, test-only work). +**Operational runbook story.** Assess whether the feature changes production operational behavior — a service, the schedule, a port, an env var, a migration / bootstrap step, a CLI flag, a monitoring surface, or a new failure mode / gotcha. If so, add a final story that updates the affected runbook under `docs/runbooks/` (the README's Living-document rule keeps the runbooks moving with the behavior), `blockedBy` every story whose operational change it documents so it lands last. Omit it for features with no operator-visible prod-runtime effect (pure internal logic, analysis / decision-layer changes, test-only work). ### Phase 5 — Order by dependency, name with parallelism convention @@ -180,7 +180,7 @@ Compute how many issues you're about to create: Run the cap-check script with that count: ``` -uv run python scripts/check_linear_cap.py --needed --json +uv run python scripts/linear/check_linear_cap.py --needed --json ``` The script queries the Linear GraphQL API directly (key from the repo-root `.env`), paginates the whole workspace, and prints one JSON line — e.g. `{"active": 243, "cap": 250, "buffer": 7, "needed": 12, "margin": 2, "required": 14, "ok": false}`. It applies a 2-issue safety margin internally (`required = needed + margin`), so you pass only the raw `needed` count. Exit code mirrors `ok`: `0` = clear, `1` = cap risk, `2` = error. @@ -188,7 +188,7 @@ The script queries the Linear GraphQL API directly (key from the repo-root `.env Decision: - **`ok == true` (exit 0)** — cap risk is low. Note `active` in your working memory and proceed to 7a. -- **`ok == false` (exit 1)** — surface to the operator *before any writes*. Report `active`, `buffer`, `needed`, and `required` from the JSON. Recommend running the consolidation rollup per the `project_linear_consolidation` memory and the `/linear-consolidate` skill — `uv run python scripts/linear_consolidation_candidates.py` ranks which shipped feature would free the most slots, and `scripts/check_linear_cap.py --breakdown` shows where the active issues sit. Ask whether to (a) pause for rollup, (b) proceed accepting that the cap may fire mid-drafting (you will catch the error gracefully and surface the remaining drafts inline as a hand-off), or (c) trim story count if you can identify a fold. +- **`ok == false` (exit 1)** — surface to the operator *before any writes*. Report `active`, `buffer`, `needed`, and `required` from the JSON. Recommend running the consolidation rollup per the `project_linear_consolidation` memory and the `/linear-consolidate` skill — `uv run python scripts/linear/linear_consolidation_candidates.py` ranks which shipped feature would free the most slots, and `scripts/linear/check_linear_cap.py --breakdown` shows where the active issues sit. Ask whether to (a) pause for rollup, (b) proceed accepting that the cap may fire mid-drafting (you will catch the error gracefully and surface the remaining drafts inline as a hand-off), or (c) trim story count if you can identify a fold. - **exit 2** — the script failed (missing `LINEAR_API_KEY` in `.env`, or a network/API error; the stderr message says which). Surface it to the operator and resolve the cause before proceeding — don't fall back to a manual MCP count. The cap is occasionally enforced at counts above 250 — Linear's exact threshold depends on workspace age and account state. Treat the check as early-warning, not exact predictor. If a create fails despite a pre-flight `ok == true`, surface the blocker and inline the remaining drafts in the operator hand-off. diff --git a/.claude/skills/linear-consolidate/SKILL.md b/.claude/skills/linear-consolidate/SKILL.md index 9f8c64ffe..c1eee5850 100644 --- a/.claude/skills/linear-consolidate/SKILL.md +++ b/.claude/skills/linear-consolidate/SKILL.md @@ -1,11 +1,11 @@ --- name: linear-consolidate -description: Use to consolidate AlphaMind's Linear workspace under the free-tier 250-issue cap by rolling up Done feature sub-issues into the parent Issue's body and archiving the sub-issues. Triggers on `/linear-consolidate ` and operator phrases like "free up Linear slots", "I'm hitting the Linear free-tier limit", "consolidate Done features", "roll up the X sub-issues", "clean up Linear", "the Linear workspace is full", "archive shipped sub-issues without losing history". The skill verifies parity between Linear sub-issue bodies and local archive files at `docs/_archive/implementation///`, resolves drift via shipped `src/` as tiebreaker (the local archive is the in-repo canonical record), drafts a rolled-up parent body that opens with a non-bullet paragraph (Linear's renderer drops bullets in the second list otherwise), saves it, and archives the sub-issues via `scripts/archive_linear_issues.py` after operator approval. The supporting scripts (`check_linear_cap.py`, `export_linear_issues.py`, `linear_parity_diff.py`, `archive_linear_issues.py`) hit the Linear GraphQL API directly — the Linear MCP cannot count issues, archive them, or return un-truncated bodies. Do NOT use for non-AlphaMind workspaces, in-flight features, or features without local archive copies. +description: Use to consolidate AlphaMind's Linear workspace under the free-tier 250-issue cap by rolling up Done feature sub-issues into the parent Issue's body and archiving the sub-issues. Triggers on `/linear-consolidate ` and operator phrases like "free up Linear slots", "I'm hitting the Linear free-tier limit", "consolidate Done features", "roll up the X sub-issues", "clean up Linear", "the Linear workspace is full", "archive shipped sub-issues without losing history". The skill verifies parity between Linear sub-issue bodies and local archive files at `docs/_archive/implementation///`, resolves drift via shipped `src/` as tiebreaker (the local archive is the in-repo canonical record), drafts a rolled-up parent body that opens with a non-bullet paragraph (Linear's renderer drops bullets in the second list otherwise), saves it, and archives the sub-issues via `scripts/linear/archive_linear_issues.py` after operator approval. The supporting scripts (`check_linear_cap.py`, `export_linear_issues.py`, `linear_parity_diff.py`, `archive_linear_issues.py`) hit the Linear GraphQL API directly — the Linear MCP cannot count issues, archive them, or return un-truncated bodies. Do NOT use for non-AlphaMind workspaces, in-flight features, or features without local archive copies. --- # Linear consolidation under free-tier ceiling -The operator names a Done AlphaMind feature whose Linear sub-issues should be rolled up into the parent Issue's body and then archived, freeing slots against the free-tier 250-issue cap. The output of a successful run is: the parent Issue body updated with a `## Shipped stories` index of every sub-issue, every Done sub-issue archived via `scripts/archive_linear_issues.py`, and the workspace's active (non-archived) issue count reduced by exactly the number archived. +The operator names a Done AlphaMind feature whose Linear sub-issues should be rolled up into the parent Issue's body and then archived, freeing slots against the free-tier 250-issue cap. The output of a successful run is: the parent Issue body updated with a `## Shipped stories` index of every sub-issue, every Done sub-issue archived via `scripts/linear/archive_linear_issues.py`, and the workspace's active (non-archived) issue count reduced by exactly the number archived. This skill leans on four scripts under `scripts/` that hit the Linear GraphQL API directly — the Linear MCP server can neither count issues, archive them, nor return un-truncated issue bodies: @@ -29,7 +29,7 @@ A feature name (e.g. `Configuration management`, `Collector`, `Portfolio state`) 1. **Parent ALP-ID** — via `mcp__linear-server__list_issues` filtered by `query=`, or by reading the project bullet in `docs/project-tracker.md`. 2. **Local archive path** at `docs/_archive/implementation///` — e.g. `foundation/configuration/`, `01-data-layer/collector/`, `06-risk-guardrails/breach-behavior/`. List with `ls` to confirm the directory exists and holds one `.md` per expected sub-issue. -If the operator doesn't know which feature to consolidate, run `uv run python scripts/linear_consolidation_candidates.py` — it lists every parent whose sub-issues are all Done, ranked by the slots each rollup would free. +If the operator doesn't know which feature to consolidate, run `uv run python scripts/linear/linear_consolidation_candidates.py` — it lists every parent whose sub-issues are all Done, ranked by the slots each rollup would free. If the parent isn't in `Done` status, **stop** — the feature is in flight, its sub-issues should not be archived (orchestration may still need the dependency graph). Surface to the operator and exit. @@ -61,7 +61,7 @@ This isn't optional. A pilot run found one cosmetic drift (`18 entries` vs `19 e ### 3. Archiving runs through the script, not the MCP or the UI -The Linear MCP exposes no archive or delete tool — `save_issue` only updates fields. Archiving happens via `scripts/archive_linear_issues.py`, which calls the GraphQL `issueArchive` mutation. The script is **dry-run by default**: a plain invocation prints exactly what it would archive and changes nothing; `--apply` performs the archive. Never hand the operator a manual Cmd+Delete instruction — the script is the mechanism, and the operator's role is to approve the dry-run plan before you re-run with `--apply`. +The Linear MCP exposes no archive or delete tool — `save_issue` only updates fields. Archiving happens via `scripts/linear/archive_linear_issues.py`, which calls the GraphQL `issueArchive` mutation. The script is **dry-run by default**: a plain invocation prints exactly what it would archive and changes nothing; `--apply` performs the archive. Never hand the operator a manual Cmd+Delete instruction — the script is the mechanism, and the operator's role is to approve the dry-run plan before you re-run with `--apply`. ### 4. Parity verification runs through the diff script @@ -76,7 +76,7 @@ Parity-checking 10–30 sub-issue bodies against local files is a deterministic 3. Fetch the parent body (`get_issue`) and the Done sub-issues (`list_issues parentId=ALP-X state=Done`). 4. List the local archive files (`ls`). 5. Build an ID-to-filename mapping by user-story index (the local files use `01-…`, `02-…`, `03a-…` prefixes; Linear sub-issue titles start with `01 — `, `02 — `, etc.). -6. Capture the baseline cap count: run `uv run python scripts/check_linear_cap.py` and note the `active (non-archived) issues` figure. Phase 6 re-runs the script; the drop should equal the number of sub-issues archived. +6. Capture the baseline cap count: run `uv run python scripts/linear/check_linear_cap.py` and note the `active (non-archived) issues` figure. Phase 6 re-runs the script; the drop should equal the number of sub-issues archived. ### Phase 1.5 — Materialize missing archives (when needed) @@ -85,7 +85,7 @@ If `docs/_archive/implementation///` doesn't exist, or holds few 1. Confirm the `` segment matches the existing convention — `ls docs/_archive/implementation/` first (`foundation`, `01-data-layer`, `02-distillation-layer`, `03-analysis-layer`, `04-decision-layer`, `05-execution-layer`, `06-risk-guardrails`). 2. Run the export script — it resolves the parent, fetches every sub-issue body via GraphQL (full, no truncation), and writes one `.md` per sub-issue named by user-story index (`03a — Bracket-thesis coverage cross-validator` → `03a-bracket-thesis-coverage-cross-validator.md`): - uv run python scripts/export_linear_issues.py ALP-X \ + uv run python scripts/linear/export_linear_issues.py ALP-X \ --out-dir docs/_archive/implementation// Add `--include-archived` if some sub-issues were already archived in a prior partial run. @@ -97,7 +97,7 @@ If `docs/_archive/implementation///` doesn't exist, or holds few Run the parity-diff script against the pre-existing local archive: - uv run python scripts/linear_parity_diff.py ALP-X \ + uv run python scripts/linear/linear_parity_diff.py ALP-X \ --archive-dir docs/_archive/implementation// It prints a per-story table — `MATCH`, `DIFFERS`, `LINEAR-ONLY` (a sub-issue with no local file), `LOCAL-ONLY` (a local file with no sub-issue) — followed by a unified diff for every `DIFFERS` row. Exit code is `0` when every story matches, `1` when any drift is found. @@ -150,15 +150,15 @@ Show the proposed body in a markdown code block. Wait for operator approval befo 1. Dry-run the archive to produce the plan — it changes nothing: - uv run python scripts/archive_linear_issues.py --parent ALP-X + uv run python scripts/linear/archive_linear_issues.py --parent ALP-X The script lists every Done sub-issue as `WOULD-ARCHIVE` and reports any non-Done sub-issue as `SKIP-NOT-DONE` (see "Features with active gaps" below — handled automatically). 2. Show the operator the plan and get explicit approval. 3. On approval, re-run with `--apply`: - uv run python scripts/archive_linear_issues.py --parent ALP-X --apply + uv run python scripts/linear/archive_linear_issues.py --parent ALP-X --apply -4. Recount: `uv run python scripts/check_linear_cap.py`. Compare `active` against the Phase 1 baseline — the drop should equal the number archived. If it didn't, something went wrong (a per-issue archive failed, network issue) — the script's own output flags `FAILED` rows; surface and investigate. The freed `buffer` is the headline result of the run. +4. Recount: `uv run python scripts/linear/check_linear_cap.py`. Compare `active` against the Phase 1 baseline — the drop should equal the number archived. If it didn't, something went wrong (a per-issue archive failed, network issue) — the script's own output flags `FAILED` rows; surface and investigate. The freed `buffer` is the headline result of the run. To archive a hand-picked set rather than a whole parent's Done children, use `--ids ALP-a,ALP-b,…` instead of `--parent`. @@ -176,7 +176,7 @@ Some features have non-Done sub-issues mixed into a contiguous Done range (e.g. The "To-dos" project holds standalone Done issues with no sub-issues — one-off bug fixes (e.g. ALP-266 "session.py should fail loudly…", ALP-267 "Suppress benign aclose() warning…"). These need no rollup; archive them directly with `archive_linear_issues.py --ids ALP-a,ALP-b,…` after confirming the list with the operator. -For dead-weight beyond shipped features, `uv run python scripts/linear_stale_sweep.py` finds non-archived Canceled / Duplicate issues — they count against the cap too, and can be archived directly via `--ids`. +For dead-weight beyond shipped features, `uv run python scripts/linear/linear_stale_sweep.py` finds non-archived Canceled / Duplicate issues — they count against the cap too, and can be archived directly via `--ids`. ### Parent not in Done status @@ -190,4 +190,4 @@ Skip the feature. The dependency graph is still load-bearing for `/orchestrate` ## Status checks -When the operator asks for a status check ("what have we cleaned up so far?" / "how much headroom is left?"), compute it on demand — `uv run python scripts/check_linear_cap.py --breakdown` reports the current active count, buffer to the cap, and the per-state/per-project split. Don't maintain a running tally in memory; the live count is authoritative and the scripts already produce it. +When the operator asks for a status check ("what have we cleaned up so far?" / "how much headroom is left?"), compute it on demand — `uv run python scripts/linear/check_linear_cap.py --breakdown` reports the current active count, buffer to the cap, and the per-state/per-project split. Don't maintain a running tally in memory; the live count is authoritative and the scripts already produce it. diff --git a/.claude/skills/operate-prod/SKILL.md b/.claude/skills/operate-prod/SKILL.md new file mode 100644 index 000000000..868f2d36f --- /dev/null +++ b/.claude/skills/operate-prod/SKILL.md @@ -0,0 +1,58 @@ +--- +name: operate-prod +description: Use for operational tasks on the AlphaMind production Windows machine — deploying latest main (the update loop), running a manual invocation, watching or post-morteming a scheduled invocation, restarting NSSM services, and symptom-driven triage. Triggers on operator phrases like "update prod", "deploy to prod", "run a manual invocation", "why did the 10am run fail", "restart the monitor", "the monitor looks wedged". Routes onto the canonical runbooks under docs/runbooks/ and carries the agent-side directives (autonomy gate, never-autonomously list, self-correction duty). NOT for the one-time bootstrap or genesis cutover (Operator-executed), and NOT for feedback-loop cadence work (use feedback-review / feedback-validate / feedback-retrospective). +--- + +# Operating prod + +The Operator names a prod task. Route it onto its runbook and execute from inside that +doc — canonical procedures live in `docs/runbooks/` ([map](../../../docs/runbooks/README.md)); +read the routed runbook end-to-end before acting, never from memory of it. This skill +adds only what the runbooks don't carry: routing, the agent-side guardrails, and the +duty to keep the docs true. + +## Routes + +| Task | Runbook | +|---|---| +| Update / deploy latest `main` | `docs/runbooks/update-loop.md` | +| Run a manual invocation | `docs/runbooks/invocations.md`, watch it via `docs/runbooks/monitoring.md` | +| Watch / post-mortem an invocation | `docs/runbooks/monitoring.md` | +| Restart a service | `docs/runbooks/services.md` | +| Triage a symptom | `docs/runbooks/troubleshooting.md` | +| Command-center access / passkeys / LAN | `docs/runbooks/command-center.md` | + +Named, not routed: first-time bootstrap (`docs/runbooks/bootstrap.md`) and genesis +cutover (`docs/runbooks/genesis-cutover.md`) are one-time Operator-supervised +procedures — surface them, don't execute them. Feedback-loop cadence work belongs to +the `feedback-review` / `feedback-validate` / `feedback-retrospective` skills; the prod +operational context is `docs/runbooks/feedback-loop.md`. + +## Standing directives + +- Prod is Windows: NSSM services, elevated PowerShell, repo checkout at + `$env:USERPROFILE\AlphaMind` on branch `main`, clean. +- `.env` never auto-sources (per CLAUDE.md) — load it before anything that hits a + vendor API; the CRLF-safe form is in `docs/runbooks/bootstrap.md` § 2.1. +- Single-writer discipline: write the DB only through the sanctioned CLIs and + services; ad-hoc SQL is read-only. +- Confirm the Alpaca `account_number` before treating broker output as prod evidence. + +## Autonomy gate + +Before any step that stops services, evaluate the service-stop precondition in +`docs/runbooks/update-loop.md`: market open AND open positions → halt and obtain +explicit Operator confirmation in-session; otherwise proceed. + +Never autonomously, regardless of gate state: halt-mode changes, force-close, +cancel-order, DB writes outside the sanctioned CLIs, genesis cutover. + +## Self-correction duty + +After every executed procedure, compare observed prod behavior against the runbook you +followed and against this skill. On divergence, produce the correction in the same +session: a new experiential gotcha appends to `docs/runbooks/troubleshooting.md`; a +runbook or skill inaccuracy becomes an edit to that file. Deliver corrections as a +docs-only PR from a fresh branch, then restore the checkout to clean `main` — the +update loop's pull step requires it. If pushing or opening a PR is unavailable from +prod, present the exact diff to the Operator in-session instead. diff --git a/.claude/skills/refine-issue/SKILL.md b/.claude/skills/refine-issue/SKILL.md index 012aae233..86b9c6e96 100644 --- a/.claude/skills/refine-issue/SKILL.md +++ b/.claude/skills/refine-issue/SKILL.md @@ -101,7 +101,7 @@ Rewrite the **entire** body via `save_issue`, conforming to **Operational runbook impact.** Assess whether the work changes production operational behavior — a service, the schedule, a port, an env var, a migration / bootstrap step, a CLI flag, a monitoring surface, or a new failure mode / gotcha. If so, add an acceptance criterion that the -same change updates `scripts/RUNBOOK_production.md` (its Living-document rule: the runbook moves +same change updates the affected runbook under `docs/runbooks/` (the README's Living-document rule: the runbooks move with the behavior). Skip for work with no operator-visible prod-runtime effect (pure internal logic, analysis / decision-layer changes, test-only work). diff --git a/CLAUDE.md b/CLAUDE.md index 58576fa70..97110a625 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,8 +2,10 @@ AlphaMind: autonomous LLM swing-trading system (4–72h horizon, ~60–80 US equities), currently in the paper-trading phase. This file is project-wide rules + a navigation -map. Process playbooks live in `docs/agents/`; per-package orientation lives in nested -`CLAUDE.md` files; design intent (point-in-time, **not** code-truth) in `docs/design/`. +map. Process playbooks live in `docs/agents/`; production operational runbooks in +`docs/runbooks/` (routed by the `operate-prod` skill); per-package orientation lives in +nested `CLAUDE.md` files; design intent (point-in-time, **not** code-truth) in +`docs/design/`. ## Navigation diff --git a/CONTEXT.md b/CONTEXT.md index 3844b317b..c04296686 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,8 +1,18 @@ -# AlphaMind — Execution & Broker Boundary +# AlphaMind — Canonical Vocabulary -Canonical vocabulary for how AlphaMind relates to the external broker. This is a -glossary, not a spec — it fixes the words so both the architecture conversation and the -code stay precise about *who owns which fact*. +Canonical vocabulary for AlphaMind. This is a glossary, not a spec — it fixes the words +so both the architecture conversation and the code stay precise about *who owns which +fact* and *who holds which authority*. + +## Actors + +**Operator**: +The human with decision authority over the trading system — the escalation target for +any fork an agent cannot resolve from evidence, and the executor of supervised one-time +procedures. Agents execute operator workflows on the Operator's behalf; they do not +hold the Operator's authority. +_Avoid_: operator (for an agent running prod procedures — the agent operates prod; it +is not the Operator) ## Invocation lifecycle diff --git a/README.md b/README.md index e38f8ed2c..4c1580449 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Secrets live in `.env` (referenced by env-var name from `config/*.yaml`). See [` ## Scripts ```bash -uv run python scripts/validate_universe.py +uv run python scripts/ops/validate_universe.py ``` Validates `config/assets.yaml` against the five inclusion criteria documented in [`docs/design/asset-universe-validation.md`](docs/design/asset-universe-validation.md). diff --git a/config/assets.yaml b/config/assets.yaml index 423be3742..1db5e40df 100644 --- a/config/assets.yaml +++ b/config/assets.yaml @@ -1,7 +1,7 @@ # Asset universe — authoritative ticker list scoping all external data collection. # Validation procedure: docs/design/asset-universe-validation.md -# Run: uv run python scripts/validate_universe.py -# Discover: uv run python scripts/validate_universe.py --discover +# Run: uv run python scripts/ops/validate_universe.py +# Discover: uv run python scripts/ops/validate_universe.py --discover last_full_validation: 2026-04-25 diff --git a/config/command-center.yaml b/config/command-center.yaml index f76cf3b11..5cb18072a 100644 --- a/config/command-center.yaml +++ b/config/command-center.yaml @@ -4,7 +4,7 @@ bind: host: "127.0.0.1" # loopback for v1 zero-config installs. For LAN access # from other machines, widen this (0.0.0.0 or the # trading machine's LAN IP) together with the - # `access:` block below. See RUNBOOK_command_center.md + # `access:` block below. See docs/runbooks/command-center.md # § "LAN access (local network)" for the supported # recipe (hostname choice, exact YAML, re-enroll, # firewall). The old direct "switch to LAN IP" path diff --git a/config/security.yaml b/config/security.yaml index 43b20fa72..028c118e4 100644 --- a/config/security.yaml +++ b/config/security.yaml @@ -11,7 +11,7 @@ webauthn: # rpId requires passkey re-enrollment (old creds # bound to prior rpId). For LAN: set to your # chosen hostname (e.g. "alphamind.local"). - # See RUNBOOK_command_center.md § "LAN access + # See docs/runbooks/command-center.md § "LAN access # (local network)" and the access: block docs. relying_party_name: "AlphaMind Command Center" # cookies_secure: false # DEPLOY_TIME (ALP-725). False for http LAN/localhost; diff --git a/docs/_handoffs/runbook-split-operate-prod-spec.md b/docs/_handoffs/runbook-split-operate-prod-spec.md new file mode 100644 index 000000000..26ec31606 --- /dev/null +++ b/docs/_handoffs/runbook-split-operate-prod-spec.md @@ -0,0 +1,125 @@ +# Runbook split + scripts/ housecleaning + /operate-prod skill — implementation spec + +Grilled with the Operator 2026-06-10. Implementation-ready; the Linear issue (project: +Operational tooling) is filed after a workspace consolidation pass — the cap sits at +260/250 (ALP-842 rollup frees 20 slots; stale ALP-924 frees 1). The `CONTEXT.md` +glossary change (Actors → **Operator**) is already committed on this branch. + +## 1. Split `scripts/RUNBOOK_production.md` into `docs/runbooks/` + +Content moves verbatim — no editorializing beyond the one precondition addition below. +Pre-fix history blocks (§8.9 etc.) stay with their gotchas. Each new doc opens with a +one-line scope sentence. `scripts/RUNBOOK_production.md` and +`scripts/RUNBOOK_command_center.md` are deleted; no tombstones — `CLAUDE.md` nav and +the README map are the discovery path. + +| New file (`docs/runbooks/`) | Source | +|---|---| +| `README.md` | New: one-line-per-doc map + the boundary rule: this directory holds procedures executed against the live prod system; verification companion docs live beside their scripts under `scripts/verify/`. | +| `update-loop.md` | §1 | +| `bootstrap.md` | §2 | +| `invocations.md` | §3 + §4 | +| `monitoring.md` | §5 | +| `services.md` | Preamble service table + §7 + §10 | +| `troubleshooting.md` | §8 — the living gotchas doc; agents append per the skill's accumulation duty | +| `feedback-loop.md` | §9 — cross-points `docs/agents/feedback-loop-skills.md` | +| `command-center.md` | `scripts/RUNBOOK_command_center.md` with §6 folded in | +| `genesis-cutover.md` | Unchanged | + +**The one substantive addition.** `update-loop.md` and `services.md` gain the +service-stop precondition: if the market is open AND open positions exist, obtain +explicit Operator confirmation before stopping services; otherwise proceed. + +## 2. `scripts/` reorganization + +| Destination | Files | +|---|---| +| `scripts/services/` | All 9 `install_*` / `uninstall_*` `.ps1` | +| `scripts/linear/` | `archive_linear_issues.py`, `check_linear_cap.py`, `export_linear_issues.py`, `linear_consolidation_candidates.py`, `linear_parity_diff.py`, `linear_stale_sweep.py` | +| `scripts/verify/` | All 9 `verify_*.py`, `build_e2e_report.py`, `snapshot_prod_for_debug_e2e.py`, `_e2e_report_assets/`, and the 4 companion docs: `RUNBOOK_end_to_end_verification.md`, `RUNBOOK_genesis_verify.md`, `RUNBOOK_position_thesis_model.md`, `RUNBOOK_counterfactual_replay_engine.md` | +| `scripts/ops/` | `reconcile_alpaca_state.py`, `run_revoked_key_drill.py`, `watch_invocation.sh`, `report_emergency_invocations.py`, `report_flag_rates.py`, `validate_universe.py` | +| `scripts/` root | `gen_package_nav.py`; `mutation/` unchanged | +| Deleted | `spike_alp288_structured_output.py` (referenced nowhere) | + +Shim scripts (4-line wrappers over `src/alphamind/scripts/`) move freely — their tests +import the package. The standalone monoliths are loaded by tests via +`importlib.util.spec_from_file_location` with hardcoded `parents[2] / "scripts" / +".py"` paths: update `_SCRIPT_PATH` in `tests/scripts/test_build_e2e_report.py`, +`tests/scripts/test_verify_debug_e2e.py`, +`tests/scripts/test_verify_position_thesis_model.py` (and sweep `tests/scripts/` for any +other path-based loader). `_e2e_report_assets/` moves together with +`build_e2e_report.py`; confirm its asset resolution is `__file__`-relative. + +## 3. The `/operate-prod` skill + +One skill at `.claude/skills/operate-prod/SKILL.md`, task passed as the argument. No +`references/` directory — canonical procedures live in `docs/runbooks/` and the skill +never restates them (per `docs/agents/` skill-prose discipline). + +**Routing table.** `update` → `update-loop.md` · `invoke` → `invocations.md` (+ +`monitoring.md` to watch completion) · `watch` → `monitoring.md` · `restart` → +`services.md` (restarts honor the update-loop ordering rule) · `triage` → +`troubleshooting.md`. + +**Pointer-only entries** (named, not routed): feedback-loop cadences → the +`feedback-review` / `feedback-validate` / `feedback-retrospective` skills + +`docs/runbooks/feedback-loop.md`; bootstrap → `bootstrap.md` (Operator-supervised); +genesis cutover → `genesis-cutover.md` (Operator-executed — the skill does not run it). + +**Standing directives.** Prod is Windows / NSSM / elevated PowerShell. `.env` never +auto-sources — load it explicitly. Single-writer discipline: no DB writes outside +sanctioned CLIs. Confirm the Alpaca `account_number` before treating broker output as +prod evidence. Establish local time + timezone before reporting any time. + +**Autonomy gate.** Before any service-stopping route, evaluate the runbook +precondition; inside the risk window, halt and obtain Operator confirmation. +Never-autonomously: halt-mode changes, force-close, cancel-order, unsanctioned DB +writes, genesis cutover. + +**Self-correction duty.** After every executed procedure, compare observed behavior +against the runbook followed and against this skill. Where they diverge, propose the +correction: new experiential gotchas append to `docs/runbooks/troubleshooting.md`; +runbook/skill inaccuracies become edits. Deliver corrections as a docs-only PR from a +fresh branch (CI skips docs paths; cheap merge), then restore the checkout to clean +`main` — the update loop requires it. If pushing or opening a PR is unavailable from +prod, present the exact diff to the Operator in-session instead. + +## 4. Reference updates + +- `src/alphamind/command_center/__main__.py` + `config.py`: message/docstring text + `RUNBOOK_command_center.md` → `docs/runbooks/command-center.md`; keep § anchors + aligned with the new doc's headings. +- `tests/command_center/test_main.py` (≈line 135): assertion text matches the new + message. +- `.claude/skills/draft-user-stories/SKILL.md`, `.claude/skills/refine-issue/SKILL.md`: + `scripts/RUNBOOK_production.md` pointers → `docs/runbooks/`. +- Root `CLAUDE.md`: navigation mention of runbook location. +- `docs/project-tracker.md`: live pointers. +- `.ps1` installers and `verify_*` / `src/alphamind/scripts/*` docstrings that name + `scripts/RUNBOOK_*` paths: grep-sweep and update. +- `docs/design/**` stays untouched — point-in-time, not code-truth. +- Final gate: no `scripts/RUNBOOK_` references remain outside `docs/design/`, + `docs/_archive/`, `.archive/`. + +## 5. Dead-pointer guard extension + +`tests/docs/test_claude_md_pointers.py`: extend the scan set to `docs/runbooks/**/*.md` +and `.claude/skills/operate-prod/SKILL.md`; narrow the `.claude/` skip to +`.claude/worktrees/`. Fenced-code-block skip semantics unchanged. The runbooks carry +`%USERPROFILE%`-style Windows paths — the existing absolute-machine-path exemption must +keep them unflagged. + +## 6. Delivery + verification + +Single PR (docs + scripts + skill + reference updates + guard extension together — the +`src/`/`tests/` edits make it CI-gated, so the extended guard validates the new pointer +web before merge). Verification: the standard lint chain, then +`uv run pytest tests/docs/ tests/scripts/ tests/command_center/ -n auto`, then the §4 +final-gate grep. + +## Out of scope + +- Two-tier migration of the four standalone monoliths into `src/alphamind/scripts/` + (separate follow-up issue if ever). +- Runbook content rewrites beyond the move and the §1 precondition addition. +- Any change to the three feedback-loop skills. diff --git a/docs/agents/implementation-ready-issue.md b/docs/agents/implementation-ready-issue.md index 2e4fb5481..1b2022048 100644 --- a/docs/agents/implementation-ready-issue.md +++ b/docs/agents/implementation-ready-issue.md @@ -113,5 +113,5 @@ The issue-body-specific subset of `docs/agents/linear.md` — read that for the cosmetic, don't chase the diff. - **Status names are case-sensitive** (`Todo`, not `todo`). An authoring skill leaves status at `Todo` unless told otherwise — it makes an issue ready, it doesn't start it. -- **Active-issue cap** (free tier, ~250): `uv run python scripts/check_linear_cap.py` for +- **Active-issue cap** (free tier, ~250): `uv run python scripts/linear/check_linear_cap.py` for the count; mind it when an authoring run creates new issues or files spin-offs. diff --git a/docs/agents/linear.md b/docs/agents/linear.md index 5ebfd5d39..91c53cf0e 100644 --- a/docs/agents/linear.md +++ b/docs/agents/linear.md @@ -18,7 +18,7 @@ Referenced from the root `CLAUDE.md` "Linear" section. ## Active-issue cap The repo is on Linear's free tier (250 active issues). For the exact active count, use -`uv run python scripts/check_linear_cap.py` (`--breakdown` / `--needed N` / `--json`) — +`uv run python scripts/linear/check_linear_cap.py` (`--breakdown` / `--needed N` / `--json`) — don't paginate `list_issues`. A recount must pass `includeArchived=false` to see post-delete state. Cmd+Delete soft-archives (populates `archivedAt`) and frees the cap. diff --git a/docs/project-tracker.md b/docs/project-tracker.md index d5f9c8612..6fc43a901 100644 --- a/docs/project-tracker.md +++ b/docs/project-tracker.md @@ -69,9 +69,9 @@ Cross-cutting policies and reference specs that constrain implementation but are - [x] **LLM output validation** ([ALP-127](https://linear.app/alphamind-jatassi/issue/ALP-127)) — _done_ — [design](design/testing/llm-output-validation.md). Four surgical changes closing out the feature: canonical `ValidationError`/`ValidationResult`/`ValidationWarning` hoisted to `commands/validation_results.py` (ALP-520), `find_bare_prefix_citations` added to `analysis/synthesizer/models.py` and wired into the three consumer-side decision validators (ALP-521), parametrized `tests/test_llm_output_validation_contract.py` enforcing cross-cutting invariants over all 7 LLM-output harnesses (ALP-522), and the design doc rewritten to reflect the per-agent + shared-core as-built shape (ALP-523). Follow-ups: `_REF_ID_RE` centralization ([ALP-531](https://linear.app/alphamind-jatassi/issue/ALP-531)), decision-layer `_build_retry_message` consolidation ([ALP-532](https://linear.app/alphamind-jatassi/issue/ALP-532)). - [x] **Paper-evaluation harness** ([ALP-130](https://linear.app/alphamind-jatassi/issue/ALP-130)) — _done_ — [design](design/05-execution-layer/paper-evaluation-harness.md) - [ ] **Counterfactual replay engine** ([ALP-129](https://linear.app/alphamind-jatassi/issue/ALP-129)) — _stories drafted_ — [design](design/05-execution-layer/counterfactual-replay-engine.md) -- [x] **Command center** ([ALP-128](https://linear.app/alphamind-jatassi/issue/ALP-128)) — _done_ — [design](design/command-center.md), [pipeline schema](design/pipeline-control-and-events-schema.md), [monitor schema](design/monitor-control-and-events-schema.md). 22 sub-issues landed across 7 waves (2026-05-26). The third long-running daemon (alongside pipeline + monitor) — FastAPI + uvicorn backend with Pydantic-at-boundary discipline, layered `Overrides` bundles in `build_app`, dual-session-factory split (cc-writer + foreign-reader), TaskGroup-disciplined event multiplexer / control proxy / alert engine; React + Vite + TanStack Router/Query/Table + Shadcn/UI frontend with 11 operator views (live run watcher, run history + failure log, per-invocation detail + source-brief viewer, activity log explorer, portfolio dashboard, position detail, theses dashboard + detail, risk dashboard + regime timeline + cal-mix, generic config editor framework + 8 per-file editor pages + resolved-config viewer + git-history diff). 17 default alert rules with hot-reload on `config/alerts.yaml`, ack/snooze persistence, Discord channel + in-app SSE fanout. WebAuthn + cookie session + CSRF auth; 8 control verbs proxied to pipeline + monitor with full audit-log trail. Verify script (`scripts/verify_command_center.py`) drives 7 end-to-end checks (boot, passkey, 8 verbs, alert fires, SSE roundtrip, frontend build, clean teardown). NSSM service-install script. RUNBOOK covering bring-up + day-to-day + recovery + 6 troubleshooting modes. 4 wave-end /code-review passes returned 60 verified findings; top-15 of each (45 total) applied as wave-end followups. Post-merge cleanup tracked: per-site audit to drive L4 broad-except / L9 internal-Pydantic / L19 async-over-sync ceilings back down (these climbed because the operator console added warranted boundary models + Protocol stubs that the antipattern scanner can't distinguish from internal types); 25+ lower-severity findings (cut from top-15) for incremental polish; ALP-664/665 pipeline + monitor `/control` + `/events` production-wiring end-to-end verification tracked separately. +- [x] **Command center** ([ALP-128](https://linear.app/alphamind-jatassi/issue/ALP-128)) — _done_ — [design](design/command-center.md), [pipeline schema](design/pipeline-control-and-events-schema.md), [monitor schema](design/monitor-control-and-events-schema.md). 22 sub-issues landed across 7 waves (2026-05-26). The third long-running daemon (alongside pipeline + monitor) — FastAPI + uvicorn backend with Pydantic-at-boundary discipline, layered `Overrides` bundles in `build_app`, dual-session-factory split (cc-writer + foreign-reader), TaskGroup-disciplined event multiplexer / control proxy / alert engine; React + Vite + TanStack Router/Query/Table + Shadcn/UI frontend with 11 operator views (live run watcher, run history + failure log, per-invocation detail + source-brief viewer, activity log explorer, portfolio dashboard, position detail, theses dashboard + detail, risk dashboard + regime timeline + cal-mix, generic config editor framework + 8 per-file editor pages + resolved-config viewer + git-history diff). 17 default alert rules with hot-reload on `config/alerts.yaml`, ack/snooze persistence, Discord channel + in-app SSE fanout. WebAuthn + cookie session + CSRF auth; 8 control verbs proxied to pipeline + monitor with full audit-log trail. Verify script (`scripts/verify/verify_command_center.py`) drives 7 end-to-end checks (boot, passkey, 8 verbs, alert fires, SSE roundtrip, frontend build, clean teardown). NSSM service-install script. RUNBOOK covering bring-up + day-to-day + recovery + 6 troubleshooting modes. 4 wave-end /code-review passes returned 60 verified findings; top-15 of each (45 total) applied as wave-end followups. Post-merge cleanup tracked: per-site audit to drive L4 broad-except / L9 internal-Pydantic / L19 async-over-sync ceilings back down (these climbed because the operator console added warranted boundary models + Protocol stubs that the antipattern scanner can't distinguish from internal types); 25+ lower-severity findings (cut from top-15) for incremental polish; ALP-664/665 pipeline + monitor `/control` + `/events` production-wiring end-to-end verification tracked separately. - [ ] **Feedback loop** ([ALP-131](https://linear.app/alphamind-jatassi/issue/ALP-131)) — _requirements pending_ — [design](design/feedback-loop.md) -- [x] **Debug e2e mode** ([ALP-493](https://linear.app/alphamind-jatassi/issue/ALP-493)) — _done_ — synthetic portfolio + log-only broker + progress events for agent-driven e2e verification on the scheduler `--debug-e2e` flag — [design](design/debug-e2e-mode.md). 9 sub-issues landed across 5 waves (2026-05-17). Side effects: 3 decision harnesses (analyst/strategist/PM) refactored through `_harness_core.invoke_sdk` to satisfy the single-SDK-emit-point invariant (finishing ALP-466's deferred consolidation); `ProgressEmitter` Protocol lives in `_kernel/progress.py` with a re-export from `scheduler/progress.py`; legacy per-feature verify suite (19 scripts + 20 testable bodies + 18 tests + 17 runbooks) ripped out in favor of `scripts/verify_debug_e2e.py` as the sole pipeline e2e surface; `scripts/build_e2e_report.py` rewritten for the single-invocation archive; `scripts/RUNBOOK_end_to_end_verification.md` rewritten around the debug-e2e flow. +- [x] **Debug e2e mode** ([ALP-493](https://linear.app/alphamind-jatassi/issue/ALP-493)) — _done_ — synthetic portfolio + log-only broker + progress events for agent-driven e2e verification on the scheduler `--debug-e2e` flag — [design](design/debug-e2e-mode.md). 9 sub-issues landed across 5 waves (2026-05-17). Side effects: 3 decision harnesses (analyst/strategist/PM) refactored through `_harness_core.invoke_sdk` to satisfy the single-SDK-emit-point invariant (finishing ALP-466's deferred consolidation); `ProgressEmitter` Protocol lives in `_kernel/progress.py` with a re-export from `scheduler/progress.py`; legacy per-feature verify suite (19 scripts + 20 testable bodies + 18 tests + 17 runbooks) ripped out in favor of `scripts/verify/verify_debug_e2e.py` as the sole pipeline e2e surface; `scripts/verify/build_e2e_report.py` rewritten for the single-invocation archive; `scripts/verify/RUNBOOK_end_to_end_verification.md` rewritten around the debug-e2e flow. - [x] **Debug e2e resume** ([ALP-689](https://linear.app/alphamind-jatassi/issue/ALP-689)) — _done_ — [design](design/debug-e2e-resume.md). Resume capability for partial-failure e2e runs — `--resume-from :` flag on `scheduler run --debug-e2e` hydrates the 9 SDK-phase outputs from a prior archive (one `phase_outputs/.json` per phase, Pydantic-serialized) and re-runs from the named phase forward. Deterministic phases (phase1, snapshot_assembly, distillation, pre_processor) re-run cheaply from `wipe_and_seed`. Cuts ~10–12 min and up to ~6 Sonnet + 3 Opus calls off each iteration when bumping a per-agent latency / token budget after a failure. Adds one new post-run check (`check_deterministic_prefix`) to the verify wrapper that hashes the distillation outputs against the source archive to catch non-determinism slipping into the upstream-re-run path. 7 sub-issues landed across 5 waves (2026-05-26). Side effects: unified the legacy `/invocations//` and date-partitioned `///` layouts under the latter (8+ production callers migrated + `emitter_factory` now takes `as_of`); per-phase Pydantic boundary models added under each of the 9 agents' `models.py`; `pipeline/{analysis,decision}.py` extended with `_replay_*_phase` short-circuit helpers that emit `phase_done(..., replayed_from=)`; `_PHASE_DEPENDENCIES` DAG pinned in `scheduler/debug_e2e/resume.py`; `RUNBOOK_end_to_end_verification.md` extended with a "Resuming a failed run" section. - [x] **Architecture refactoring (2026-05-12 audit)** ([ALP-454](https://linear.app/alphamind-jatassi/issue/ALP-454)) — _done_ — [audit](../audit-alphamind-2026-05-12.html), [summary](audit-summary-2026-05-12.md). 27 sub-issues ([ALP-455](https://linear.app/alphamind-jatassi/issue/ALP-455) through [ALP-481](https://linear.app/alphamind-jatassi/issue/ALP-481)) landed across 12 waves between 2026-05-12 and 2026-05-15. Both system-spanning import cycles eliminated (10-module `portfolio_state ↔ risk_guardrails ↔ distillation ↔ persistence`; 12-module `decision ↔ execution`); `_kernel/` introduced (Money/Price NewType-Decimal, NewType IDs, regime/calibration enums, atomic_io); `commands/` introduced (OMS wire-format types + BrokerDispatch Protocol); 8 `import-linter` contracts wired into CI; Pydantic-at-boundary discipline restored (363 → 192 internal types, with frozen-dataclass mirrors for the warranted boundary types); 5 god modules decomposed; `tests/data_sources/` vendor mocks replaced by Protocols + in-memory fakes (244 → 0). Remaining propagation tracked as [ALP-483](https://linear.app/alphamind-jatassi/issue/ALP-483) (`library_snapshot.py` relocation), [ALP-484](https://linear.app/alphamind-jatassi/issue/ALP-484)–[ALP-487](https://linear.app/alphamind-jatassi/issue/ALP-487) (q3/q6/q7/qualitative compute-split propagation), [ALP-488](https://linear.app/alphamind-jatassi/issue/ALP-488) (test-side mypy disable-error-code cleanup); [ALP-482](https://linear.app/alphamind-jatassi/issue/ALP-482) (composition-root audit) concluded keep. diff --git a/docs/runbooks/README.md b/docs/runbooks/README.md new file mode 100644 index 000000000..512f362c6 --- /dev/null +++ b/docs/runbooks/README.md @@ -0,0 +1,32 @@ +# Production runbooks + +Operator workflows for AlphaMind in production — on the Windows trading machine, +against the live `alphamind.db` and paper Alpaca account. This directory holds +**procedures executed against the live prod system**; verification-artifact companion +docs live beside their scripts under `scripts/verify/`. Agents reach these through the +`operate-prod` skill, which routes situations onto the right runbook and carries the +agent-side directives (autonomy gate, never-autonomously list, self-correction duty). + +| Runbook | Answers | +|---|---| +| [update-loop.md](update-loop.md) | Deploying latest `main`: pull, sync, migrate, restart, verify | +| [services.md](services.md) | The seven NSSM services, restarting one, ports + paths reference | +| [invocations.md](invocations.md) | Running a manual invocation; the scheduler's run-type schedule | +| [monitoring.md](monitoring.md) | Watching an invocation live; after-the-fact investigation | +| [troubleshooting.md](troubleshooting.md) | Symptom-indexed gotchas — the living doc agents append to | +| [command-center.md](command-center.md) | Command-center bring-up, access, passkeys, recovery | +| [feedback-loop.md](feedback-loop.md) | Feedback-loop CLIs and cadences on prod | +| [bootstrap.md](bootstrap.md) | One-time first-run bring-up (Operator-supervised) | +| [genesis-cutover.md](genesis-cutover.md) | One-time genesis cutover (Operator-executed) | + +The split docs keep the original monolith's section numbering (update-loop = § 1, +bootstrap = § 2, invocations = §§ 3–4, monitoring = § 5, services = §§ 7 + 10, +troubleshooting = § 8, feedback-loop = § 9), so cross-doc references like +`monitoring.md § 5.4` stay stable; a doc opening at "§ 7" is complete, not truncated. + +> **Living documents — agents, keep them current.** If you follow a procedure here +> and it's wrong, outdated, or doesn't match actual prod behavior, update the runbook +> in the same change that produces the fix. Don't paper over a broken step with an +> ad-hoc workaround — make the runbook reflect reality. New monitoring patterns, new +> gotchas, new failure modes, schedule changes, port changes — they all belong here so +> the next operator (human or agent) inherits the lesson. diff --git a/docs/runbooks/bootstrap.md b/docs/runbooks/bootstrap.md new file mode 100644 index 000000000..5f4d41faf --- /dev/null +++ b/docs/runbooks/bootstrap.md @@ -0,0 +1,303 @@ +# First-time bootstrap + +One-time first-run procedure standing up the seven NSSM services from a bare Windows +machine — Operator-supervised, not an agent route. The recurring deploy procedure is +[update-loop.md](update-loop.md). + +## 2. First-time bootstrap + +**Run § 2 exactly once, the very first time AlphaMind comes up on the +production machine.** Subsequent operations all go through update-loop.md § 1. + +> **If you are performing the broker-boundary genesis cutover (ADR-0001–0005)** +> — swapping to a new Alpaca paper account and a fresh DB — use +> `docs/runbooks/genesis-cutover.md` as your primary procedure. That runbook +> supersedes § 2.2 (the fresh DB replaces the additive migration) and is the +> current first-run procedure for the new design. The account-swap step +> (§ 2.1) and the `--fresh-start` cold-start invocation (§ 2.4) remain +> relevant: § 2.1 for populating the new account's credentials in `.env`, +> and § 2.4 for the `synthetic_id_count == 0` verification and hard-fail +> paths. Return to § 2.5 onward to install the five remaining NSSM services +> after the genesis cutover bootstrap step. + +The collector is already running on this box (it was installed first to +accumulate the months of distillation-calibration data the analysis layer +needs). Steps 2.5–2.6 below assume that and skip re-installing it. + +### 2.1 Populate `.env` + +Copy `.env.example` to `.env` at the repo root, then fill in real values for +the vendor keys it lists. The template is incomplete — three additional keys +are required by prod but missing from `.env.example`. Append them after the +existing block: + +```bash +# === Append to .env after the vendor-key block === + +# Required: Claude SDK OAuth token. Every pipeline + monitor invocation's +# SDK subprocess inherits this from the process environment. Generate via +# `claude setup-token` on a machine signed into the operator's Anthropic +# account, then copy the value here. +CLAUDE_CODE_OAUTH_TOKEN= + +# Optional: Discord alert fanout. When unset, the command center logs +# "Discord alerts disabled" at startup and the in-app channel still works. +# Generate via Discord → Server Settings → Integrations → Webhooks → New +# Webhook. Treat the URL as a secret. +ALPHAMIND_DISCORD_WEBHOOK= + +# Optional but strongly recommended: pinned session-cookie signing key. +# When unset, the command center mints a fresh key on every restart and +# every active operator session is invalidated. Generate a 32-byte +# base64 random value: +# python -c "import secrets; print(secrets.token_urlsafe(32))" +# Placeholder is fine for first boot — the command center auto-generates +# an in-process secret if this is empty. Pin it before the first scheduled +# restart you care about. +COMMAND_CENTER_SESSION_SECRET= +``` + +`ALPACA_PAPER_KEY` and `ALPACA_PAPER_SECRET` (already present in the +template) must be populated with paper-account credentials from +`app.alpaca.markets/paper/dashboard/overview`. Live keys are deliberately +left blank — the `--fresh-start` argparse blocks `--mode live` and the +scheduler defaults `--mode paper` so no service can accidentally route to +the live endpoint. + +**CRLF gotcha.** On Windows the editor will save `.env` with `\r\n` line +endings. Bare `source .env` under Git Bash leaves a trailing `\r` on every +value — `CLAUDE_CODE_OAUTH_TOKEN` then fails bearer auth several minutes into +a pipeline run. For any manual CLI invocation, source via: + +```bash +set -a && source <(tr -d '\r' < .env) && set +a +``` + +NSSM-managed services bypass this — each entrypoint calls `load_dotenv()`, +which parses the repo-root `.env` from the service's working directory (NSSM +`AppDirectory`) and strips line endings, so the `\r` never survives. No shell +`source` step is involved. (The install scripts do **not** inject the vendor +keys via NSSM's `AppEnvironmentExtra`; the only env value set that way is the +command-center session secret in § 2.5.) + +### 2.2 Bring the DB to alembic head + +> **Cutover to the new broker-boundary design (ADR-0001–0005) requires a +> fresh DB, not an additive migration.** The procedure in +> `docs/runbooks/genesis-cutover.md` — swapping to a new Alpaca paper +> account and a new empty DB file — **supersedes this step** for that +> one-time event. After the genesis cutover § 2 does not apply; the system +> is already bootstrapped and all subsequent operations go through update-loop.md § 1. +> +> The text below applies to the **legacy** setup (old Alpaca account, old +> DB) and to any incremental schema updates after genesis. + +```powershell +uv run alembic upgrade head +uv run alembic current # confirm at heads +``` + +The collector has been writing into this DB for months; do not delete or +re-create it for incremental updates. Migrations are additive. + +### 2.3 Build the command-center frontend + +```powershell +cd src\alphamind\command_center\frontend +bun install --frozen-lockfile +bun run build +Test-Path .\dist\index.html # must return True +cd ..\..\..\..\.. +``` + +If `bun --version` reports anything older than 1.1.x, upgrade first — earlier +versions silently mutate `bun.lock` on `--frozen-lockfile`. + +### 2.4 Cold-start bootstrap invocation + +The paper-Alpaca account must be at zero positions with starting cash before +this runs. The bootstrap fetches Alpaca's reported cash, writes the +`cash_ledger` + `drawdown_state` singletons, and runs one `market_open` +invocation. The invocation drives the full pipeline (fill collection → analysis → +decision → command execution broker dispatch) — the PM's accepted commands must land +on Alpaca with real broker order ids. The synthetic `alp-{order_id}` +placeholder is deleted (ALP-847): an order with no broker counterpart — a +monitor-enforced protective leg (armed Intent the continuous monitor +enforces) or a not-yet-routed order — carries a NULL `alpaca_order_id`, never +a placeholder. Refuses to run if either singleton already exists. + +```bash +set -a && source <(tr -d '\r' < .env) && set +a && \ + uv run python -m alphamind.scheduler run \ + --fresh-start \ + --once market_open \ + --reason "first-run bootstrap" +``` + +Expected: one full invocation followed by the process exiting 0. Both +singletons committed to the DB, and — if the PM produced any envelopes — +the corresponding entry + broker-enforced protective legs (the native +equity bracket's take-profit + first stop) should appear on the Alpaca side +with real UUIDs; monitor-enforced legs carry a NULL `alpaca_order_id` (no +broker order — they are not on Alpaca, by design). Wall-clock is +**~25–35 min** on a cold-cache cold-start — every SDK call pays first-fill +`cache_write` cost (no warm prompt cache), the deterministic distillation +step takes ~3–5 min against the full prod data layer, and at least one of +the Sonnet phases (`adaptive` is the usual culprit) typically dominates at +~6–8 min. This is the same band as the steady-state ~25–50 min scheduled +invocations document in invocations.md § 4; budget accordingly and don't restart +the process if it looks "stuck" inside that window — run the e2e progress +monitor (`scripts/verify/RUNBOOK_end_to_end_verification.md` § Monitoring +progress mid-run) against the invocation's archive to see live phase +transitions. Confirm: + +```bash +uv run python -c " +import sqlite3, os +db = sqlite3.connect(os.path.expandvars(r'%USERPROFILE%\AlphaMind\data\alphamind.db')) +print('cash_ledger:', db.execute('SELECT current_cash_usd FROM cash_ledger').fetchone()) +print('drawdown_state:', db.execute('SELECT equity_high_water_mark_usd FROM drawdown_state').fetchone()) +print('order_count:', db.execute('SELECT COUNT(*) FROM orders').fetchone()) +print('synthetic_id_count:', db.execute(\"SELECT COUNT(*) FROM orders WHERE alpaca_order_id LIKE 'alp-%'\").fetchone()) +print('orphan_pending_no_broker_id:', db.execute(\"SELECT COUNT(*) FROM orders o WHERE o.alpaca_order_id IS NULL AND o.status NOT IN ('PENDING_SUBMIT','CANCELLED','REJECTED') AND NOT EXISTS (SELECT 1 FROM bracket_legs bl WHERE bl.order_id = o.order_id AND bl.enforcement_binding = 'monitor_enforced')\").fetchone()) +" +``` + +Both singleton rows should match Alpaca's reported cash on the freshly-reset +account to the cent. **`synthetic_id_count` must be `0` — by construction**: +ALP-847 deleted the `alp-{order_id}` mint, so no code can persist a synthetic +placeholder; a non-zero count would mean a stale pre-ALP-847 DB (re-baseline +it). A NULL `alpaca_order_id` is now the *expected* steady state for a +monitor-enforced leg or a not-yet-routed (`PENDING_SUBMIT`) order — NOT a +defect. The meaningful check is **`orphan_pending_no_broker_id` must be `0`**: +an active order with no broker id that is *not* a monitor-enforced leg means +broker dispatch was bypassed (the invocation did not reach Alpaca). Regression +checking should also confirm via `TradingClient.get_orders(status=ALL)` that +the broker-enforced orders are visible on the Alpaca side. + +**Hard-fail paths.** `--fresh-start` refuses to run when: + +- **Alpaca reports any open positions.** The error names the offending + symbol(s). Reset the Alpaca account first. +- **Alpaca reports any open orders.** `--fresh-start` requires a flat account — + positions empty **and** the broker order book empty. The error names the + offending order symbol(s). Cancel them (reset the account) first. +- **`cash_ledger` already has a row.** The error includes the existing + `current_cash_usd`. Positions/cash are a derived **Projection** rebuilt each + invocation from the broker-event log applied to the live broker snapshot + (ALP-854 / ADR-0001) — there is no auto-correct adjudication; a + re-bootstrap is never the correct path once the singleton is populated. +- **`drawdown_state` already has a row.** Same shape, same recovery. + +The CLI exits with code 2 (distinct from the generic scheduler-error code 1) +and prints the message verbatim to stderr — no traceback. Match on the +prefix `--fresh-start:` to filter precondition failures from scheduler +crashes. + +**Recovery if the cold-start invocation itself fails after the singletons +committed.** The two rows are committed in their own transaction *before* +the invocation runs, so a fill collection error or transient external dependency +leaves them in place and a retry of `--fresh-start` will hard-fail. Two +options, in preference order: + +1. **Re-run without `--fresh-start`.** The singletons are already populated + correctly; the bootstrap step is no longer needed: + + ```bash + uv run python -m alphamind.scheduler run \ + --once market_open \ + --reason "retry after bootstrap" + ``` + +2. **Wipe the singletons and re-bootstrap** — only if the persisted values + are wrong (e.g., the Alpaca fetch returned a transient zero-cash + mid-reset). On the prod machine: + + ```sql + DELETE FROM drawdown_state; + DELETE FROM cash_ledger; + ``` + + Then re-run the `--fresh-start --once market_open --reason ...` form. + +### 2.5 Install the six remaining NSSM services + +Collector is already installed. Install the other six in dependency order +from an elevated PowerShell prompt at the repo root. `install_monitor_service.ps1` +and `install_safety_core_service.ps1` each install **both** their supervised +service and its dedicated watchdog — set the `ObjectName` on all four (each +watchdog must run under the same account so its `nssm restart ` has +permission): + +```powershell +.\scripts\services\install_pipeline_scheduler_service.ps1 +nssm set alphamind-scheduler ObjectName .\ # prompts for pw + +.\scripts\services\install_monitor_service.ps1 +nssm set alphamind-monitor ObjectName .\ +nssm set alphamind-monitor-watchdog ObjectName .\ + +.\scripts\services\install_safety_core_service.ps1 +nssm set alphamind-safety-core ObjectName .\ +nssm set alphamind-safety-core-watchdog ObjectName .\ + +.\scripts\services\install_command_center_service.ps1 +nssm set AlphaMindCommandCenter ObjectName .\ +``` + +Pin the command-center session secret on the service environment so restarts +don't kick operators out: + +```powershell +$secret = python -c "import secrets; print(secrets.token_urlsafe(32))" +nssm set AlphaMindCommandCenter AppEnvironmentExtra "COMMAND_CENTER_SESSION_SECRET=$secret" +``` + +(Also copy the value into `.env`'s `COMMAND_CENTER_SESSION_SECRET=` line so +manual CLI invocations get the same key.) + +### 2.6 Start the six new services + +Start each supervised service **before its watchdog** (a watchdog probes its +target's heartbeat and would restart a service that hasn't beaten yet): + +```powershell +nssm start alphamind-scheduler +nssm start alphamind-monitor +nssm start alphamind-monitor-watchdog +nssm start alphamind-safety-core +nssm start alphamind-safety-core-watchdog +nssm start AlphaMindCommandCenter +Get-Service alphamind-collector, alphamind-scheduler, alphamind-monitor, alphamind-monitor-watchdog, alphamind-safety-core, alphamind-safety-core-watchdog, AlphaMindCommandCenter +``` + +All seven should be `Running`. + +### 2.7 Register the first passkey + +The command-center daemon mints a one-time setup token on first boot. +Retrieve it from the log: + +```powershell +Get-Content "$env:USERPROFILE\AlphaMind\logs\command_center.out.log" -Tail 50 | + Select-String "command_center setup token" +``` + +Open `http://localhost:8090/` in a browser on the Windows machine (RDP in if +remote), paste the token, choose a username, and complete WebAuthn +registration with Windows Hello or a hardware key. **Use the `localhost` +hostname, not `127.0.0.1`** — the WebAuthn relying-party id is `localhost` +(`config/security.yaml`), so a passkey ceremony loaded from the `127.0.0.1` +origin is rejected by the browser. See +[command-center.md](command-center.md) § Register the first passkey for details and § +Adding a passkey for enrolling a backup authenticator. + +### 2.8 Verify green + +```bash +set -a && source <(tr -d '\r' < .env) && set +a && \ + uv run python scripts/verify/verify_command_center.py +``` + +Expect `7/7 checks passed`. AlphaMind is live in paper-trading mode. + diff --git a/scripts/RUNBOOK_command_center.md b/docs/runbooks/command-center.md similarity index 93% rename from scripts/RUNBOOK_command_center.md rename to docs/runbooks/command-center.md index 9ef359970..fe5b085fb 100644 --- a/scripts/RUNBOOK_command_center.md +++ b/docs/runbooks/command-center.md @@ -12,12 +12,12 @@ pipeline scheduler and the continuous monitor); the three are loosely coupled via loopback HTTP — the command center proxies operator actions to the pipeline's `/control/*` and the monitor's `/control/*` surfaces, and consumes both daemons' `/events` SSE streams. See -[`docs/design/command-center.md`](../docs/design/command-center.md) for the +[`docs/design/command-center.md`](../design/command-center.md) for the full design. ## Purpose -A green `scripts/verify_command_center.py` run proves the command center is +A green `scripts/verify/verify_command_center.py` run proves the command center is ready to serve traffic on a production machine: - The FastAPI app binds to the configured loopback port within 30 s. @@ -62,7 +62,7 @@ per-step PASS / FAIL report. with capped backoff until the upstreams come up, and the control proxy reaches them per-request. -The full prod runtime is **seven** NSSM services (see `RUNBOOK_production.md`'s +The full prod runtime is **seven** NSSM services (see [services.md](services.md)'s service table): `alphamind-collector`, `alphamind-scheduler`, `alphamind-monitor`, `alphamind-monitor-watchdog`, `alphamind-safety-core`, `alphamind-safety-core-watchdog`, and `AlphaMindCommandCenter`. The @@ -71,7 +71,7 @@ core (ADR-0004 / ALP-857); each supervised service's dedicated out-of-process watchdog (`alphamind-monitor-watchdog` / `alphamind-safety-core-watchdog`, ALP-941) restarts its target on heartbeat staleness. The command center does not surface the safety-core services' state — they are monitored via their logs -(`safety_core.*.log`) and the heartbeat file, per `RUNBOOK_production.md` §7/§9. +(`safety_core.*.log`) and the heartbeat file, per [services.md](services.md) and [troubleshooting.md](troubleshooting.md) § 8.9. ## Initial bring-up @@ -95,8 +95,8 @@ target revision matches `head` after upgrade. ### 2. Build the frontend bundle The command center serves the SPA from -`src/alphamind/command_center/frontend/dist/`. Bun produces this bundle -from the TypeScript / React source under `frontend/src/`. +`src/alphamind/command_center/frontend` (`dist` bundle). Bun produces this bundle +from the TypeScript / React source under `src/alphamind/command_center/frontend/src/`. ```bash cd src/alphamind/command_center/frontend @@ -106,12 +106,12 @@ bun run build `bun install --frozen-lockfile` is strict on the lockfile — if dependencies drifted, it fails instead of silently rewriting. `bun run build` runs `tsc -b -&& vite build` → `dist/`. Confirm `dist/index.html` exists before continuing. +&& vite build` → `dist`. Confirm `dist/index.html` exists before continuing. ### 3. Install the NSSM service ```powershell -.\scripts\install_command_center_service.ps1 +.\scripts\services\install_command_center_service.ps1 nssm set AlphaMindCommandCenter ObjectName .\ nssm start AlphaMindCommandCenter Get-Service AlphaMindCommandCenter @@ -171,12 +171,12 @@ while logged in — no setup token required. ### 6. Run the verify script -The end-to-end gate is `scripts/verify_command_center.py`. Run it from the +The end-to-end gate is `scripts/verify/verify_command_center.py`. Run it from the repo root after the service is up: ```bash set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/verify_command_center.py + uv run python scripts/verify/verify_command_center.py ``` Expected output (one line per check, all PASS): @@ -203,6 +203,17 @@ Exit code 0 confirms readiness. Any FAIL line short-circuits to exit code 1. The full remote (internet) path via VPS Caddy + WireGuard remains deferred to a future story (see design doc § Access surfaces and the old "Operator handover note" content now superseded by the LAN section). For day-to-day on the trading machine, RDP/console + localhost still works; LAN extends it without RDP. +The dashboard surfaces, in order of operator priority: + +- **Controls** — pause/resume, trigger emergency invocation, switch profile, + cancel order, force-close position, set halt mode. +- **Live run watcher** — current invocation's phase + agent SSE stream. +- **Run history** — last N invocations with status + duration + cost. +- **Activity log** — every `activity_log` row, filterable. +- **Portfolio / theses / risk** — current state snapshot. +- **Alerts** — active + acked + history. +- **Configs** — per-file editor + git diff view. + ### Restarting the service A service restart re-mints the in-process session-cookie signing secret @@ -392,8 +403,8 @@ pipeline / monitor: nssm stop alphamind-collector ``` 2. Restore the most recent `alphamind.db` backup (the pipeline scheduler's - `archive/` directory carries periodic snapshots; pick the most recent - verified-good copy). + `%USERPROFILE%\AlphaMind\archive\` directory carries periodic snapshots; + pick the most recent verified-good copy). 3. `uv run alembic upgrade head` to bring the restored DB's schema current. 4. Restart all seven services in the conventional start order (each supervised service before its watchdog): @@ -536,13 +547,13 @@ hard-coded allow-list on the server. - Passkey registration + login succeeds end-to-end. - Dashboard renders, control verbs work, SSE events flow, no console errors about origin / CSRF / cookie. - - Run `scripts/verify_command_center.py` locally on the trading + - Run `scripts/verify/verify_command_center.py` locally on the trading machine (it continues to use loopback for its internal checks) — all PASS. - Reboot / restart service; confirm the LAN URL still works after. If the mixed-bind warning fires on startup it explicitly says "Edit the -access: block ... see RUNBOOK_command_center.md for LAN setup." +access: block ... see docs/runbooks/command-center.md for LAN setup." ### Hostname suggestion block (example output) @@ -565,7 +576,7 @@ LAN access suggestion (ALP-724 plan): relying_party_id: "alphamind.local" 4. Restart service; re-enroll via setup token from LAN URL 5. Firewall: allow LAN subnet → :8090 - Full recipe + troubleshooting: RUNBOOK_command_center.md § "LAN access (local network)" + Full recipe + troubleshooting: docs/runbooks/command-center.md § "LAN access (local network)" ``` (The exact emission logic and wording land in 03a; docs here describe the diff --git a/docs/runbooks/feedback-loop.md b/docs/runbooks/feedback-loop.md new file mode 100644 index 000000000..cf9a2c472 --- /dev/null +++ b/docs/runbooks/feedback-loop.md @@ -0,0 +1,142 @@ +# Feedback loop — prod cadences and CLIs + +Operating the feedback loop on prod: the analytics/validation/retrospective CLIs and +their cadences. The skills' shared reference is +[docs/agents/feedback-loop-skills.md](../agents/feedback-loop-skills.md). + +## 9. Feedback loop — analytics CLIs, cadences, and review skills + +The feedback loop (ALP-131) is the month-over-month system-improvement substrate: +it pairs the system's reasoning artifacts with the outcomes they produced and +computes deterministic, conditionable metrics over that pairing. It is **read-only +over trading state** — it measures and proposes, never mutating live trading-state +records — and runs **out-of-pipeline** as a set of operator CLIs plus three review +skills. The deliberative pipeline gained three prod-runtime behaviors that feed it +(documented in their own sections, cross-referenced below). + +The end-to-end spine is verified out-of-pipeline by +`scripts/verify/verify_feedback_loop.py` (it runs `alembic upgrade head` on a scratch DB, +seeds a controlled closed-position scenario, and drives resolver → metrics → digest ++ snapshot → validation → supersession → retrospective, asserting each stage). Run +it after a change that touches the feedback-loop tables or the resolver: + +```bash +uv run python scripts/verify/verify_feedback_loop.py # exits 0 on PASS, 1 on any FAIL +``` + +### 9.1 Prod-runtime behavior changes (the pipeline now feeds the loop) + +Three behaviors the deliberative pipeline gained — each documented in full where it +lands, listed here so the feedback-loop picture is complete: + +- **`agent_calls` telemetry + provenance every invocation** — every LLM agent call + persists one `agent_calls` row + a four-file provenance tree. Active in production + (ALP-907). See invocations.md § 3 ("agent_calls telemetry capture is active in production") and + the services.md § 10 reference path. The metrics' `WindowDataset` reads `agent_calls` for the + cost / process / citation-chain tiers. +- **Per-invocation thesis resolution (`ACTIVE → RESOLVED`)** — every deliberative + invocation runs a `thesis_resolution` step between fill collection and snapshot + assembly, authoring each closed-position thesis's component outcomes + resolution + category. See monitoring.md § 5.7 ("Thesis-resolution skip warnings"). The outcome-tier metrics + calibrate against these resolved theses; a per-thesis data gap is a skip-WARNING, + not a fault. +- **Distillation anomaly emission to `activity_log`** — the distillation layer emits + `DISTILLATION_ANOMALY_FLAG` activity-log entries (ALP-881) when a threshold in its + anomaly taxonomy trips; the digest's notable-shift surface reads them. + +### 9.2 The combined feedback-loop migration (apply before anything reads) + +The six feedback-loop tables (`agent_calls`, `validations`, `validation_outcomes`, +`weekly_digest_snapshots`, `retrospective_reports`, `retrospective_decisions`) and +the `activity_log` event-type / source CHECK widenings land in the combined +migration **ALP-879** (`c001fb0000ff_feedback_loop_tables_and_activity_log_check_widen`). +It is applied by the standard `alembic upgrade head` in the update loop (update-loop.md § 1) / +first-run bootstrap (bootstrap.md § 2). Until it is applied on the prod box the `agent_calls` +inserts silently no-op and every analytics CLI errors on the missing table. + +### 9.3 Analytics-read CLI — digest, metric, snapshot + +`python -m alphamind.feedback_loop.digest.cli` (all subcommands emit JSON on stdout; +`--db-path` defaults to the live DB, `--config-dir` to `config/`): + +- `digest [--week YYYY-MM-DD] [--trajectory-weeks N]` — the full six-section + `WeeklyDigest` for the week (headline outcomes, process pulse, 8–12-week + trajectory, validation status, notable shifts, replay queue). +- `metric [--window N] [--week …]` — one `MetricResult` aggregated over + the trailing `--window` weeks. +- `metrics-list` — the registered metric ids (a metric gated on an absent + dependency, e.g. PM-accuracy before ALP-129, is simply absent). +- `snapshot [--week …] [--trajectory-weeks N]` — **producer**: writes the week's + `WeeklyDigest` into `weekly_digest_snapshots`, idempotently (the `week_start` + UNIQUE makes a re-run a reported no-op). + +All four are DB-only — no vendor API, no `.env` needed. + +**Cadence.** The `snapshot` producer is intended to run **once per week, Sunday +8am ET**, snapshotting the week just closed (omitting `--week` targets the prior +completed week). It is **not yet wired into the scheduler** — run it from an +operator cron / CI job, or by hand. `digest` / `metric` are on-demand reads the +`/feedback-review` skill drives. + +### 9.4 Validation CLI — register, evaluate, list, detect-supersessions + +`python -m alphamind.feedback_loop.validation.cli` (JSON on stdout; `register` / +`evaluate` take a JSON payload on `--input`, a path or `-` for stdin). DB-only: + +- `register` — snapshot the registering invocation's provenance (regime + + model id from `agent_calls`), freeze the pre-registered criteria, persist the + validation. +- `evaluate --validation-id … --outcome-id … --evaluated-at …` — compute the + pre/post metric comparison, derive the verdict + rollback status, write the + `validation_outcomes` row (or short-circuit if the validation was superseded). +- `list` — the pending (unevaluated, un-superseded) validations. +- `detect-supersessions` — **producer**: mark each active validation whose + post-edit window was crossed by a conditioning shift (regime transition, model + version change, or a commit to the watched artifact) `superseded`. + +**Cadence.** Run `detect-supersessions` **once per pipeline invocation and once per +commit** (the two events that can break a validation's conditioning context). It is +**not yet scheduler-wired** — drive it from the pipeline-adjacent cron / a commit +hook / CI, or via `/feedback-validate`. + +### 9.5 Retrospective CLI — ingest, save-report, capture-decision + +`python -m alphamind.feedback_loop.retrospective.cli` (DB-only): + +- `ingest --start … --end …` — pull the Phase-1 retrospective data set over the + window (the analytics-spine `WindowDataset` + the unresolved pending-rollback + follow-ups). +- `save-report --start … --end … --markdown-file …` — persist the rendered report + metadata row + write the markdown under + `data/retrospective_reports/{report_id}/report.md`. +- `capture-decision --report-id … --item-identifier … --decision-type … --verdict … + --rationale …` — record one walkthrough decision (promotion candidate or + follow-up), optionally linking a spawned validation. + +### 9.6 The three review skills (headless) + +The operator drives the loop through three Claude skills, all **headless** — they +read the analytics through the CLIs above and emit markdown the operator reads: + +- **`/feedback-review`** — weekly process-metric / monthly outcome-metric review and + ad-hoc agent/metric deep-dives, over the digest CLI. +- **`/feedback-validate`** — register a pre-registered validation for a prompt / + config edit, or evaluate one whose window has elapsed. Enforces pre-registration + discipline + confounder controls + posterior-band reporting (confirmation bias is + the failure mode it exists to defeat). +- **`/feedback-retrospective`** — open-ended quarterly LLM-driven deep retrospective: + read many invocations end-to-end, surface patterns the deterministic metrics did + not catch, propose promotion candidates. + +The interactive dashboard for this surface (View F) is **deferred to ALP-686** — the +loop is headless-only for now. + +### 9.7 PM-accuracy verification follows story 06e / ALP-129 + +The PM-accuracy + modification-effectiveness metrics (and their +`counterfactual_replays` loader sub-bundle) are gated on the counterfactual replay +engine (ALP-129). They are **verified when story 06e (ALP-887) lands** alongside that +engine — the spine verify above does not assert them, and `metrics-list` simply +omits the gated metric ids until the dependency is present. This is a known, +intentional gap, not a missing check. + diff --git a/docs/runbooks/genesis-cutover.md b/docs/runbooks/genesis-cutover.md index 40b4dbab4..28d510b17 100644 --- a/docs/runbooks/genesis-cutover.md +++ b/docs/runbooks/genesis-cutover.md @@ -34,8 +34,8 @@ re-creates a husk at genesis (a Broker-Owned Fact with no Intent). stream, bracket-stops, greeks, entry-window, recovery sweep), `alphamind-safety-core` (**new** — isolated breach + price-staleness), and the **out-of-process watchdog** `alphamind-safety-core-watchdog` for the safety core - (per ADR-0004). Install scripts: `scripts/install_pipeline_scheduler_service.ps1`, - `scripts/install_monitor_service.ps1`, `scripts/install_safety_core_service.ps1` + (per ADR-0004). Install scripts: `scripts/services/install_pipeline_scheduler_service.ps1`, + `scripts/services/install_monitor_service.ps1`, `scripts/services/install_safety_core_service.ps1` (installs both safety-core + watchdog). ## New account checklist (provision + verify before cutover) diff --git a/docs/runbooks/invocations.md b/docs/runbooks/invocations.md new file mode 100644 index 000000000..d2f353711 --- /dev/null +++ b/docs/runbooks/invocations.md @@ -0,0 +1,164 @@ +# Running invocations + +Manual (`--once`) invocations and the scheduler daemon's run-type schedule. Watching a +run in flight is [monitoring.md](monitoring.md). + +## 3. Running a manual invocation + +Use cases: ad-hoc smoke after a config edit, off-schedule analysis pass, +characterizing a specific run-type's behavior. + +A manual invocation runs in your shell — it does NOT go through the +scheduler service and does NOT honor the schedule. The scheduler service can +keep running in parallel; the two processes share the DB but APScheduler's +overlap-dedup keys off scheduled run-types only, so a manual `--once` won't +collide with the daemon's scheduled fire. + +```bash +set -a && source <(tr -d '\r' < .env) && set +a && \ + uv run python -m alphamind.scheduler run \ + --once \ + --reason "" +``` + +Where `` is one of: +`market_open`, `market_hours_rolling`, `pre_close`, `off_hours_rolling`, +`weekend_saturday`, `weekend_sunday`, `emergency`. + +The process exits when the invocation completes (~25–50 min steady-state in +prod — the full agent roster runs on every run-type; see § 4). All +artifacts land in the same archive layout as scheduled runs — under +`%USERPROFILE%\AlphaMind\archive\\\`. + +**agent_calls telemetry capture is active in production (ALP-907).** Every LLM +agent call (the 9 analysis + decision agents, run inside the SDK subprocess +worker) persists one `agent_calls` table row plus a four-file provenance tree +under +`%USERPROFILE%\AlphaMind\data\provenance\invocations\\agent_calls\\` +(`system_prompt.md`, `output_schema.json`, `tools_definition.json`, +`output.json`; the row's `output_artifact_ref` points at that directory). The +worker opens one telemetry SQLite session per agent call and commits it before +returning. Telemetry persistence is **logged-not-fatal**: a capture/commit error +is logged (`agent_calls telemetry commit failed in subprocess worker` / +`agent_calls capture failed …`) and never fails the agent call or the +invocation. Delivery requires the `agent_calls` table to exist — the combined +feedback-loop migration (ALP-879) must be applied on the prod box first; +until then the insert silently no-ops (swallowed) and the table stays empty. + +**Post-deploy spot-check** (after the migration is applied): run one `--once` +invocation, then confirm the table populated and the tree exists: + +```bash +uv run python -c "import os, sqlite3; \ +db = sqlite3.connect(os.path.expandvars(r'%USERPROFILE%\AlphaMind\data\alphamind.db')); \ +print(db.execute('SELECT count(*), sum(success) FROM agent_calls').fetchone())" +``` + +```powershell +Get-ChildItem "$env:USERPROFILE\AlphaMind\data\provenance\invocations" -Recurse -Filter output.json | Select-Object -First 5 +``` + +**Do not pass `--fresh-start`.** That flag is reserved for the one-time +bootstrap in bootstrap.md § 2.4; it hard-fails when the singletons already exist. + +**A manual `--once` run is NOT observable on the SSE streams.** The +`--once` code path (`_run_once` in `src/alphamind/scheduler/__main__.py`) +runs the invocation in your shell as its own process; it never constructs +the `SSEEventEmitter` or the `/control` + `/events` HTTP server — those exist +only in the daemon path (`_run_daemon`). So while a manual run executes, the +scheduler daemon's `127.0.0.1:8765/events` stream (monitoring.md § 5.1) and the command +center's Live Run dashboard (monitoring.md § 5.3) — which consumes that same stream — show +only heartbeats. They reflect the daemon, and the daemon is idle; your run is +elsewhere. (Don't be fooled into thinking the run stalled: collect → distill +then SSE silence is exactly what a healthy out-of-process run looks like on +those surfaces.) + +To watch a manual `--once` invocation live, use: + +- **The process's own stdout/stderr** — the terminal you launched it in. This + is the richest live view: per-phase `distillation.orchestrator` completions, + per-agent `analysis.*.runner` lines, command execution dispatch, and the final exit + code all stream here. +- **The structured `pipeline.log`** (monitoring.md § 5.4) — `_run_once` does call + `configure_pipeline_logging()`, so records land in the rotating log + alongside the daemon's (filter by the invocation id). +- **The `invocations` DB row** (monitoring.md § 5.5) and **per-invocation archive** (monitoring.md § 5.6), + which are written identically to a scheduled run. + +--- + +## 4. Scheduled invocations (the daemon's job) + +Once `alphamind-scheduler` is running as an NSSM service, APScheduler drives +all scheduled invocations automatically against the NYSE calendar. The +production schedule (timezone `US/Eastern`, defined in +`config/scheduler.yaml`): + +The schedule is **Tier B** (ALP-745): every trigger fires at a distinct +minute. The `as_of` is second-resolution and cron fires land on `:00`, so a +distinct minute means a distinct `as_of` — no two scheduled runs ever share +one (the condition that previously crashed one of two coinciding distillation +passes on the `distillation_ticker_baseline` UNIQUE constraint). + +| Run type | Cron (US/Eastern) | Fires (ET) | Notes | +|------------------------|----------------------------|-------------------------------------------|----------------------------------------------| +| `market_open` | `35 9 * * mon-fri` | 09:35 weekdays | NYSE-gated; fires just after the 09:30 open for fresh live quotes | +| `market_hours_rolling` | `0 13 * * mon-fri` | 13:00 weekdays (single mid-day read) | NYSE-gated; dedup-gated within 30 min | +| `pre_close` | `0 15 * * mon-fri` | 15:00 weekdays | NYSE-gated; sole owner of the close slot | +| `weekend_sunday` | `0 18 * * sun` | 18:00 Sunday | Unconditional | +| `off_hours_rolling` | (unscheduled) | Manual / emergency only | Valid run type; overlay retained, not on cron | +| `weekend_saturday` | (unscheduled) | Manual / emergency only | Valid run type; overlay retained, not on cron | +| `emergency` | (not scheduled) | On-demand, operator-triggered (command-center control → scheduler `trigger_emergency_invocation`) | Cooldown-gated per `config/breach_behavior.yaml`; auto breach-cascade is **not** wired in this build (monitoring.md § 5.7) | + +~3 scheduled invocations per trading weekday (`market_open` + `market_hours_rolling` ++ `pre_close`), ~16 per full trading week including the Sunday run. There is no +longer any same-minute collision for the dedup window to "collapse" — every slot +is distinct by construction. **Typical steady-state wall-clock per invocation is +~25–50 minutes**: deterministic distillation takes ~5–6 min, and the 9 SDK agents +(~1–8 min each, domain researchers serial within their phase) dominate the rest; +the adaptive researcher consuming its full budget pushes a run toward the upper +band. Observed: 29.7 / 40.8 min `market_open`, ~49 min `market_hours_rolling` +(2026-06-09/10). The "~60–180 seconds" figure previously documented here +predates the full agent roster — a sub-5-minute completion now means the run +aborted early, not that it was fast. +An `emergency` invocation is **not** a fast path: it runs the full agent roster — +the same roster as `market_open`, with the heaviest adaptive-researcher budget +(`config/run_types/emergency.yaml`) — so it lands comparable to or slower than a +scheduled run. + +The cadence and time windows live in `config/scheduler.yaml`. Per-trigger +agent budgets live in `config/run_types/*.yaml`. Edits to either take effect +on the next scheduler restart — APScheduler reads them at boot. To apply a +schedule change: + +```powershell +nssm restart alphamind-scheduler +``` + +No DB migration, no data churn — the service restart re-reads configs and +re-arms the jobs. + +**To pause scheduled invocations without taking the service down**, use the +command center's pause control (Dashboard → Controls → Pause), or hit the +loopback verb directly: + +```bash +curl -X POST http://127.0.0.1:8765/control/pause \ + -H "Content-Type: application/json" \ + -d '{"reason":"operator pause via CLI"}' +``` + +Resume: + +```bash +curl -X POST http://127.0.0.1:8765/control/resume \ + -H "Content-Type: application/json" \ + -d '{}' +``` + +Pause/resume mutates an in-process flag; APScheduler skips fires while +paused, and the flag survives the service running but does NOT survive a +restart (a restarted scheduler is unpaused by default). Use the +command-center UI for the audited path — it writes an `OPERATOR_CONSOLE` row +to `activity_log` for each pause/resume. + diff --git a/docs/runbooks/monitoring.md b/docs/runbooks/monitoring.md new file mode 100644 index 000000000..96c041eeb --- /dev/null +++ b/docs/runbooks/monitoring.md @@ -0,0 +1,357 @@ +# Monitoring invocations + +Real-time and after-the-fact observation of pipeline invocations: SSE streams, the +structured log, the `invocations` table, archives, and the agent-watch patterns. + +## 5. Monitoring scheduled invocations + +**Important: production scheduled invocations do NOT emit a `progress.jsonl` +file.** The scheduler uses a no-op progress emitter in production; the +JSONL-stream variant is part of a separate harness that's not on the daemon +code path. Don't try to tail a file under the archive directory hoping to +see phase events — there is none. Use the surfaces below instead. + +The right surface depends on whether you're watching in real time or +investigating after the fact, and whether you want a structured event stream +or free-form log records. + +### 5.1 Real-time: the scheduler's `/events` SSE stream (canonical) + +This is the primary live-monitoring surface for agents and operators — +**for daemon-driven invocations only** (scheduled fires + emergency +cascades). A manual `--once` CLI run is a separate process and does not +publish here; watch it via its process stdout / `pipeline.log` instead +(invocations.md § 3). + +```bash +curl -N http://127.0.0.1:8765/events +``` + +Emits SSE frames for each of these event types: + +| Event | When | +|-------------------------|----------------------------------------------------------| +| `invocation_started` | A daemon-driven invocation begins (scheduled / emergency; NOT a manual `--once` CLI run) | +| `phase_transition` | The orchestrator advances to a new pipeline phase | +| `agent_started` | An SDK call begins (analyst, strategist, PM, etc.) | +| `agent_succeeded` | The SDK call returned a valid response | +| `agent_retrying` | Parse/validation failure; the harness is retrying | +| `agent_failed` | All retries exhausted; the invocation will abort | +| `invocation_ended` | The invocation reached a terminal state (success/failure)| +| `next_trigger_changed` | APScheduler recomputed its next fire | +| `heartbeat` | 15 s idle keep-alive | + +**For agents:** call this stream via the `Monitor` tool so each emitted SSE +frame becomes a notification. Filter with grep/jq for the events you care +about. Silence beyond 15 s indicates the SSE stream is wedged (scheduler +process crashed, network filter on the loopback interface, etc.) — not that +"nothing is happening." Heartbeats are the proof-of-life signal. + +Cross-field invariants worth knowing: + +- `phase_transition` is forward-only — an invocation never goes backward + through phases. +- `agent_succeeded` and `agent_failed` are mutually exclusive per + `(invocation_id, agent_name)` — exactly one fires per agent per + invocation. +- `invocation_started` for an emergency-triggered invocation carries + `run_type: emergency` and the `invocation_id` returned by the + `trigger_emergency_invocation` control verb. + +### 5.2 Real-time: the monitor's `/events` SSE stream + +The continuous monitor publishes its own SSE stream on `127.0.0.1:8766`. In the +**current build** it emits fills plus an idle keep-alive: + +```bash +curl -N http://127.0.0.1:8766/events +``` + +| Event | When | +|-----------------|-----------------------------------------------| +| `fill_received` | A fill arrived from the trade-updates stream | +| `heartbeat` | 15 s idle keep-alive | + +Watch this stream to see fills landing between invocations. + +**What is *not* on this stream (and why).** The monitor's SSE emitter still +*defines* `websocket_connected` / `websocket_disconnected` / `breach_detected` / +`emergency_invocation_triggered` / `greeks_refreshed`, but their producers are +not composed into the running monitor, so none of them are emitted: + +- **`breach_detected` / `emergency_invocation_triggered`** — breach detection + + price-staleness moved out of the monitor into the isolated, **log-only safety + core** (ADR-0004 / ALP-857). The safety core has no SSE surface and writes + nothing; it logs to `safety_core.log` (troubleshooting.md § 8.9, its breach + price-staleness signals subsection). The monitor proper no longer + runs a breach loop. +- **`greeks_refreshed`** — the greeks-refresh task runs, but its SSE emit adapter + is not wired; confirm greeks via `monitor.log` / the per-position diagnostics. +- **`websocket_connected` / `websocket_disconnected`** — defined but never + emitted anywhere in the current tree; track websocket health via + `monitor.log` / `monitor.err.log` instead. + +### 5.3 Real-time: the command center Live Run dashboard + +`http://localhost:8090/` → log in → Live Run. The dashboard merges both +daemons' SSE streams + decorates them with activity-log rows and per-agent +status. Best surface when an operator is sitting at the console. + +### 5.4 After-the-fact: the structured pipeline log + +```powershell +Get-Content "$env:USERPROFILE\AlphaMind\logs\pipeline.log" -Tail 200 +``` + +Per-line structured records. The `.log` file is the structured Python-logger +emit (one JSON-ish record per line including `invocation_id`, `phase`, +`agent`, and any exception with full traceback). The `.out.log` and +`.err.log` siblings are NSSM's raw stdout/stderr capture — useful only when +the daemon crashed before the structured logger could write. + +To filter to a single invocation across the three log files: + +```powershell +Select-String -Path "$env:USERPROFILE\AlphaMind\logs\pipeline.*.log" -Pattern "inv-20260527T143000Z-" +``` + +### 5.5 After-the-fact: the `invocations` DB table + +Each invocation gets exactly one row recording its lifecycle: + +```bash +uv run python -c " +import sqlite3, os +db = sqlite3.connect(os.path.expandvars(r'%USERPROFILE%\AlphaMind\data\alphamind.db')) +db.row_factory = sqlite3.Row +for row in db.execute('SELECT invocation_id, trigger_type, trigger_source, trigger_reason, start_at, fill_collection_completed_at, command_execution_completed_at, git_sha_at_invocation, staleness_flag FROM invocations ORDER BY start_at DESC LIMIT 20').fetchall(): + print(dict(row)) +" +``` + +The table does **not** carry a single `status` column. Completion is read +from the phase timestamps: a row with a non-null `command_execution_completed_at` +succeeded through command execution; a row with `start_at` set but +`command_execution_completed_at` NULL either is still in-flight or aborted mid-pipeline +(cross-check the SSE stream / `pipeline.log`). Useful columns: +`trigger_type` (how the invocation was launched — `scheduled`, `manual`, +or `emergency`), `trigger_source` (the run-type / origin label — for +scheduled fires `market_open`, `market_hours_rolling`, `pre_close`, +`weekend_sunday`, or `off_hours_rolling`; `cli` for a +manual `--once` run; `operator_console` for a command-center action; +`continuous_monitor` for an emergency invocation), +`trigger_reason` (free-form), `fill_collection_completed_at` / `command_execution_completed_at` +(lifecycle), `git_sha_at_invocation` (the repo HEAD when the invocation +ran — useful for confirming which code version a run executed under), +`command_execution_summary_json` (PM command-submission detail), +`staleness_flag`, and `resolved_config_snapshot_path` (points into the +archive directory). + +### 5.6 Per-invocation archive directory + +Every invocation writes a directory under +`%USERPROFILE%\AlphaMind\archive\\\` (UTC date, +not ET). The invocation ID format is `inv-YYYYMMDDTHHMMSSZ-<8-hex>`. + +Contents include `resolved_config.json` (the full resolved config tree as it +existed for that invocation — useful when reconciling "what config did this +run actually use") and per-agent diagnostic subdirs under +`analysis//` and `decision//` (each carries the assembled +input bundle, the prompt sent to the SDK, the SDK response, and a +`metadata.json` with token + wall-clock + tool-call counts). + +This is the most expensive surface to read but the highest fidelity. Use +it when § 5.4 / § 5.5 don't tell you why an agent produced a malformed +brief. + +### 5.7 What to watch for + +**Stalled agent.** An `agent_started` event with no matching +`agent_succeeded` / `agent_failed` after that agent's configured +`latency_budget_seconds` (in `config/run_types/.yaml`). The orchestrator +will eventually time out and emit `agent_failed`, but if you're watching +live and the budget is generous, the stall is visible first as +unexplained silence in the SSE stream. + +**Failed invocation.** An `invocation_ended` event reporting a non-success +terminal state. The `invocations` row has **no `status` or `exit_reason` +column** — a failed run is identified by `command_execution_completed_at` being NULL +(often `fill_collection_completed_at` too). Tail `pipeline.log` (and the daemon's +`pipeline.err.log`) for the traceback that names the failing layer. + +**Thesis-resolution skip warnings (expected, not a fault).** Every deliberative +invocation runs a thesis-resolution step (`phase: thesis_resolution`) between +fill collection and snapshot assembly: it moves each closed-position thesis +`ACTIVE → RESOLVED`, authoring the component outcomes + resolution category the +snapshot's recent-resolutions feed and thesis-quality aggregates read. The step +resolves each eligible thesis **independently** — a per-thesis data gap logs a +`WARNING` in `pipeline.log` (naming the `thesis_id` + `position_id` + the +specific gap) and **skips** that thesis, leaving it `ACTIVE`, then continues +resolving the rest. It never aborts the invocation. A skip-warning is therefore +**expected operator-visible behavior, not an error** — most commonly an +option-lifecycle-closed thesis (expiry / assignment / exercise) that carries no +`POSITION_CLOSED` activity-log entry yet, so the exit method can't be read. The +skipped thesis stays `ACTIVE` and is retried next invocation. Investigate only +if the *same* thesis logs the skip on every invocation indefinitely (a stuck +thesis, not a transient gap). + +**Skipped invocations.** When you expected a scheduled fire (per invocations.md § 4) and +no `invocation_started` event arrived: + +- **NYSE holiday or early close** — weekday fires are NYSE-calendar-gated. + No fire on a market holiday is expected, not a bug. +- **Daemon paused** — check the command center Controls panel. Pause flag + doesn't survive a restart. +- **Dedup suppressed it** — the 30-minute dedup window suppresses a + `market_hours_rolling` fire if some invocation completed phase 2 within the + lookback (e.g., a manual or emergency run just before the 13:00 slot). Under + Tier B no two scheduled triggers share a minute, so this only happens around + off-schedule runs, never between two scheduled fires. The + `next_trigger_changed` events on the SSE stream show what APScheduler + thinks comes next. + +**Emergency invocations.** In the current build these are **operator-initiated**, +not auto-cascaded from a breach. The expected sequence: + +1. Operator: command center → Controls → Trigger emergency invocation (or + `POST` the scheduler's `trigger_emergency_invocation` verb). This writes an + `EMERGENCY_INVOCATION_REQUESTED` row to `activity_log`. +2. Scheduler: the `emergency_receiver` task polls `activity_log`, picks up the + request, and — unless the 30-minute cooldown + (`config/breach_behavior.yaml` § `emergency_invocation_cooldown_minutes`; + a `margin_call` request bypasses it) suppresses it — dispatches one emergency + invocation. +3. Scheduler `/events`: `invocation_started` with `run_type: emergency`, then the + usual phase + agent events through `invocation_ended`. The `invocations` row + records `trigger_type: emergency` / `trigger_source: continuous_monitor`. + +If you requested an emergency and no `invocation_started` follows within a few +seconds, the cooldown gate suppressed it — check the cooldown window and the +recent `activity_log` rows for the prior emergency. + +> **Auto breach→emergency cascade is not wired in this build.** The monitor no +> longer runs a breach loop, and the isolated safety core only logs + heartbeats +> and writes nothing (ADR-0004 / ALP-857), so nothing currently enqueues an +> `EMERGENCY_INVOCATION_REQUESTED` row on a breach. The auto-trigger evaluator +> still exists in the tree but is not composed into the running monitor — so a +> breach surfaces in `safety_core.log` (troubleshooting.md § 8.9, its breach + price-staleness signals subsection) and via the broker-enforced +> bracket floor, not as an automatic emergency invocation. + +**Cost / token regression.** The `metadata.json` files under each agent's +diagnostic subdir record token counts under a `tokens_used` object +(`input_tokens` / `cache_read_tokens` / `cache_write_tokens` / +`output_tokens`). A healthy production run has the +bulk of input volume landing in `cache_read_tokens` (the prompt cache is +hit). If `input_tokens` is large and `cache_read_tokens` is near zero, +something invalidated the cache — investigate before the next invocation +amplifies the cost. + +### 5.8 Agent monitoring pattern + +The canonical pattern for an agent asked to "watch the next scheduled +invocation": + +1. Look up the next fire time from invocations.md § 4's schedule table relative to current + ET. Confirm via `next_trigger_changed` events on the SSE stream. +2. Open `curl -N http://127.0.0.1:8765/events` via the `Monitor` tool so SSE + frames become notifications. +3. Wait for `invocation_started`. Record the `invocation_id`. +4. Watch for `phase_transition` → `agent_started` → `agent_succeeded` pairs + covering every phase through `command_execution`. +5. Wait for `invocation_ended`. On success, summarize cost + wall-clock + from the archive's per-agent `metadata.json` files. On failure, fetch + the traceback from `pipeline.log` / `pipeline.err.log` — the + `invocations` row carries no `exit_reason`; a NULL `command_execution_completed_at` + is the failure signal. +6. Surface the `invocation_id` to the operator in any report — it's the + entry point for follow-up archive inspection. + +**SSE caveat (why § 5.9 exists).** In practice, consuming the § 5.1 SSE +stream through the `Monitor` tool has proven an unreliable per-agent +notifier — `agent_*` frames do not always surface as notifications — and a +manual `--once` run does not publish to the daemon stream at all (invocations.md § 3). For a +robust, granular per-agent watch, and the *only* live option for manual runs, +prefer the archive-based pattern in § 5.9. + +### 5.9 Granular agent-completion watch via the archive (Monitor tool) + +Each analysis / decision agent writes its diagnostic subdir +(`analysis//` or `decision//`) **atomically on completion**, so +the appearance — or mtime change, on a retry — of that agent's +`metadata.json` is a high-fidelity per-agent finish marker carrying +`success`, `stop_reason`, `wall_clock_seconds`, `retry_count`, and the +token / cache counts. Unlike the SSE stream this works identically for +scheduled and manual `--once` runs and reads straight off the WAL DB + +filesystem. + +The canonical watcher is **`scripts/ops/watch_invocation.sh`** — pass it to the +`Monitor` tool and it streams one notification per milestone, self-terminating on +a terminal state (including one that already happened *before* it attached). +Select the invocation one of three ways: + +```bash +# newest scheduled fire of a run-type (default lookback = now-5min): +bash scripts/ops/watch_invocation.sh --scheduled market_hours_rolling +# a manual --once run, by a substring of its --reason: +bash scripts/ops/watch_invocation.sh --reason "my run reason" +# attach to a specific invocation id: +bash scripts/ops/watch_invocation.sh --inv inv-20260602T170000Z-193173de +``` + +Optional `--since ` widens/narrows the discovery window; `--tz-offset +` (default `6` = MDT) sets the prod-log local offset used to scope abort lines +(pass `7` for MST in winter). Paths are overridable via the `DB` / `LOG` / `ERR` / +`ARCHROOT` env vars (prod defaults are baked in). + +It emits, as they happen: `FILL-COLLECTION-COMPLETE` (ingestion + fill-collection summary); +`ACT …` for every `activity_log` row of the run. A `RECONCILIATION_ALERT` row now +means only an **orphan fill** that could not integrate (a poison-pill fill against a +terminal/over-filled position, ALP-761) — the reconcile-adjudication path is deleted +(ALP-854 / ADR-0001), so a positions/cash mismatch is silently rebuilt, never alerted. +The watch also emits all command-execution actions (`ORDER_SUBMITTED` / `PM_DECISION` / +`CAPITAL_RESERVED` / …); +`DISTILLATION` when `composite_state` is written; `AGENT / success=… +wall=… stop=…` per agent (the strategist's line is prefixed `>>> STRATEGIST`); +`FILLS` on each `unprocessed`→`processed`/`quarantined` transition; `FAULT(log|err)` +on a fresh non-benign traceback; and the terminal `COMMAND-EXECUTION-COMPLETE` (+ a final +positions / cash / fills dump) or `ABORT`. It is **silent while waiting** — no +routine "still waiting" ticks — except a one-shot `WARN scheduler port 8765 DOWN`. + +**Why it covers "silence ≠ success."** A healthy run is proven only by +`COMMAND-EXECUTION-COMPLETE` (the sole positive success signal — there is no `status` column, +§ 5.5). A hard crash writes **no** `metadata.json`, so the fault/abort legs catch +what the per-agent leg can't: fresh `Traceback`/`CRITICAL`/`ERROR` in +`pipeline.log` + `pipeline.err.log` (with the benign Windows asyncio-teardown noise +— `ProactorBasePipeTransport`, `WinError 121`, `no close frame`, `ResourceWarning` +— filtered out), plus the driver's `scheduled trigger= failed` / +`scheduler exited with error`. And the **at-attach terminal check** handles a run +that already finished or aborted before the watch started: it checks +`command_execution_completed_at` and greps the log (scoped to this run's own start + trigger) +once up front, emitting `ALREADY-COMPLETE` / `ALREADY-ABORTED` and exiting instead +of polling a dead invocation. (Before this check, a watcher armed after a fast-fail +abort polled a dead 17:00Z run for 20 min — 2026-06-02.) + +Gotchas baked into the script (worth knowing when reading its output): + +- Some agents (`synthesizer`, `strategist`) record `tool_calls_used` instead of + `retry_count`; a missing `retry` field is benign, not a failure. +- The `FILLS unprocessed N→M` leg confirms fill-collection ingestion before analysis even + starts — how both the 2026-06-01 poison-pill wedge and the 2026-06-02 ALP-824 + cross-writer fix were verified. +- A manual `--once` run does not publish to SSE (§ 5.1), and its + `scheduler exited with error` may land only in the launching shell's stderr (not + `pipeline.log`) — so for a manual run the **launching process's exit code** stays + the authoritative terminal signal; the watcher is the live milestone view. +- Set the `Monitor` `timeout` generously (or `persistent: true`) — a full-roster + run takes ~25–50 min (invocations.md § 4); the watch exits itself on the terminal state. +- The ACT field separator is a TAB, not a control character: the sqlite3 CLI + (≥ 3.49; prod runs 3.51) escapes control characters on output, so `char(31)` + arrives as the literal text `^_` — the field split silently failed and every + ACT line emitted empty until 2026-06-10. The same CLI emits CRLF; `q()` strips + the `\r` (command substitution hides this, `while read` pipelines don't). +- The fault legs filter benign noise **per traceback block**, not per line: the + SDK-subprocess teardown noise (`_ProactorBasePipeTransport.__del__` → + `ValueError: I/O operation on closed pipe`, after every agent completion) puts + its benign marker and its bare `Traceback` header on separate lines, so a + per-line filter leaks a spurious FAULT pair per agent. A real traceback emits + one line: `FAULT(log|err) Traceback … => `. + diff --git a/docs/runbooks/services.md b/docs/runbooks/services.md new file mode 100644 index 000000000..f3e032625 --- /dev/null +++ b/docs/runbooks/services.md @@ -0,0 +1,180 @@ +# Services: table, restarts, ports and paths + +The seven NSSM services, restarting a single one, and the ports/paths reference. +The full-cycle deploy order is [update-loop.md](update-loop.md), and its +service-stop precondition applies to any restart that stops the monitor or +safety core — evaluate it there before stopping either. + +The seven NSSM services are the supported runtime form on prod: + +| Service | Module entry | Log basename | +|----------------------------------|--------------------------------------------------|--------------------| +| `alphamind-collector` | `python -m alphamind.collector run` | `collector.*.log` | +| `alphamind-scheduler` | `python -m alphamind.scheduler run` | `pipeline.*.log` | +| `alphamind-monitor` | `python -m alphamind.execution.continuous_monitor run` | `monitor.*.log` | +| `alphamind-monitor-watchdog` | `python -m alphamind.execution.continuous_monitor watchdog` | `monitor_watchdog.*.log` | +| `alphamind-safety-core` | `python -m alphamind.execution.continuous_monitor.safety_core run` | `safety_core.*.log` | +| `alphamind-safety-core-watchdog` | `python -m alphamind.execution.continuous_monitor.safety_core watchdog` | `safety_core_watchdog.*.log` | +| `AlphaMindCommandCenter` | `python -m alphamind.command_center` | `command_center.*.log` | + +**Safety core + out-of-process watchdog (ADR-0004 / ALP-857).** Breach detection ++ price-staleness — the lone safety item with **no broker floor** — is isolated +into the `alphamind-safety-core` service, which reads the **broker snapshot** + +**REST latest-quote snapshots** (polled, no second market-data websocket — +ALP-940), beats a heartbeat file under +`%USERPROFILE%\AlphaMind\logs\safety_core.heartbeat`, and **writes nothing** to +the DB. The safety core polls REST quotes rather than opening its own +`StockDataStream`: Alpaca's free IEX plan allows exactly **one** authenticated +market-data websocket per account, so a second stream collided with the +monitor's (`connection limit exceeded`) and starved both feeds. REST latest-quote +is not connection-limited, so dropping it frees the single market-data connection +for the `alphamind-monitor` proper. The `alphamind-safety-core-watchdog` service is a **dedicated +out-of-process** watchdog: it probes that heartbeat file and runs +`nssm restart alphamind-safety-core` when it goes stale (a loop-resident watchdog +cannot catch a freeze of its own loop). The `alphamind-monitor` proper now runs +only **precision/data** tasks (options stops, greeks, entry-window, fill stream + +recovery sweep) and is fail-safe under the broker floor — a frozen monitor +degrades precision while positions stay broker-protected. The watchdog must run +under the **same Windows account** as the safety core so its `nssm restart` has +permission. + +**Monitor watchdog (ALP-941).** The monitor gets the same treatment: its +supervisor beats `%USERPROFILE%\AlphaMind\logs\monitor.heartbeat` while its +event loop turns, and the `alphamind-monitor-watchdog` service probes that file +and runs `nssm restart alphamind-monitor` when the beat is older than +`monitor_watchdog_tick_seconds × watchdog_cadence_multiplier` (15 s × 10 = +150 s by default). The monitor's **in-process** stall watchdog runs on the very +event loop it guards, so a frozen loop (the 2026-06-09 wedge: 8 h of no fills +with NSSM "Running") starves it — only an out-of-process watchdog bounds that +failure. A frozen loop also self-captures its blocking frame: the faulthandler +deadman dumps the frozen stack to `monitor_faulthandler.log` before the restart +lands. The same-account rule applies to this watchdog too. + +Logs land under `%USERPROFILE%\AlphaMind\logs\`. DB is +`%USERPROFILE%\AlphaMind\data\alphamind.db`. Invocation archives land under +`%USERPROFILE%\AlphaMind\archive\\\`. + + +## 7. Restarting a single service + +For any one service, in PowerShell as admin: + +```powershell +nssm restart +Get-Service +``` + +The valid service names are: + +- `alphamind-collector` +- `alphamind-scheduler` +- `alphamind-monitor` +- `alphamind-monitor-watchdog` +- `alphamind-safety-core` +- `alphamind-safety-core-watchdog` +- `AlphaMindCommandCenter` (note: mixed case, no hyphen) + +NSSM's `restart` is `stop` + `start`. The install scripts leave NSSM's default +stop-method timeouts in place (no explicit `AppStopMethod*` is set); in practice +a stop that sits in `StopPending` past ~30 s is wedged — see troubleshooting.md § 8.3. On stop, +SIGINT/SIGTERM **triggers** a co-operative shutdown: each in-flight task gets a +`CancelledError` and a bounded grace window to wind down — scheduler **10 s**, +monitor **5 s**, command center **30 s** (`supervisor_shutdown_timeout_seconds` +per daemon; the scheduler supervisor uses a manually-bounded wait rather than +`TaskGroup.__aexit__` so the timeout actually bounds exit). Tasks that don't exit +in the window are abandoned and NSSM/the SCM force-terminates the process; there +is no separate SIGTERM "fallback" the supervisor itself fires. + +A plain `nssm restart ` works for any single service with all the +others running: no service declares an SCM dependency on another, so the SCM +never refuses the stop half of the restart. That is a deliberate topology +invariant (ALP-945, the 2026-06-10 incident — a `DependOnService` edge from +the command center made the SCM refuse every monitor-watchdog restart, +wedging the monitor for 42 minutes): **never declare an SCM dependency on a +watchdog-supervised service.** + +**One restart gotcha that cost real diagnosis time (2026-06-02 monitor wedge):** + +- **`Get-Service … Running` does NOT mean healthy.** NSSM reports `Running` + while the *wrapper* process is alive. Two liveness layers now apply: + - **The safety core** (breach + price-staleness, the no-floor item) is + supervised by the **dedicated out-of-process `alphamind-safety-core-watchdog`** + (ADR-0004 / ALP-857): it probes the core's heartbeat file + (`%USERPROFILE%\AlphaMind\logs\safety_core.heartbeat`) and runs + `nssm restart alphamind-safety-core` when the beat goes stale past + `breach_evaluation_cadence_seconds × watchdog_cadence_multiplier` (default + 10×). A loop-resident watchdog cannot catch a freeze of its own loop, so the + watchdog is its own process — a frozen safety core is restarted, not masked. + - **The monitor proper** keeps its in-process per-cadence stall watchdog for + its precision/data tasks (`cadence_seconds × watchdog_cadence_multiplier` + forces `os._exit(1)` → NSSM restart) — but that watchdog runs on the very + event loop it guards, so a freeze of the **loop itself** (the 2026-06-09 + ALP-941 wedge) starves it. The **dedicated out-of-process + `alphamind-monitor-watchdog`** closes that gap: it probes the monitor's + heartbeat file (`%USERPROFILE%\AlphaMind\logs\monitor.heartbeat`) and runs + `nssm restart alphamind-monitor` when the beat goes stale past + `monitor_watchdog_tick_seconds × watchdog_cadence_multiplier` (default + 15 s × 10 = 150 s). On a loop freeze the faulthandler deadman also dumps + the frozen stack to `monitor_faulthandler.log` — read it to identify the + blocking frame **before** diagnosing further. The monitor no longer runs + breach detection, so a monitor wedge **degrades precision while positions + stay broker-protected** (the broker floor + the isolated safety core + hold). The **one residual exception is the control-surface HTTP server**, + registered `watched=False` (it blocks in `await server.serve()`); that + task is not auto-recycled in-process, though a full loop freeze that takes + it down now is (via the external watchdog). + + For everything, confirm a restart by the process **StartTime** + (fresh PID / StartTime) **and** a behavioral signal — the daemon's SSE + heartbeat (`8765` / `8766`), the safety-core / monitor heartbeat files' + mtime advancing, fresh structured-log lines, fills flowing — never by + `Get-Service Status` alone. Read the StartTime with: + ```powershell + $p = (Get-CimInstance Win32_Service -Filter "Name=''").ProcessId + (Get-Process -Id $p).StartTime + ``` + See troubleshooting.md § 8.9 for the fill-stream and underlying-stream auto-recovery details. + +**When the operator can restart a single service without coordinating the +others:** + +| Restart this | When | Side effects | +|---------------------|---------------------------------------------------------------|-----------------------------------------------------------------------------------| +| collector | After a `config/collector_schedule.yaml` or `config/data_sources.yaml` edit | Up to one cadence-cycle of dropped vendor reads | +| scheduler | After a `config/scheduler.yaml`, `config/run_types/*.yaml`, or `config/main.yaml` edit | Pause flag cleared; an in-flight invocation is cancelled mid-pipeline | +| monitor | After a `config/continuous_monitor.yaml` edit affecting precision/data tasks | Halt-mode flag persists; in-flight fill-stream reconnects from the recovery path. Its watchdog tolerates a restart within the stall bound (150 s default); for a longer outage stop `alphamind-monitor-watchdog` first | +| monitor-watchdog | After a `monitor_watchdog_tick_seconds` or `watchdog_cadence_multiplier` edit | None to positions; the monitor is unsupervised against loop freezes for the restart window | +| safety-core | After a `config/continuous_monitor.yaml` cadence/threshold or a `config/guardrails.yaml` / profile `gross_exposure_pct` / `position_max_size_pct` edit | Brief gap in breach/staleness detection only; positions stay broker-protected. Its watchdog tolerates a restart within the stall bound; for a longer outage stop the watchdog first (else it `nssm restart`s the core mid-restart) | +| safety-core-watchdog | After a `watchdog_cadence_multiplier` edit | None to positions; the safety core is unsupervised for the restart window | +| AlphaMindCommandCenter | After a frontend rebuild or alert-rule edit | Active SSE clients reconnect; operator sessions persist if `COMMAND_CENTER_SESSION_SECRET` is pinned (bootstrap.md § 2.5), else are invalidated | + +**When to use the full update loop (update-loop.md § 1) instead of a single restart:** + +- Any pull from main that touches `src/`, `pyproject.toml`, `uv.lock`, or a + migration file. (Always run update-loop.md § 1 in this case — start by checking + `git log --oneline ORIG_HEAD..` for migrations before assuming a single + service restart suffices.) +- Anything you're uncertain about. The full loop is ~3 min on a clean run + and idempotent — over-restarting costs nothing. + +--- + + +## 10. Reference: ports and paths + +| What | Where | +|------------------------------|------------------------------------------------------------------| +| Scheduler `/control` + `/events` | `127.0.0.1:8765` (loopback only) | +| Monitor `/control` + `/events` | `127.0.0.1:8766` (loopback only) | +| Command center API + UI | binds `127.0.0.1:8090` (loopback only); **browse via `http://localhost:8090`** — WebAuthn relying-party id is `localhost` | +| Safety core heartbeat (no port) | `%USERPROFILE%\AlphaMind\logs\safety_core.heartbeat` (file; the out-of-process watchdog probes it — the safety core has no control port and writes nothing to the DB) | +| Monitor heartbeat (file) | `%USERPROFILE%\AlphaMind\logs\monitor.heartbeat` (the `alphamind-monitor-watchdog` service probes it; written by the monitor supervisor's watchdog loop — ALP-941) | +| Monitor freeze stack dump | `%USERPROFILE%\AlphaMind\logs\monitor_faulthandler.log` (the faulthandler deadman dumps the frozen main-thread stack here when the monitor's event loop wedges) | +| Production DB | `%USERPROFILE%\AlphaMind\data\alphamind.db` | +| Daemon logs | `%USERPROFILE%\AlphaMind\logs\.{out,err,log}.log` (`safety_core.*`, `safety_core_watchdog.*` for the safety-core services; `monitor_watchdog.*` for the monitor watchdog) | +| Invocation archives | `%USERPROFILE%\AlphaMind\archive\\\` | +| Agent-call provenance | `%USERPROFILE%\AlphaMind\data\provenance\invocations\\agent_calls\\` (one dir per LLM agent call; `system_prompt.md` + `output_schema.json` + `tools_definition.json` + `output.json`; paired with one `agent_calls` table row) | +| Retrospective reports | `\data\retrospective_reports\\report.md` (markdown body; the `retrospective_reports` row carries the metadata + `report_file_ref`) | +| Feedback-loop CLIs (feedback-loop.md § 9) | `python -m alphamind.feedback_loop.{digest,validation,retrospective}.cli`; spine verify `scripts/verify/verify_feedback_loop.py` | +| Config files | `\config\*.yaml` (incl. `feedback.yaml` sample-size floors, `digest.yaml` shift thresholds), `\config\run_types\*.yaml`, `\config\profiles\*.yaml` | +| `.env` | `\.env` | diff --git a/docs/runbooks/troubleshooting.md b/docs/runbooks/troubleshooting.md new file mode 100644 index 000000000..71e9c9280 --- /dev/null +++ b/docs/runbooks/troubleshooting.md @@ -0,0 +1,266 @@ +# Troubleshooting — common operational gotchas + +The living gotchas doc: symptom-indexed diagnoses accumulated from prod experience. +Agents append new gotchas here as they hit them (see the `operate-prod` skill's +self-correction duty). + +## 8. Common operational gotchas + +### 8.1 Service won't start, `.err.log` mentions `NoSuchTableError` + +Migrations are behind. Run update-loop.md § 1.3–1.5. + +### 8.2 Command center returns 404 at `/` + +Frontend bundle missing. Run bootstrap.md § 2.3 (just the `bun install` + `bun run build` +part); no service restart needed — FastAPI's `StaticFiles` mount picks up +the new `dist` bundle on the next request. + +### 8.3 NSSM stop hangs in `StopPending` past 30 s + +The supervisor is blocked on a task that won't cancel gracefully within its +shutdown grace window (services.md § 7). Force the issue: + +```powershell +nssm kill # SIGKILL equivalent +Get-Service # should now be Stopped +``` + +Then investigate the hang in the `.out.log` / `.err.log` before restarting. +A regular hang here is a bug worth filing. + +### 8.4 `gh pr checks` says CI is green but a service won't start + +CI doesn't exercise the Windows service supervisors. A Linux-only behavior +in the merged code (POSIX `os.fsync` on a directory handle is the classic +example) can pass CI and crash on prod. Tail the failing service's +`.err.log` and file the issue against `main`. + +### 8.5 Pipeline invocation aborts with a Claude SDK auth error 5 minutes in + +`CLAUDE_CODE_OAUTH_TOKEN` is set but carries a trailing `\r` from a bare +`source .env` under Git Bash. Re-source via the `tr -d '\r'` form (bootstrap.md § 2.1 +CRLF gotcha) before running manual invocations. Services bypass this — they +read the env from NSSM's `AppEnvironmentExtra` so they're unaffected. + +### 8.6 Operator browser session invalidated after every restart + +`COMMAND_CENTER_SESSION_SECRET` is not pinned on the service environment. +Run the `nssm set AlphaMindCommandCenter AppEnvironmentExtra ...` step from +bootstrap.md § 2.5. Existing sessions stay invalidated this one time; subsequent +restarts preserve them. + +### 8.7 Looking for `progress.jsonl` under an invocation archive and not finding one + +Expected — monitoring.md § 5 explains. Prod uses the no-op progress emitter; the JSONL +artifact is part of a different harness. Use the SSE stream (monitoring.md § 5.1), the +structured pipeline log (monitoring.md § 5.4), or the `invocations` table (monitoring.md § 5.5) +instead. + +### 8.8 Scheduled invocation didn't fire when invocations.md § 4's schedule said it would + +Walk monitoring.md § 5.7's "Skipped invocations" checklist: NYSE holiday, daemon paused, +or dedup suppressed it (only around an off-schedule run — under Tier B no two +scheduled triggers share a minute). The `next_trigger_changed` events on the +SSE stream are the source of truth for what the scheduler thinks comes next. + +### 8.9 Monitor wedge — fill-stream or underlying-stream silent (connected-but-not-delivering) + +#### Current state (post-ALP-825 hardening) + +Both the fill (trade-updates) stream and the underlying-price stream now have +**two** independent auto-recovery layers: + +1. **Budget-neutral reconnect on RTH silence.** Each stream runs a + `StreamActivityMonitor` that beats the stall watchdog on every poll slice and + raises `StreamStalledError` when the market is open and no frames arrive within + the configured timeout (`fill_stream_stale_timeout_seconds = 900 s` for the fill + stream; `underlying_stream_stale_timeout_seconds = 60 s` for the price stream). + The consumer catches `StreamStalledError` and rebuilds the stream + **budget-neutrally** — without consuming a reconnect attempt — so a persistently + silent stream keeps recovering rather than exhausting its budget and exiting. REST + recovery + the fill-backfill backstop re-capture any gap from a fill-stream + reconnect. Off-hours, the silence check is disengaged and the clock resets so the + closed-market gap is not charged against the first RTH window. + +2. **Per-cadence stall watchdog.** Every watched task drives its loop through + `supervised_loop`, which beats the watchdog at the top of each iteration. A task + that beats on every poll slice (including idle slices between frames) keeps the + watchdog satisfied. A genuinely-blocked task that stops beating trips + `os._exit(1)` within `cadence_seconds × watchdog_cadence_multiplier` (the + `watchdog_cadence_multiplier` default is 10×), and NSSM auto-restarts the + monitor process. This backstop catches a task wedge that the in-stream reconnect + could not resolve — but it runs on the monitor's own event loop, so it cannot + catch a freeze of the loop itself. + +3. **Out-of-process watchdog on the whole event loop (ALP-941).** The supervisor + beats `monitor.heartbeat` each watchdog-loop pass; the separate + `alphamind-monitor-watchdog` service probes the file and runs + `nssm restart alphamind-monitor` when the beat is older than + `monitor_watchdog_tick_seconds × watchdog_cadence_multiplier` (150 s default). + This is the only layer that catches a **frozen event loop** (the 2026-06-09 + wedge: loop blocked, process alive, layers 1–2 starved with it). When it + fires, `monitor_faulthandler.log` holds the frozen main-thread stack — the + faulthandler deadman dumped it before the restart — so the blocking frame is + preserved for diagnosis. + +**Operator's primary action** after any suspected monitor wedge: + +1. Confirm a fresh PID / StartTime (the watchdog fired and NSSM auto-restarted): + ```powershell + $p = (Get-CimInstance Win32_Service -Filter "Name='alphamind-monitor'").ProcessId + (Get-Process -Id $p).StartTime + ``` +2. Confirm a behavioral signal — an `8766` heartbeat within ~18 s of connecting + (the SSE keep-alive cadence is 15 s): + ```bash + curl -N 127.0.0.1:8766/events + ``` +3. If the process PID/StartTime is fresh and the heartbeat is live, **auto-recovery + succeeded** — no manual restart is needed. Confirm fills are flowing again + (check `fill_records` for rows dated today; check `monitor.log` for fresh + trade-stream connect lines). + +**Manual services.md § 7 restart is the fallback** when auto-recovery is itself suspect (e.g. +the StartTime is stale, no `8766` heartbeat is appearing, or `monitor.err.log` +shows a `reconnect budget exhausted` entry): `nssm stop alphamind-monitor` + +`nssm start alphamind-monitor` — no other service needs touching. Confirm by +StartTime + heartbeat, not `Get-Service`. + +#### Pre-fix history (2026-06-02 — for reference only) + +Before ALP-825: the continuous monitor's Alpaca trade-updates websocket failed on +`OSError: [WinError 121] The semaphore timeout period has expired`, and the +`websockets` library reconnect-looped *internally* without ever raising — so the +fill consumer parked on `await queue.get()` and starved. The process stayed +`Running` (NSSM only restarts on process exit, and the in-process watchdog didn't +trip) and captured **no fills for ~11 h**, with real-time breach/stop monitoring +off the whole time. ALP-819 fixed the fill-stream silent-wedge path; ALP-832 +applied the same fix to the underlying-price stream; ALP-826 added the per-cadence +watchdog backstop. + +The old detection signals (if you ever see them again, the post-fix recovery path +above should already be firing): + +- `monitor.err.log` shows repeated `trading stream websocket error, restarting + connection: no close frame received or sent` + `OSError: [WinError 121]`. +- `fill_records` has no rows for fills you know landed on Alpaca, and the matching + local order rows stay `PENDING` with `filled_quantity = 0`. +- `curl -N 127.0.0.1:8766/events` emits no `heartbeat` within ~18 s. + +#### Breach + price-staleness signals (safety core) + +Breach detection + the price-staleness guard now live in the isolated **safety +core** (ADR-0004 / ALP-857), which surfaces them by being **loud in its log** — +it has no SSE / `/events` channel (monitoring.md § 5.2) and writes nothing to the DB. Watch +`safety_core.log` for two distinct lines, and interpret them differently: + +- **Breach** — `safety_core BREACH: broker snapshot breaches at ` + (logged `CRITICAL`). A gross-exposure or single-name-concentration limit is + breached on the **broker snapshot**. Positions stay protected by the + broker-enforced bracket floor; the safety core does **not** submit or trigger + an emergency (monitoring.md § 5.7) — it detects and stays alive. + +- **Price feed globally stale** — `safety_core: underlying price feed globally + stale — writer appears wedged; stale=… missing=…` (logged `ERROR`). Every + open-position ticker is simultaneously stale (older than + `underlying_price_max_age_seconds`, 900 s) or never received — the safety + core's price feed (subscribed to the **broker snapshot's** equity symbols) has + gone cold. The safety core keeps beating its heartbeat and re-logs this each + tick; if the feed task has *crashed*, the process exits and NSSM restarts it, + but a connected-but-silent feed is logged, **not** auto-recovered (the safety + core has no in-stream reconnect) — restart `alphamind-safety-core` (services.md § 7) if it + persists. A *single* stale ticker (e.g. a just-opened position whose quote + stream hasn't warmed up) is excluded from that tick silently — broker bracket + legs still protect it — and is not logged as the global-stale condition. + +#### After a monitor restart — expect a brief projection-rebuild lag + +- The startup replay + the 15-min `activities_backfill` re-capture missed + fills into the broker-event log (and `fill_records` as `unprocessed`); the + **next Phase-1** integrates them (PENDING→OPEN) and the **projection rebuild** + (ALP-854) folds the event log onto the live broker snapshot, re-deriving + positions/cash. There is no `_reconcile_cash` writeback — the Projection is + rebuilt, not adjudicated (ADR-0001). Cash drift and position divergence + persist only until that Phase-1 runs. +- The monitor's underlying-price stream (used by the greeks + options-stop + tasks) subscribes to *known* open positions from the DB projection, so a fill + that hasn't integrated through Phase-1 yet isn't priced for options-stop + enforcement until the next Phase-1. This is subscription lag, **not** a dead + price feed. The position's broker-side bracket legs still protect it meanwhile; + it clears at the next Phase-1. + +### 8.10 PM equity CLOSE / reduce — protective legs cancelled before the sell, remainder auto re-bracketed (ALP-937 / ALP-938) + +A native equity bracket's take-profit + stop legs reserve 100% of the position's +shares at the broker (`held_for_orders`). So a PM-directed equity CLOSE (full **or** +partial) first cancels the broker-enforced protective legs via `submit_cancel`, +**then** submits the SIMPLE close sell. This is expected and visible in the +submission log as leg cancels immediately preceding the close order — not a bug. + +Two operational consequences to know: + +- **Partial-reduce remainder is auto re-bracketed (ALP-938).** A *partial* close + cancels **all** protective legs (you cannot partially cancel an OCO leg), so the + reduce briefly leaves the remaining shares broker-unprotected. Fill collection now + marks the position with a durable `positions.reprotection_needed=1` flag, and a + post-fill-collection re-bracket step — running every invocation right after fill + collection commits, **before** the slow thesis/LLM steps — automatically re-protects + the remainder: it submits a fresh standalone OCO (take-profit + stop) at the + **original** protective levels for the remaining shares and appends the new legs onto + the same still-ACTIVE bracket, **without waiting for the PM**. On success the flag + clears. The continuous-monitor position-level max-loss guardrail backstops the brief + window between the reduce filling and the re-bracket landing (same cancel-and-review + model used for corporate actions). A partially-reduced equity position should + therefore read with an ACTIVE bracket carrying the original CANCELLED legs **plus** + fresh ACTIVE take-profit + stop legs; a position still showing `reprotection_needed=1` + with all-CANCELLED legs after an invocation means the re-bracket submit failed (see + next bullet). + +- **"ALP-938 NAKED POSITION: re-bracket OCO submit FAILED" CRITICAL alert in + `pipeline.log`.** If the automatic re-bracket OCO submit fails (broker gateway failure + or permanent rejection — e.g. shares-not-available, OCO ineligibility), the step logs + `ALP-938 NAKED POSITION: re-bracket OCO submit FAILED for ...` at CRITICAL, + **leaves `reprotection_needed=1` set so the next invocation retries**, and never aborts + the invocation. The remaining shares stay broker-unprotected and the monitor max-loss + guardrail is the active backstop until a retry succeeds or the PM re-evaluates. + **Operator action:** confirm the monitor is live (§ 8.9) and watch the named position's + P/L; if the submit keeps failing across consecutive invocations, investigate the broker + rejection in the submission log and — if the next scheduled PM pass is far off — + consider a manual invocation (invocations.md § 3) so the PM re-brackets or closes the position. + +- **"NAKED POSITION" CRITICAL alert in `pipeline.log` (close dispatch, ALP-937).** If the + close sell itself is rejected *after* the protective legs were already cancelled, the + dispatcher logs `ALP-937 NAKED POSITION: equity CLOSE of ...` at CRITICAL. The + position is broker-unprotected and the monitor max-loss guardrail is the only backstop + until the PM re-evaluates next invocation. **Operator action:** confirm the monitor is + live (§ 8.9), watch the named position's P/L, and — if the next scheduled PM pass is far + off — consider a manual invocation (invocations.md § 3) so the PM re-brackets or closes it. + +### 8.11 Equity CLOSE rejected `position_state_drift` (ALP-943) + +An equity CLOSE (PM-directed or engine-envelope) was rejected at dispatch by the +live-position drift guard: between the invocation snapshot the decision was made +on and the moment of execution, the broker position changed — it exited entirely +(a monitor re-protection stop or any protective leg filled), flipped side, or +shrank below the requested quantity. The guard re-checks the live broker +position (and, on a rejected protective-leg cancel, the leg's actual broker +state) before any close order reaches the wire; a flat or side-flipped position +rejects the command, and a shrunken one submits a sell clamped to the live +quantity instead. The submission log carries the rejection with +`gateway_reason=position_state_drift` plus a one-line message naming the symbol, +the expected side/qty, and what the broker actually reported, and the PM path +records a `command_abandoned` activity entry. + +This is the guard working, not a fault. Without it, the sell executes against a +flat position and Alpaca opens an unmanaged opposite-side position +(`sell_to_open` — the 2026-06-09 −4 MRVL short). + +**Operator action: none.** The position is already flat or smaller than the +projection believed — there is nothing left for the rejected CLOSE to do. The +next invocation's fill collection integrates the missed exit fills and the PM +re-evaluates with corrected state; protection for any surviving remainder is the +monitor's auto re-protection job (ALP-938, § 8.10). Only investigate if the same +symbol rejects across consecutive invocations — that means fill integration is +not catching the projection up to broker reality. + diff --git a/docs/runbooks/update-loop.md b/docs/runbooks/update-loop.md new file mode 100644 index 000000000..df2a2e37c --- /dev/null +++ b/docs/runbooks/update-loop.md @@ -0,0 +1,139 @@ +# Standard update loop + +Deploying latest `main` to the prod Windows machine: pull, sync, migrate, restart, +verify. Service table, ports, and paths live in [services.md](services.md); +troubleshooting in [troubleshooting.md](troubleshooting.md). + +> **Service-stop precondition.** If the market is open AND open positions exist, +> obtain explicit Operator confirmation before stopping services — § 1.4 takes the +> monitor and safety core down, leaving Monitor-enforced legs unwatched and breach +> detection dark until § 1.6 completes. Outside that window, proceed. + +## 1. The standard update loop + +**Run this every time you touch prod, even for "I just want to restart the +monitor" tasks.** The order is non-negotiable: a migration applied while a +service is reading the schema corrupts the SQLAlchemy metadata cache, and a +service started before its migrations are in place crashes at first DB read. + +From the repo root in PowerShell (elevated — NSSM control needs admin): + +### 1.1 Pull main + +```powershell +cd $env:USERPROFILE\AlphaMind # the prod checkout (NSSM AppDirectory points here) +git fetch origin +git status # MUST be on `main` and clean +git pull --ff-only origin main +``` + +If `git status` shows you on a feature branch or with uncommitted changes, +stop — investigate before pulling. Prod runs `main`, full stop. + +### 1.2 Sync dependencies + +```powershell +uv sync +``` + +Idempotent; only re-resolves when `uv.lock` changed. If the frontend changed +(check `git log --oneline -5 -- src/alphamind/command_center/frontend/`), +also: + +```powershell +cd src\alphamind\command_center\frontend +bun install --frozen-lockfile +bun run build +cd ..\..\..\..\.. +``` + +### 1.3 Check for pending migrations + +```powershell +uv run alembic current # what's applied to the DB +uv run alembic heads # what the codebase expects +``` + +If the two revisions match, **skip to § 1.6**. Nothing to migrate; just +restart any services whose code changed. + +### 1.4 Stop services (reverse of the start order) + +The stop order is operational convention — no service declares an SCM +dependency on another, so the SCM enforces no ordering (the topology +invariant; services.md § 7). Stop the +command center first so its loopback consumers sit quiet through the +migration window rather than reconnect-looping against stopped upstreams. +Collector is independent of the trading DB schema but stop it too to keep +the snapshot quiet during the migration. Stop each **watchdog before the +service it supervises** so the watchdog does not `nssm restart` its target +while you are taking it down. + +```powershell +nssm stop AlphaMindCommandCenter +nssm stop alphamind-safety-core-watchdog +nssm stop alphamind-safety-core +nssm stop alphamind-monitor-watchdog +nssm stop alphamind-monitor +nssm stop alphamind-scheduler +nssm stop alphamind-collector +Get-Service alphamind-collector, alphamind-scheduler, alphamind-monitor, alphamind-monitor-watchdog, alphamind-safety-core, alphamind-safety-core-watchdog, AlphaMindCommandCenter +``` + +All seven should report `Stopped`. The install scripts leave NSSM's default +stop-method timeouts in place; if one hangs in `StopPending` past ~30 s, see +troubleshooting.md § 8.3. + +### 1.5 Apply migrations + +```powershell +uv run alembic upgrade head +uv run alembic current # confirm now equals heads +``` + +If the upgrade fails, **do not restart services.** Read the traceback, +restore the DB from the most recent backup (see [command-center.md](command-center.md) +§ Recovery → DB corruption restore), then re-apply. + +### 1.6 Restart services (conventional start order) + +The start order is operational convention, not SCM-enforced. Start each +supervised service **before its watchdog** so the watchdog finds a fresh +heartbeat on its first probe (and does not restart a service that is still +coming up). The command center comes last only by symmetry with § 1.4 — it +starts fine before its upstreams (its consumers reconnect with capped +backoff until they come up). + +```powershell +nssm start alphamind-collector +nssm start alphamind-scheduler +nssm start alphamind-monitor +nssm start alphamind-monitor-watchdog +nssm start alphamind-safety-core +nssm start alphamind-safety-core-watchdog +nssm start AlphaMindCommandCenter +Get-Service alphamind-collector, alphamind-scheduler, alphamind-monitor, alphamind-monitor-watchdog, alphamind-safety-core, alphamind-safety-core-watchdog, AlphaMindCommandCenter +``` + +All seven should report `Running` within ~60 s (NSSM's `SERVICE_DELAYED_AUTO_START` ++ the supervisors' own startup time). If any stays in `StartPending` past +that, tail its `.err.log` immediately. + +### 1.7 Verify green + +```powershell +# Tail the recent end of each daemon's log; look for "startup complete" / +# "supervisor started" / equivalent. No traceback should appear in the last +# 60 s of any .err.log. +Get-Content "$env:USERPROFILE\AlphaMind\logs\pipeline.out.log" -Tail 30 +Get-Content "$env:USERPROFILE\AlphaMind\logs\monitor.out.log" -Tail 30 +Get-Content "$env:USERPROFILE\AlphaMind\logs\command_center.out.log" -Tail 30 +``` + +Optional gate (runs in ~1 min, exercises 7 command-center checks): + +```bash +set -a && source <(tr -d '\r' < .env) && set +a && \ + uv run python scripts/verify/verify_command_center.py +``` + diff --git a/scripts/RUNBOOK_production.md b/scripts/RUNBOOK_production.md deleted file mode 100644 index a53678e11..000000000 --- a/scripts/RUNBOOK_production.md +++ /dev/null @@ -1,1569 +0,0 @@ -# Production Operations Runbook - -The canonical operator workflow for AlphaMind in production — on the Windows -trading machine, against the live `alphamind.db` and paper Alpaca account. - -This runbook covers the day-to-day operating loop, the one-time first-run -bootstrap, manual invocations, scheduled invocations, how to monitor those -invocations, command-center access, and service restarts. It assumes the -seven NSSM services are the supported runtime form on prod: - -| Service | Module entry | Log basename | -|----------------------------------|--------------------------------------------------|--------------------| -| `alphamind-collector` | `python -m alphamind.collector run` | `collector.*.log` | -| `alphamind-scheduler` | `python -m alphamind.scheduler run` | `pipeline.*.log` | -| `alphamind-monitor` | `python -m alphamind.execution.continuous_monitor run` | `monitor.*.log` | -| `alphamind-monitor-watchdog` | `python -m alphamind.execution.continuous_monitor watchdog` | `monitor_watchdog.*.log` | -| `alphamind-safety-core` | `python -m alphamind.execution.continuous_monitor.safety_core run` | `safety_core.*.log` | -| `alphamind-safety-core-watchdog` | `python -m alphamind.execution.continuous_monitor.safety_core watchdog` | `safety_core_watchdog.*.log` | -| `AlphaMindCommandCenter` | `python -m alphamind.command_center` | `command_center.*.log` | - -**Safety core + out-of-process watchdog (ADR-0004 / ALP-857).** Breach detection -+ price-staleness — the lone safety item with **no broker floor** — is isolated -into the `alphamind-safety-core` service, which reads the **broker snapshot** + -**REST latest-quote snapshots** (polled, no second market-data websocket — -ALP-940), beats a heartbeat file under -`%USERPROFILE%\AlphaMind\logs\safety_core.heartbeat`, and **writes nothing** to -the DB. The safety core polls REST quotes rather than opening its own -`StockDataStream`: Alpaca's free IEX plan allows exactly **one** authenticated -market-data websocket per account, so a second stream collided with the -monitor's (`connection limit exceeded`) and starved both feeds. REST latest-quote -is not connection-limited, so dropping it frees the single market-data connection -for the `alphamind-monitor` proper. The `alphamind-safety-core-watchdog` service is a **dedicated -out-of-process** watchdog: it probes that heartbeat file and runs -`nssm restart alphamind-safety-core` when it goes stale (a loop-resident watchdog -cannot catch a freeze of its own loop). The `alphamind-monitor` proper now runs -only **precision/data** tasks (options stops, greeks, entry-window, fill stream + -recovery sweep) and is fail-safe under the broker floor — a frozen monitor -degrades precision while positions stay broker-protected. The watchdog must run -under the **same Windows account** as the safety core so its `nssm restart` has -permission. - -**Monitor watchdog (ALP-941).** The monitor gets the same treatment: its -supervisor beats `%USERPROFILE%\AlphaMind\logs\monitor.heartbeat` while its -event loop turns, and the `alphamind-monitor-watchdog` service probes that file -and runs `nssm restart alphamind-monitor` when the beat is older than -`monitor_watchdog_tick_seconds × watchdog_cadence_multiplier` (15 s × 10 = -150 s by default). The monitor's **in-process** stall watchdog runs on the very -event loop it guards, so a frozen loop (the 2026-06-09 wedge: 8 h of no fills -with NSSM "Running") starves it — only an out-of-process watchdog bounds that -failure. A frozen loop also self-captures its blocking frame: the faulthandler -deadman dumps the frozen stack to `monitor_faulthandler.log` before the restart -lands. The same-account rule applies to this watchdog too. - -Logs land under `%USERPROFILE%\AlphaMind\logs\`. DB is -`%USERPROFILE%\AlphaMind\data\alphamind.db`. Invocation archives land under -`%USERPROFILE%\AlphaMind\archive\\\`. - -`RUNBOOK_command_center.md` is the sibling runbook for passkey registration, -credential recovery, and alert-engine configuration. This runbook is -otherwise self-contained. - -> **Living document — agents, keep it current.** If you follow a procedure -> here and it's wrong, outdated, or doesn't match actual prod behavior, -> update this runbook in the same change that produces the fix. Don't paper -> over a broken step with an ad-hoc workaround — make the runbook reflect -> reality. New monitoring patterns, new gotchas, new failure modes, schedule -> changes, port changes — they all belong here so the next operator (human -> or agent) inherits the lesson. - ---- - -## 1. The standard update loop - -**Run this every time you touch prod, even for "I just want to restart the -monitor" tasks.** The order is non-negotiable: a migration applied while a -service is reading the schema corrupts the SQLAlchemy metadata cache, and a -service started before its migrations are in place crashes at first DB read. - -From the repo root in PowerShell (elevated — NSSM control needs admin): - -### 1.1 Pull main - -```powershell -cd $env:USERPROFILE\AlphaMind # the prod checkout (NSSM AppDirectory points here) -git fetch origin -git status # MUST be on `main` and clean -git pull --ff-only origin main -``` - -If `git status` shows you on a feature branch or with uncommitted changes, -stop — investigate before pulling. Prod runs `main`, full stop. - -### 1.2 Sync dependencies - -```powershell -uv sync -``` - -Idempotent; only re-resolves when `uv.lock` changed. If the frontend changed -(check `git log --oneline -5 -- src/alphamind/command_center/frontend/`), -also: - -```powershell -cd src\alphamind\command_center\frontend -bun install --frozen-lockfile -bun run build -cd ..\..\..\..\.. -``` - -### 1.3 Check for pending migrations - -```powershell -uv run alembic current # what's applied to the DB -uv run alembic heads # what the codebase expects -``` - -If the two revisions match, **skip to § 1.6**. Nothing to migrate; just -restart any services whose code changed. - -### 1.4 Stop services (reverse of the start order) - -The stop order is operational convention — no service declares an SCM -dependency on another, so the SCM enforces no ordering (the topology -invariant; § 7). Stop the -command center first so its loopback consumers sit quiet through the -migration window rather than reconnect-looping against stopped upstreams. -Collector is independent of the trading DB schema but stop it too to keep -the snapshot quiet during the migration. Stop each **watchdog before the -service it supervises** so the watchdog does not `nssm restart` its target -while you are taking it down. - -```powershell -nssm stop AlphaMindCommandCenter -nssm stop alphamind-safety-core-watchdog -nssm stop alphamind-safety-core -nssm stop alphamind-monitor-watchdog -nssm stop alphamind-monitor -nssm stop alphamind-scheduler -nssm stop alphamind-collector -Get-Service alphamind-collector, alphamind-scheduler, alphamind-monitor, alphamind-monitor-watchdog, alphamind-safety-core, alphamind-safety-core-watchdog, AlphaMindCommandCenter -``` - -All seven should report `Stopped`. The install scripts leave NSSM's default -stop-method timeouts in place; if one hangs in `StopPending` past ~30 s, see -§ 8.3. - -### 1.5 Apply migrations - -```powershell -uv run alembic upgrade head -uv run alembic current # confirm now equals heads -``` - -If the upgrade fails, **do not restart services.** Read the traceback, -restore the DB from the most recent backup (see `RUNBOOK_command_center.md` -§ Recovery → DB corruption restore), then re-apply. - -### 1.6 Restart services (conventional start order) - -The start order is operational convention, not SCM-enforced. Start each -supervised service **before its watchdog** so the watchdog finds a fresh -heartbeat on its first probe (and does not restart a service that is still -coming up). The command center comes last only by symmetry with § 1.4 — it -starts fine before its upstreams (its consumers reconnect with capped -backoff until they come up). - -```powershell -nssm start alphamind-collector -nssm start alphamind-scheduler -nssm start alphamind-monitor -nssm start alphamind-monitor-watchdog -nssm start alphamind-safety-core -nssm start alphamind-safety-core-watchdog -nssm start AlphaMindCommandCenter -Get-Service alphamind-collector, alphamind-scheduler, alphamind-monitor, alphamind-monitor-watchdog, alphamind-safety-core, alphamind-safety-core-watchdog, AlphaMindCommandCenter -``` - -All seven should report `Running` within ~60 s (NSSM's `SERVICE_DELAYED_AUTO_START` -+ the supervisors' own startup time). If any stays in `StartPending` past -that, tail its `.err.log` immediately. - -### 1.7 Verify green - -```powershell -# Tail the recent end of each daemon's log; look for "startup complete" / -# "supervisor started" / equivalent. No traceback should appear in the last -# 60 s of any .err.log. -Get-Content "$env:USERPROFILE\AlphaMind\logs\pipeline.out.log" -Tail 30 -Get-Content "$env:USERPROFILE\AlphaMind\logs\monitor.out.log" -Tail 30 -Get-Content "$env:USERPROFILE\AlphaMind\logs\command_center.out.log" -Tail 30 -``` - -Optional gate (runs in ~1 min, exercises 7 command-center checks): - -```bash -set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/verify_command_center.py -``` - ---- - -## 2. First-time bootstrap - -**Run § 2 exactly once, the very first time AlphaMind comes up on the -production machine.** Subsequent operations all go through § 1. - -> **If you are performing the broker-boundary genesis cutover (ADR-0001–0005)** -> — swapping to a new Alpaca paper account and a fresh DB — use -> `docs/runbooks/genesis-cutover.md` as your primary procedure. That runbook -> supersedes § 2.2 (the fresh DB replaces the additive migration) and is the -> current first-run procedure for the new design. The account-swap step -> (§ 2.1) and the `--fresh-start` cold-start invocation (§ 2.4) remain -> relevant: § 2.1 for populating the new account's credentials in `.env`, -> and § 2.4 for the `synthetic_id_count == 0` verification and hard-fail -> paths. Return to § 2.5 onward to install the five remaining NSSM services -> after the genesis cutover bootstrap step. - -The collector is already running on this box (it was installed first to -accumulate the months of distillation-calibration data the analysis layer -needs). Steps 2.5–2.6 below assume that and skip re-installing it. - -### 2.1 Populate `.env` - -Copy `.env.example` to `.env` at the repo root, then fill in real values for -the vendor keys it lists. The template is incomplete — three additional keys -are required by prod but missing from `.env.example`. Append them after the -existing block: - -```bash -# === Append to .env after the vendor-key block === - -# Required: Claude SDK OAuth token. Every pipeline + monitor invocation's -# SDK subprocess inherits this from the process environment. Generate via -# `claude setup-token` on a machine signed into the operator's Anthropic -# account, then copy the value here. -CLAUDE_CODE_OAUTH_TOKEN= - -# Optional: Discord alert fanout. When unset, the command center logs -# "Discord alerts disabled" at startup and the in-app channel still works. -# Generate via Discord → Server Settings → Integrations → Webhooks → New -# Webhook. Treat the URL as a secret. -ALPHAMIND_DISCORD_WEBHOOK= - -# Optional but strongly recommended: pinned session-cookie signing key. -# When unset, the command center mints a fresh key on every restart and -# every active operator session is invalidated. Generate a 32-byte -# base64 random value: -# python -c "import secrets; print(secrets.token_urlsafe(32))" -# Placeholder is fine for first boot — the command center auto-generates -# an in-process secret if this is empty. Pin it before the first scheduled -# restart you care about. -COMMAND_CENTER_SESSION_SECRET= -``` - -`ALPACA_PAPER_KEY` and `ALPACA_PAPER_SECRET` (already present in the -template) must be populated with paper-account credentials from -`app.alpaca.markets/paper/dashboard/overview`. Live keys are deliberately -left blank — the `--fresh-start` argparse blocks `--mode live` and the -scheduler defaults `--mode paper` so no service can accidentally route to -the live endpoint. - -**CRLF gotcha.** On Windows the editor will save `.env` with `\r\n` line -endings. Bare `source .env` under Git Bash leaves a trailing `\r` on every -value — `CLAUDE_CODE_OAUTH_TOKEN` then fails bearer auth several minutes into -a pipeline run. For any manual CLI invocation, source via: - -```bash -set -a && source <(tr -d '\r' < .env) && set +a -``` - -NSSM-managed services bypass this — each entrypoint calls `load_dotenv()`, -which parses the repo-root `.env` from the service's working directory (NSSM -`AppDirectory`) and strips line endings, so the `\r` never survives. No shell -`source` step is involved. (The install scripts do **not** inject the vendor -keys via NSSM's `AppEnvironmentExtra`; the only env value set that way is the -command-center session secret in § 2.5.) - -### 2.2 Bring the DB to alembic head - -> **Cutover to the new broker-boundary design (ADR-0001–0005) requires a -> fresh DB, not an additive migration.** The procedure in -> `docs/runbooks/genesis-cutover.md` — swapping to a new Alpaca paper -> account and a new empty DB file — **supersedes this step** for that -> one-time event. After the genesis cutover § 2 does not apply; the system -> is already bootstrapped and all subsequent operations go through § 1. -> -> The text below applies to the **legacy** setup (old Alpaca account, old -> DB) and to any incremental schema updates after genesis. - -```powershell -uv run alembic upgrade head -uv run alembic current # confirm at heads -``` - -The collector has been writing into this DB for months; do not delete or -re-create it for incremental updates. Migrations are additive. - -### 2.3 Build the command-center frontend - -```powershell -cd src\alphamind\command_center\frontend -bun install --frozen-lockfile -bun run build -Test-Path .\dist\index.html # must return True -cd ..\..\..\..\.. -``` - -If `bun --version` reports anything older than 1.1.x, upgrade first — earlier -versions silently mutate `bun.lock` on `--frozen-lockfile`. - -### 2.4 Cold-start bootstrap invocation - -The paper-Alpaca account must be at zero positions with starting cash before -this runs. The bootstrap fetches Alpaca's reported cash, writes the -`cash_ledger` + `drawdown_state` singletons, and runs one `market_open` -invocation. The invocation drives the full pipeline (fill collection → analysis → -decision → command execution broker dispatch) — the PM's accepted commands must land -on Alpaca with real broker order ids. The synthetic `alp-{order_id}` -placeholder is deleted (ALP-847): an order with no broker counterpart — a -monitor-enforced protective leg (armed Intent the continuous monitor -enforces) or a not-yet-routed order — carries a NULL `alpaca_order_id`, never -a placeholder. Refuses to run if either singleton already exists. - -```bash -set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python -m alphamind.scheduler run \ - --fresh-start \ - --once market_open \ - --reason "first-run bootstrap" -``` - -Expected: one full invocation followed by the process exiting 0. Both -singletons committed to the DB, and — if the PM produced any envelopes — -the corresponding entry + broker-enforced protective legs (the native -equity bracket's take-profit + first stop) should appear on the Alpaca side -with real UUIDs; monitor-enforced legs carry a NULL `alpaca_order_id` (no -broker order — they are not on Alpaca, by design). Wall-clock is -**~25–35 min** on a cold-cache cold-start — every SDK call pays first-fill -`cache_write` cost (no warm prompt cache), the deterministic distillation -step takes ~3–5 min against the full prod data layer, and at least one of -the Sonnet phases (`adaptive` is the usual culprit) typically dominates at -~6–8 min. This is the same band as the steady-state ~25–50 min scheduled -invocations document in § 4; budget accordingly and don't restart -the process if it looks "stuck" inside that window — run the e2e progress -monitor (`scripts/RUNBOOK_end_to_end_verification.md` § Monitoring -progress mid-run) against the invocation's archive to see live phase -transitions. Confirm: - -```bash -uv run python -c " -import sqlite3, os -db = sqlite3.connect(os.path.expandvars(r'%USERPROFILE%\AlphaMind\data\alphamind.db')) -print('cash_ledger:', db.execute('SELECT current_cash_usd FROM cash_ledger').fetchone()) -print('drawdown_state:', db.execute('SELECT equity_high_water_mark_usd FROM drawdown_state').fetchone()) -print('order_count:', db.execute('SELECT COUNT(*) FROM orders').fetchone()) -print('synthetic_id_count:', db.execute(\"SELECT COUNT(*) FROM orders WHERE alpaca_order_id LIKE 'alp-%'\").fetchone()) -print('orphan_pending_no_broker_id:', db.execute(\"SELECT COUNT(*) FROM orders o WHERE o.alpaca_order_id IS NULL AND o.status NOT IN ('PENDING_SUBMIT','CANCELLED','REJECTED') AND NOT EXISTS (SELECT 1 FROM bracket_legs bl WHERE bl.order_id = o.order_id AND bl.enforcement_binding = 'monitor_enforced')\").fetchone()) -" -``` - -Both singleton rows should match Alpaca's reported cash on the freshly-reset -account to the cent. **`synthetic_id_count` must be `0` — by construction**: -ALP-847 deleted the `alp-{order_id}` mint, so no code can persist a synthetic -placeholder; a non-zero count would mean a stale pre-ALP-847 DB (re-baseline -it). A NULL `alpaca_order_id` is now the *expected* steady state for a -monitor-enforced leg or a not-yet-routed (`PENDING_SUBMIT`) order — NOT a -defect. The meaningful check is **`orphan_pending_no_broker_id` must be `0`**: -an active order with no broker id that is *not* a monitor-enforced leg means -broker dispatch was bypassed (the invocation did not reach Alpaca). Regression -checking should also confirm via `TradingClient.get_orders(status=ALL)` that -the broker-enforced orders are visible on the Alpaca side. - -**Hard-fail paths.** `--fresh-start` refuses to run when: - -- **Alpaca reports any open positions.** The error names the offending - symbol(s). Reset the Alpaca account first. -- **Alpaca reports any open orders.** `--fresh-start` requires a flat account — - positions empty **and** the broker order book empty. The error names the - offending order symbol(s). Cancel them (reset the account) first. -- **`cash_ledger` already has a row.** The error includes the existing - `current_cash_usd`. Positions/cash are a derived **Projection** rebuilt each - invocation from the broker-event log applied to the live broker snapshot - (ALP-854 / ADR-0001) — there is no auto-correct adjudication; a - re-bootstrap is never the correct path once the singleton is populated. -- **`drawdown_state` already has a row.** Same shape, same recovery. - -The CLI exits with code 2 (distinct from the generic scheduler-error code 1) -and prints the message verbatim to stderr — no traceback. Match on the -prefix `--fresh-start:` to filter precondition failures from scheduler -crashes. - -**Recovery if the cold-start invocation itself fails after the singletons -committed.** The two rows are committed in their own transaction *before* -the invocation runs, so a fill collection error or transient external dependency -leaves them in place and a retry of `--fresh-start` will hard-fail. Two -options, in preference order: - -1. **Re-run without `--fresh-start`.** The singletons are already populated - correctly; the bootstrap step is no longer needed: - - ```bash - uv run python -m alphamind.scheduler run \ - --once market_open \ - --reason "retry after bootstrap" - ``` - -2. **Wipe the singletons and re-bootstrap** — only if the persisted values - are wrong (e.g., the Alpaca fetch returned a transient zero-cash - mid-reset). On the prod machine: - - ```sql - DELETE FROM drawdown_state; - DELETE FROM cash_ledger; - ``` - - Then re-run the `--fresh-start --once market_open --reason ...` form. - -### 2.5 Install the six remaining NSSM services - -Collector is already installed. Install the other six in dependency order -from an elevated PowerShell prompt at the repo root. `install_monitor_service.ps1` -and `install_safety_core_service.ps1` each install **both** their supervised -service and its dedicated watchdog — set the `ObjectName` on all four (each -watchdog must run under the same account so its `nssm restart ` has -permission): - -```powershell -.\scripts\install_pipeline_scheduler_service.ps1 -nssm set alphamind-scheduler ObjectName .\ # prompts for pw - -.\scripts\install_monitor_service.ps1 -nssm set alphamind-monitor ObjectName .\ -nssm set alphamind-monitor-watchdog ObjectName .\ - -.\scripts\install_safety_core_service.ps1 -nssm set alphamind-safety-core ObjectName .\ -nssm set alphamind-safety-core-watchdog ObjectName .\ - -.\scripts\install_command_center_service.ps1 -nssm set AlphaMindCommandCenter ObjectName .\ -``` - -Pin the command-center session secret on the service environment so restarts -don't kick operators out: - -```powershell -$secret = python -c "import secrets; print(secrets.token_urlsafe(32))" -nssm set AlphaMindCommandCenter AppEnvironmentExtra "COMMAND_CENTER_SESSION_SECRET=$secret" -``` - -(Also copy the value into `.env`'s `COMMAND_CENTER_SESSION_SECRET=` line so -manual CLI invocations get the same key.) - -### 2.6 Start the six new services - -Start each supervised service **before its watchdog** (a watchdog probes its -target's heartbeat and would restart a service that hasn't beaten yet): - -```powershell -nssm start alphamind-scheduler -nssm start alphamind-monitor -nssm start alphamind-monitor-watchdog -nssm start alphamind-safety-core -nssm start alphamind-safety-core-watchdog -nssm start AlphaMindCommandCenter -Get-Service alphamind-collector, alphamind-scheduler, alphamind-monitor, alphamind-monitor-watchdog, alphamind-safety-core, alphamind-safety-core-watchdog, AlphaMindCommandCenter -``` - -All seven should be `Running`. - -### 2.7 Register the first passkey - -The command-center daemon mints a one-time setup token on first boot. -Retrieve it from the log: - -```powershell -Get-Content "$env:USERPROFILE\AlphaMind\logs\command_center.out.log" -Tail 50 | - Select-String "command_center setup token" -``` - -Open `http://localhost:8090/` in a browser on the Windows machine (RDP in if -remote), paste the token, choose a username, and complete WebAuthn -registration with Windows Hello or a hardware key. **Use the `localhost` -hostname, not `127.0.0.1`** — the WebAuthn relying-party id is `localhost` -(`config/security.yaml`), so a passkey ceremony loaded from the `127.0.0.1` -origin is rejected by the browser. See -`RUNBOOK_command_center.md` § Register the first passkey for details and § -Adding a passkey for enrolling a backup authenticator. - -### 2.8 Verify green - -```bash -set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/verify_command_center.py -``` - -Expect `7/7 checks passed`. AlphaMind is live in paper-trading mode. - ---- - -## 3. Running a manual invocation - -Use cases: ad-hoc smoke after a config edit, off-schedule analysis pass, -characterizing a specific run-type's behavior. - -A manual invocation runs in your shell — it does NOT go through the -scheduler service and does NOT honor the schedule. The scheduler service can -keep running in parallel; the two processes share the DB but APScheduler's -overlap-dedup keys off scheduled run-types only, so a manual `--once` won't -collide with the daemon's scheduled fire. - -```bash -set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python -m alphamind.scheduler run \ - --once \ - --reason "" -``` - -Where `` is one of: -`market_open`, `market_hours_rolling`, `pre_close`, `off_hours_rolling`, -`weekend_saturday`, `weekend_sunday`, `emergency`. - -The process exits when the invocation completes (~25–50 min steady-state in -prod — the full agent roster runs on every run-type; see § 4). All -artifacts land in the same archive layout as scheduled runs — under -`%USERPROFILE%\AlphaMind\archive\\\`. - -**agent_calls telemetry capture is active in production (ALP-907).** Every LLM -agent call (the 9 analysis + decision agents, run inside the SDK subprocess -worker) persists one `agent_calls` table row plus a four-file provenance tree -under -`%USERPROFILE%\AlphaMind\data\provenance\invocations\\agent_calls\\` -(`system_prompt.md`, `output_schema.json`, `tools_definition.json`, -`output.json`; the row's `output_artifact_ref` points at that directory). The -worker opens one telemetry SQLite session per agent call and commits it before -returning. Telemetry persistence is **logged-not-fatal**: a capture/commit error -is logged (`agent_calls telemetry commit failed in subprocess worker` / -`agent_calls capture failed …`) and never fails the agent call or the -invocation. Delivery requires the `agent_calls` table to exist — the combined -feedback-loop migration (ALP-879) must be applied on the prod box first; -until then the insert silently no-ops (swallowed) and the table stays empty. - -**Post-deploy spot-check** (after the migration is applied): run one `--once` -invocation, then confirm the table populated and the tree exists: - -```bash -uv run python -c "import os, sqlite3; \ -db = sqlite3.connect(os.path.expandvars(r'%USERPROFILE%\AlphaMind\data\alphamind.db')); \ -print(db.execute('SELECT count(*), sum(success) FROM agent_calls').fetchone())" -``` - -```powershell -Get-ChildItem "$env:USERPROFILE\AlphaMind\data\provenance\invocations" -Recurse -Filter output.json | Select-Object -First 5 -``` - -**Do not pass `--fresh-start`.** That flag is reserved for the one-time -bootstrap in § 2.4; it hard-fails when the singletons already exist. - -**A manual `--once` run is NOT observable on the SSE streams.** The -`--once` code path (`_run_once` in `src/alphamind/scheduler/__main__.py`) -runs the invocation in your shell as its own process; it never constructs -the `SSEEventEmitter` or the `/control` + `/events` HTTP server — those exist -only in the daemon path (`_run_daemon`). So while a manual run executes, the -scheduler daemon's `127.0.0.1:8765/events` stream (§ 5.1) and the command -center's Live Run dashboard (§ 5.3) — which consumes that same stream — show -only heartbeats. They reflect the daemon, and the daemon is idle; your run is -elsewhere. (Don't be fooled into thinking the run stalled: collect → distill -then SSE silence is exactly what a healthy out-of-process run looks like on -those surfaces.) - -To watch a manual `--once` invocation live, use: - -- **The process's own stdout/stderr** — the terminal you launched it in. This - is the richest live view: per-phase `distillation.orchestrator` completions, - per-agent `analysis.*.runner` lines, command execution dispatch, and the final exit - code all stream here. -- **The structured `pipeline.log`** (§ 5.4) — `_run_once` does call - `configure_pipeline_logging()`, so records land in the rotating log - alongside the daemon's (filter by the invocation id). -- **The `invocations` DB row** (§ 5.5) and **per-invocation archive** (§ 5.6), - which are written identically to a scheduled run. - ---- - -## 4. Scheduled invocations (the daemon's job) - -Once `alphamind-scheduler` is running as an NSSM service, APScheduler drives -all scheduled invocations automatically against the NYSE calendar. The -production schedule (timezone `US/Eastern`, defined in -`config/scheduler.yaml`): - -The schedule is **Tier B** (ALP-745): every trigger fires at a distinct -minute. The `as_of` is second-resolution and cron fires land on `:00`, so a -distinct minute means a distinct `as_of` — no two scheduled runs ever share -one (the condition that previously crashed one of two coinciding distillation -passes on the `distillation_ticker_baseline` UNIQUE constraint). - -| Run type | Cron (US/Eastern) | Fires (ET) | Notes | -|------------------------|----------------------------|-------------------------------------------|----------------------------------------------| -| `market_open` | `35 9 * * mon-fri` | 09:35 weekdays | NYSE-gated; fires just after the 09:30 open for fresh live quotes | -| `market_hours_rolling` | `0 13 * * mon-fri` | 13:00 weekdays (single mid-day read) | NYSE-gated; dedup-gated within 30 min | -| `pre_close` | `0 15 * * mon-fri` | 15:00 weekdays | NYSE-gated; sole owner of the close slot | -| `weekend_sunday` | `0 18 * * sun` | 18:00 Sunday | Unconditional | -| `off_hours_rolling` | (unscheduled) | Manual / emergency only | Valid run type; overlay retained, not on cron | -| `weekend_saturday` | (unscheduled) | Manual / emergency only | Valid run type; overlay retained, not on cron | -| `emergency` | (not scheduled) | On-demand, operator-triggered (command-center control → scheduler `trigger_emergency_invocation`) | Cooldown-gated per `config/breach_behavior.yaml`; auto breach-cascade is **not** wired in this build (§ 5.7) | - -~3 scheduled invocations per trading weekday (`market_open` + `market_hours_rolling` -+ `pre_close`), ~16 per full trading week including the Sunday run. There is no -longer any same-minute collision for the dedup window to "collapse" — every slot -is distinct by construction. **Typical steady-state wall-clock per invocation is -~25–50 minutes**: deterministic distillation takes ~5–6 min, and the 9 SDK agents -(~1–8 min each, domain researchers serial within their phase) dominate the rest; -the adaptive researcher consuming its full budget pushes a run toward the upper -band. Observed: 29.7 / 40.8 min `market_open`, ~49 min `market_hours_rolling` -(2026-06-09/10). The "~60–180 seconds" figure previously documented here -predates the full agent roster — a sub-5-minute completion now means the run -aborted early, not that it was fast. -An `emergency` invocation is **not** a fast path: it runs the full agent roster — -the same roster as `market_open`, with the heaviest adaptive-researcher budget -(`config/run_types/emergency.yaml`) — so it lands comparable to or slower than a -scheduled run. - -The cadence and time windows live in `config/scheduler.yaml`. Per-trigger -agent budgets live in `config/run_types/*.yaml`. Edits to either take effect -on the next scheduler restart — APScheduler reads them at boot. To apply a -schedule change: - -```powershell -nssm restart alphamind-scheduler -``` - -No DB migration, no data churn — the service restart re-reads configs and -re-arms the jobs. - -**To pause scheduled invocations without taking the service down**, use the -command center's pause control (Dashboard → Controls → Pause), or hit the -loopback verb directly: - -```bash -curl -X POST http://127.0.0.1:8765/control/pause \ - -H "Content-Type: application/json" \ - -d '{"reason":"operator pause via CLI"}' -``` - -Resume: - -```bash -curl -X POST http://127.0.0.1:8765/control/resume \ - -H "Content-Type: application/json" \ - -d '{}' -``` - -Pause/resume mutates an in-process flag; APScheduler skips fires while -paused, and the flag survives the service running but does NOT survive a -restart (a restarted scheduler is unpaused by default). Use the -command-center UI for the audited path — it writes an `OPERATOR_CONSOLE` row -to `activity_log` for each pause/resume. - ---- - -## 5. Monitoring scheduled invocations - -**Important: production scheduled invocations do NOT emit a `progress.jsonl` -file.** The scheduler uses a no-op progress emitter in production; the -JSONL-stream variant is part of a separate harness that's not on the daemon -code path. Don't try to tail a file under the archive directory hoping to -see phase events — there is none. Use the surfaces below instead. - -The right surface depends on whether you're watching in real time or -investigating after the fact, and whether you want a structured event stream -or free-form log records. - -### 5.1 Real-time: the scheduler's `/events` SSE stream (canonical) - -This is the primary live-monitoring surface for agents and operators — -**for daemon-driven invocations only** (scheduled fires + emergency -cascades). A manual `--once` CLI run is a separate process and does not -publish here; watch it via its process stdout / `pipeline.log` instead -(§ 3). - -```bash -curl -N http://127.0.0.1:8765/events -``` - -Emits SSE frames for each of these event types: - -| Event | When | -|-------------------------|----------------------------------------------------------| -| `invocation_started` | A daemon-driven invocation begins (scheduled / emergency; NOT a manual `--once` CLI run) | -| `phase_transition` | The orchestrator advances to a new pipeline phase | -| `agent_started` | An SDK call begins (analyst, strategist, PM, etc.) | -| `agent_succeeded` | The SDK call returned a valid response | -| `agent_retrying` | Parse/validation failure; the harness is retrying | -| `agent_failed` | All retries exhausted; the invocation will abort | -| `invocation_ended` | The invocation reached a terminal state (success/failure)| -| `next_trigger_changed` | APScheduler recomputed its next fire | -| `heartbeat` | 15 s idle keep-alive | - -**For agents:** call this stream via the `Monitor` tool so each emitted SSE -frame becomes a notification. Filter with grep/jq for the events you care -about. Silence beyond 15 s indicates the SSE stream is wedged (scheduler -process crashed, network filter on the loopback interface, etc.) — not that -"nothing is happening." Heartbeats are the proof-of-life signal. - -Cross-field invariants worth knowing: - -- `phase_transition` is forward-only — an invocation never goes backward - through phases. -- `agent_succeeded` and `agent_failed` are mutually exclusive per - `(invocation_id, agent_name)` — exactly one fires per agent per - invocation. -- `invocation_started` for an emergency-triggered invocation carries - `run_type: emergency` and the `invocation_id` returned by the - `trigger_emergency_invocation` control verb. - -### 5.2 Real-time: the monitor's `/events` SSE stream - -The continuous monitor publishes its own SSE stream on `127.0.0.1:8766`. In the -**current build** it emits fills plus an idle keep-alive: - -```bash -curl -N http://127.0.0.1:8766/events -``` - -| Event | When | -|-----------------|-----------------------------------------------| -| `fill_received` | A fill arrived from the trade-updates stream | -| `heartbeat` | 15 s idle keep-alive | - -Watch this stream to see fills landing between invocations. - -**What is *not* on this stream (and why).** The monitor's SSE emitter still -*defines* `websocket_connected` / `websocket_disconnected` / `breach_detected` / -`emergency_invocation_triggered` / `greeks_refreshed`, but their producers are -not composed into the running monitor, so none of them are emitted: - -- **`breach_detected` / `emergency_invocation_triggered`** — breach detection + - price-staleness moved out of the monitor into the isolated, **log-only safety - core** (ADR-0004 / ALP-857). The safety core has no SSE surface and writes - nothing; it logs to `safety_core.log` (§ 8.9). The monitor proper no longer - runs a breach loop. -- **`greeks_refreshed`** — the greeks-refresh task runs, but its SSE emit adapter - is not wired; confirm greeks via `monitor.log` / the per-position diagnostics. -- **`websocket_connected` / `websocket_disconnected`** — defined but never - emitted anywhere in the current tree; track websocket health via - `monitor.log` / `monitor.err.log` instead. - -### 5.3 Real-time: the command center Live Run dashboard - -`http://localhost:8090/` → log in → Live Run. The dashboard merges both -daemons' SSE streams + decorates them with activity-log rows and per-agent -status. Best surface when an operator is sitting at the console. - -### 5.4 After-the-fact: the structured pipeline log - -```powershell -Get-Content "$env:USERPROFILE\AlphaMind\logs\pipeline.log" -Tail 200 -``` - -Per-line structured records. The `.log` file is the structured Python-logger -emit (one JSON-ish record per line including `invocation_id`, `phase`, -`agent`, and any exception with full traceback). The `.out.log` and -`.err.log` siblings are NSSM's raw stdout/stderr capture — useful only when -the daemon crashed before the structured logger could write. - -To filter to a single invocation across the three log files: - -```powershell -Select-String -Path "$env:USERPROFILE\AlphaMind\logs\pipeline.*.log" -Pattern "inv-20260527T143000Z-" -``` - -### 5.5 After-the-fact: the `invocations` DB table - -Each invocation gets exactly one row recording its lifecycle: - -```bash -uv run python -c " -import sqlite3, os -db = sqlite3.connect(os.path.expandvars(r'%USERPROFILE%\AlphaMind\data\alphamind.db')) -db.row_factory = sqlite3.Row -for row in db.execute('SELECT invocation_id, trigger_type, trigger_source, trigger_reason, start_at, fill_collection_completed_at, command_execution_completed_at, git_sha_at_invocation, staleness_flag FROM invocations ORDER BY start_at DESC LIMIT 20').fetchall(): - print(dict(row)) -" -``` - -The table does **not** carry a single `status` column. Completion is read -from the phase timestamps: a row with a non-null `command_execution_completed_at` -succeeded through command execution; a row with `start_at` set but -`command_execution_completed_at` NULL either is still in-flight or aborted mid-pipeline -(cross-check the SSE stream / `pipeline.log`). Useful columns: -`trigger_type` (how the invocation was launched — `scheduled`, `manual`, -or `emergency`), `trigger_source` (the run-type / origin label — for -scheduled fires `market_open`, `market_hours_rolling`, `pre_close`, -`weekend_sunday`, or `off_hours_rolling`; `cli` for a -manual `--once` run; `operator_console` for a command-center action; -`continuous_monitor` for an emergency invocation), -`trigger_reason` (free-form), `fill_collection_completed_at` / `command_execution_completed_at` -(lifecycle), `git_sha_at_invocation` (the repo HEAD when the invocation -ran — useful for confirming which code version a run executed under), -`command_execution_summary_json` (PM command-submission detail), -`staleness_flag`, and `resolved_config_snapshot_path` (points into the -archive directory). - -### 5.6 Per-invocation archive directory - -Every invocation writes a directory under -`%USERPROFILE%\AlphaMind\archive\\\` (UTC date, -not ET). The invocation ID format is `inv-YYYYMMDDTHHMMSSZ-<8-hex>`. - -Contents include `resolved_config.json` (the full resolved config tree as it -existed for that invocation — useful when reconciling "what config did this -run actually use") and per-agent diagnostic subdirs under -`analysis//` and `decision//` (each carries the assembled -input bundle, the prompt sent to the SDK, the SDK response, and a -`metadata.json` with token + wall-clock + tool-call counts). - -This is the most expensive surface to read but the highest fidelity. Use -it when § 5.4 / § 5.5 don't tell you why an agent produced a malformed -brief. - -### 5.7 What to watch for - -**Stalled agent.** An `agent_started` event with no matching -`agent_succeeded` / `agent_failed` after that agent's configured -`latency_budget_seconds` (in `config/run_types/.yaml`). The orchestrator -will eventually time out and emit `agent_failed`, but if you're watching -live and the budget is generous, the stall is visible first as -unexplained silence in the SSE stream. - -**Failed invocation.** An `invocation_ended` event reporting a non-success -terminal state. The `invocations` row has **no `status` or `exit_reason` -column** — a failed run is identified by `command_execution_completed_at` being NULL -(often `fill_collection_completed_at` too). Tail `pipeline.log` (and the daemon's -`pipeline.err.log`) for the traceback that names the failing layer. - -**Thesis-resolution skip warnings (expected, not a fault).** Every deliberative -invocation runs a thesis-resolution step (`phase: thesis_resolution`) between -fill collection and snapshot assembly: it moves each closed-position thesis -`ACTIVE → RESOLVED`, authoring the component outcomes + resolution category the -snapshot's recent-resolutions feed and thesis-quality aggregates read. The step -resolves each eligible thesis **independently** — a per-thesis data gap logs a -`WARNING` in `pipeline.log` (naming the `thesis_id` + `position_id` + the -specific gap) and **skips** that thesis, leaving it `ACTIVE`, then continues -resolving the rest. It never aborts the invocation. A skip-warning is therefore -**expected operator-visible behavior, not an error** — most commonly an -option-lifecycle-closed thesis (expiry / assignment / exercise) that carries no -`POSITION_CLOSED` activity-log entry yet, so the exit method can't be read. The -skipped thesis stays `ACTIVE` and is retried next invocation. Investigate only -if the *same* thesis logs the skip on every invocation indefinitely (a stuck -thesis, not a transient gap). - -**Skipped invocations.** When you expected a scheduled fire (per § 4) and -no `invocation_started` event arrived: - -- **NYSE holiday or early close** — weekday fires are NYSE-calendar-gated. - No fire on a market holiday is expected, not a bug. -- **Daemon paused** — check the command center Controls panel. Pause flag - doesn't survive a restart. -- **Dedup suppressed it** — the 30-minute dedup window suppresses a - `market_hours_rolling` fire if some invocation completed phase 2 within the - lookback (e.g., a manual or emergency run just before the 13:00 slot). Under - Tier B no two scheduled triggers share a minute, so this only happens around - off-schedule runs, never between two scheduled fires. The - `next_trigger_changed` events on the SSE stream show what APScheduler - thinks comes next. - -**Emergency invocations.** In the current build these are **operator-initiated**, -not auto-cascaded from a breach. The expected sequence: - -1. Operator: command center → Controls → Trigger emergency invocation (or - `POST` the scheduler's `trigger_emergency_invocation` verb). This writes an - `EMERGENCY_INVOCATION_REQUESTED` row to `activity_log`. -2. Scheduler: the `emergency_receiver` task polls `activity_log`, picks up the - request, and — unless the 30-minute cooldown - (`config/breach_behavior.yaml` § `emergency_invocation_cooldown_minutes`; - a `margin_call` request bypasses it) suppresses it — dispatches one emergency - invocation. -3. Scheduler `/events`: `invocation_started` with `run_type: emergency`, then the - usual phase + agent events through `invocation_ended`. The `invocations` row - records `trigger_type: emergency` / `trigger_source: continuous_monitor`. - -If you requested an emergency and no `invocation_started` follows within a few -seconds, the cooldown gate suppressed it — check the cooldown window and the -recent `activity_log` rows for the prior emergency. - -> **Auto breach→emergency cascade is not wired in this build.** The monitor no -> longer runs a breach loop, and the isolated safety core only logs + heartbeats -> and writes nothing (ADR-0004 / ALP-857), so nothing currently enqueues an -> `EMERGENCY_INVOCATION_REQUESTED` row on a breach. The auto-trigger evaluator -> still exists in the tree but is not composed into the running monitor — so a -> breach surfaces in `safety_core.log` (§ 8.9) and via the broker-enforced -> bracket floor, not as an automatic emergency invocation. - -**Cost / token regression.** The `metadata.json` files under each agent's -diagnostic subdir record token counts under a `tokens_used` object -(`input_tokens` / `cache_read_tokens` / `cache_write_tokens` / -`output_tokens`). A healthy production run has the -bulk of input volume landing in `cache_read_tokens` (the prompt cache is -hit). If `input_tokens` is large and `cache_read_tokens` is near zero, -something invalidated the cache — investigate before the next invocation -amplifies the cost. - -### 5.8 Agent monitoring pattern - -The canonical pattern for an agent asked to "watch the next scheduled -invocation": - -1. Look up the next fire time from § 4's schedule table relative to current - ET. Confirm via `next_trigger_changed` events on the SSE stream. -2. Open `curl -N http://127.0.0.1:8765/events` via the `Monitor` tool so SSE - frames become notifications. -3. Wait for `invocation_started`. Record the `invocation_id`. -4. Watch for `phase_transition` → `agent_started` → `agent_succeeded` pairs - covering every phase through `command_execution`. -5. Wait for `invocation_ended`. On success, summarize cost + wall-clock - from the archive's per-agent `metadata.json` files. On failure, fetch - the traceback from `pipeline.log` / `pipeline.err.log` — the - `invocations` row carries no `exit_reason`; a NULL `command_execution_completed_at` - is the failure signal. -6. Surface the `invocation_id` to the operator in any report — it's the - entry point for follow-up archive inspection. - -**SSE caveat (why § 5.9 exists).** In practice, consuming the § 5.1 SSE -stream through the `Monitor` tool has proven an unreliable per-agent -notifier — `agent_*` frames do not always surface as notifications — and a -manual `--once` run does not publish to the daemon stream at all (§ 3). For a -robust, granular per-agent watch, and the *only* live option for manual runs, -prefer the archive-based pattern in § 5.9. - -### 5.9 Granular agent-completion watch via the archive (Monitor tool) - -Each analysis / decision agent writes its diagnostic subdir -(`analysis//` or `decision//`) **atomically on completion**, so -the appearance — or mtime change, on a retry — of that agent's -`metadata.json` is a high-fidelity per-agent finish marker carrying -`success`, `stop_reason`, `wall_clock_seconds`, `retry_count`, and the -token / cache counts. Unlike the SSE stream this works identically for -scheduled and manual `--once` runs and reads straight off the WAL DB + -filesystem. - -The canonical watcher is **`scripts/watch_invocation.sh`** — pass it to the -`Monitor` tool and it streams one notification per milestone, self-terminating on -a terminal state (including one that already happened *before* it attached). -Select the invocation one of three ways: - -```bash -# newest scheduled fire of a run-type (default lookback = now-5min): -bash scripts/watch_invocation.sh --scheduled market_hours_rolling -# a manual --once run, by a substring of its --reason: -bash scripts/watch_invocation.sh --reason "my run reason" -# attach to a specific invocation id: -bash scripts/watch_invocation.sh --inv inv-20260602T170000Z-193173de -``` - -Optional `--since ` widens/narrows the discovery window; `--tz-offset -` (default `6` = MDT) sets the prod-log local offset used to scope abort lines -(pass `7` for MST in winter). Paths are overridable via the `DB` / `LOG` / `ERR` / -`ARCHROOT` env vars (prod defaults are baked in). - -It emits, as they happen: `FILL-COLLECTION-COMPLETE` (ingestion + fill-collection summary); -`ACT …` for every `activity_log` row of the run. A `RECONCILIATION_ALERT` row now -means only an **orphan fill** that could not integrate (a poison-pill fill against a -terminal/over-filled position, ALP-761) — the reconcile-adjudication path is deleted -(ALP-854 / ADR-0001), so a positions/cash mismatch is silently rebuilt, never alerted. -The watch also emits all command-execution actions (`ORDER_SUBMITTED` / `PM_DECISION` / -`CAPITAL_RESERVED` / …); -`DISTILLATION` when `composite_state` is written; `AGENT / success=… -wall=… stop=…` per agent (the strategist's line is prefixed `>>> STRATEGIST`); -`FILLS` on each `unprocessed`→`processed`/`quarantined` transition; `FAULT(log|err)` -on a fresh non-benign traceback; and the terminal `COMMAND-EXECUTION-COMPLETE` (+ a final -positions / cash / fills dump) or `ABORT`. It is **silent while waiting** — no -routine "still waiting" ticks — except a one-shot `WARN scheduler port 8765 DOWN`. - -**Why it covers "silence ≠ success."** A healthy run is proven only by -`COMMAND-EXECUTION-COMPLETE` (the sole positive success signal — there is no `status` column, -§ 5.5). A hard crash writes **no** `metadata.json`, so the fault/abort legs catch -what the per-agent leg can't: fresh `Traceback`/`CRITICAL`/`ERROR` in -`pipeline.log` + `pipeline.err.log` (with the benign Windows asyncio-teardown noise -— `ProactorBasePipeTransport`, `WinError 121`, `no close frame`, `ResourceWarning` -— filtered out), plus the driver's `scheduled trigger= failed` / -`scheduler exited with error`. And the **at-attach terminal check** handles a run -that already finished or aborted before the watch started: it checks -`command_execution_completed_at` and greps the log (scoped to this run's own start + trigger) -once up front, emitting `ALREADY-COMPLETE` / `ALREADY-ABORTED` and exiting instead -of polling a dead invocation. (Before this check, a watcher armed after a fast-fail -abort polled a dead 17:00Z run for 20 min — 2026-06-02.) - -Gotchas baked into the script (worth knowing when reading its output): - -- Some agents (`synthesizer`, `strategist`) record `tool_calls_used` instead of - `retry_count`; a missing `retry` field is benign, not a failure. -- The `FILLS unprocessed N→M` leg confirms fill-collection ingestion before analysis even - starts — how both the 2026-06-01 poison-pill wedge and the 2026-06-02 ALP-824 - cross-writer fix were verified. -- A manual `--once` run does not publish to SSE (§ 5.1), and its - `scheduler exited with error` may land only in the launching shell's stderr (not - `pipeline.log`) — so for a manual run the **launching process's exit code** stays - the authoritative terminal signal; the watcher is the live milestone view. -- Set the `Monitor` `timeout` generously (or `persistent: true`) — a full-roster - run takes ~25–50 min (§ 4); the watch exits itself on the terminal state. -- The ACT field separator is a TAB, not a control character: the sqlite3 CLI - (≥ 3.49; prod runs 3.51) escapes control characters on output, so `char(31)` - arrives as the literal text `^_` — the field split silently failed and every - ACT line emitted empty until 2026-06-10. The same CLI emits CRLF; `q()` strips - the `\r` (command substitution hides this, `while read` pipelines don't). -- The fault legs filter benign noise **per traceback block**, not per line: the - SDK-subprocess teardown noise (`_ProactorBasePipeTransport.__del__` → - `ValueError: I/O operation on closed pipe`, after every agent completion) puts - its benign marker and its bare `Traceback` header on separate lines, so a - per-line filter leaks a spurious FAULT pair per agent. A real traceback emits - one line: `FAULT(log|err) Traceback … => `. - ---- - -## 6. Accessing the command center - -`http://localhost:8090/` from a browser on the Windows trading machine -(loopback only in v1; the daemon binds `127.0.0.1` but browse via the -`localhost` hostname — the WebAuthn relying-party id is `localhost`). Log in -with your registered passkey. - -To reach the UI from a remote workstation, RDP into the Windows machine and -open the browser there. Do **not** SSH-port-forward 8090 — the session cookie is -`SameSite=Strict` and WebAuthn's `expected_origin` / relying-party id are pinned -to `localhost`, so auth fails from any other origin (including a forwarder or -`127.0.0.1`). Do **not** bind the daemon to `0.0.0.0` — the relying-party id is -pinned to `localhost`; binding wider just exposes the API to the LAN without auth -working. Remote-access via VPS Caddy + WireGuard is deferred (see -`RUNBOOK_command_center.md` § Operator handover note). - -The dashboard surfaces, in order of operator priority: - -- **Controls** — pause/resume, trigger emergency invocation, switch profile, - cancel order, force-close position, set halt mode. -- **Live run watcher** — current invocation's phase + agent SSE stream. -- **Run history** — last N invocations with status + duration + cost. -- **Activity log** — every `activity_log` row, filterable. -- **Portfolio / theses / risk** — current state snapshot. -- **Alerts** — active + acked + history. -- **Configs** — per-file editor + git diff view. - -See `RUNBOOK_command_center.md` for passkey management, recovery, and the -alert engine config. - ---- - -## 7. Restarting a single service - -For any one service, in PowerShell as admin: - -```powershell -nssm restart -Get-Service -``` - -The valid service names are: - -- `alphamind-collector` -- `alphamind-scheduler` -- `alphamind-monitor` -- `alphamind-monitor-watchdog` -- `alphamind-safety-core` -- `alphamind-safety-core-watchdog` -- `AlphaMindCommandCenter` (note: mixed case, no hyphen) - -NSSM's `restart` is `stop` + `start`. The install scripts leave NSSM's default -stop-method timeouts in place (no explicit `AppStopMethod*` is set); in practice -a stop that sits in `StopPending` past ~30 s is wedged — see § 8.3. On stop, -SIGINT/SIGTERM **triggers** a co-operative shutdown: each in-flight task gets a -`CancelledError` and a bounded grace window to wind down — scheduler **10 s**, -monitor **5 s**, command center **30 s** (`supervisor_shutdown_timeout_seconds` -per daemon; the scheduler supervisor uses a manually-bounded wait rather than -`TaskGroup.__aexit__` so the timeout actually bounds exit). Tasks that don't exit -in the window are abandoned and NSSM/the SCM force-terminates the process; there -is no separate SIGTERM "fallback" the supervisor itself fires. - -A plain `nssm restart ` works for any single service with all the -others running: no service declares an SCM dependency on another, so the SCM -never refuses the stop half of the restart. That is a deliberate topology -invariant (ALP-945, the 2026-06-10 incident — a `DependOnService` edge from -the command center made the SCM refuse every monitor-watchdog restart, -wedging the monitor for 42 minutes): **never declare an SCM dependency on a -watchdog-supervised service.** - -**One restart gotcha that cost real diagnosis time (2026-06-02 monitor wedge):** - -- **`Get-Service … Running` does NOT mean healthy.** NSSM reports `Running` - while the *wrapper* process is alive. Two liveness layers now apply: - - **The safety core** (breach + price-staleness, the no-floor item) is - supervised by the **dedicated out-of-process `alphamind-safety-core-watchdog`** - (ADR-0004 / ALP-857): it probes the core's heartbeat file - (`%USERPROFILE%\AlphaMind\logs\safety_core.heartbeat`) and runs - `nssm restart alphamind-safety-core` when the beat goes stale past - `breach_evaluation_cadence_seconds × watchdog_cadence_multiplier` (default - 10×). A loop-resident watchdog cannot catch a freeze of its own loop, so the - watchdog is its own process — a frozen safety core is restarted, not masked. - - **The monitor proper** keeps its in-process per-cadence stall watchdog for - its precision/data tasks (`cadence_seconds × watchdog_cadence_multiplier` - forces `os._exit(1)` → NSSM restart) — but that watchdog runs on the very - event loop it guards, so a freeze of the **loop itself** (the 2026-06-09 - ALP-941 wedge) starves it. The **dedicated out-of-process - `alphamind-monitor-watchdog`** closes that gap: it probes the monitor's - heartbeat file (`%USERPROFILE%\AlphaMind\logs\monitor.heartbeat`) and runs - `nssm restart alphamind-monitor` when the beat goes stale past - `monitor_watchdog_tick_seconds × watchdog_cadence_multiplier` (default - 15 s × 10 = 150 s). On a loop freeze the faulthandler deadman also dumps - the frozen stack to `monitor_faulthandler.log` — read it to identify the - blocking frame **before** diagnosing further. The monitor no longer runs - breach detection, so a monitor wedge **degrades precision while positions - stay broker-protected** (the broker floor + the isolated safety core - hold). The **one residual exception is the control-surface HTTP server**, - registered `watched=False` (it blocks in `await server.serve()`); that - task is not auto-recycled in-process, though a full loop freeze that takes - it down now is (via the external watchdog). - - For everything, confirm a restart by the process **StartTime** - (fresh PID / StartTime) **and** a behavioral signal — the daemon's SSE - heartbeat (`8765` / `8766`), the safety-core / monitor heartbeat files' - mtime advancing, fresh structured-log lines, fills flowing — never by - `Get-Service Status` alone. Read the StartTime with: - ```powershell - $p = (Get-CimInstance Win32_Service -Filter "Name=''").ProcessId - (Get-Process -Id $p).StartTime - ``` - See § 8.9 for the fill-stream and underlying-stream auto-recovery details. - -**When the operator can restart a single service without coordinating the -others:** - -| Restart this | When | Side effects | -|---------------------|---------------------------------------------------------------|-----------------------------------------------------------------------------------| -| collector | After a `config/collector_schedule.yaml` or `config/data_sources.yaml` edit | Up to one cadence-cycle of dropped vendor reads | -| scheduler | After a `config/scheduler.yaml`, `config/run_types/*.yaml`, or `config/main.yaml` edit | Pause flag cleared; an in-flight invocation is cancelled mid-pipeline | -| monitor | After a `config/continuous_monitor.yaml` edit affecting precision/data tasks | Halt-mode flag persists; in-flight fill-stream reconnects from the recovery path. Its watchdog tolerates a restart within the stall bound (150 s default); for a longer outage stop `alphamind-monitor-watchdog` first | -| monitor-watchdog | After a `monitor_watchdog_tick_seconds` or `watchdog_cadence_multiplier` edit | None to positions; the monitor is unsupervised against loop freezes for the restart window | -| safety-core | After a `config/continuous_monitor.yaml` cadence/threshold or a `config/guardrails.yaml` / profile `gross_exposure_pct` / `position_max_size_pct` edit | Brief gap in breach/staleness detection only; positions stay broker-protected. Its watchdog tolerates a restart within the stall bound; for a longer outage stop the watchdog first (else it `nssm restart`s the core mid-restart) | -| safety-core-watchdog | After a `watchdog_cadence_multiplier` edit | None to positions; the safety core is unsupervised for the restart window | -| AlphaMindCommandCenter | After a frontend rebuild or alert-rule edit | Active SSE clients reconnect; operator sessions persist if `COMMAND_CENTER_SESSION_SECRET` is pinned (§ 2.5), else are invalidated | - -**When to use the full update loop (§ 1) instead of a single restart:** - -- Any pull from main that touches `src/`, `pyproject.toml`, `uv.lock`, or a - migration file. (Always run § 1 in this case — start by checking - `git log --oneline ORIG_HEAD..` for migrations before assuming a single - service restart suffices.) -- Anything you're uncertain about. The full loop is ~3 min on a clean run - and idempotent — over-restarting costs nothing. - ---- - -## 8. Common operational gotchas - -### 8.1 Service won't start, `.err.log` mentions `NoSuchTableError` - -Migrations are behind. Run § 1.3–1.5. - -### 8.2 Command center returns 404 at `/` - -Frontend bundle missing. Run § 2.3 (just the `bun install` + `bun run build` -part); no service restart needed — FastAPI's `StaticFiles` mount picks up -the new `dist/` on the next request. - -### 8.3 NSSM stop hangs in `StopPending` past 30 s - -The supervisor is blocked on a task that won't cancel gracefully within its -shutdown grace window (§ 7). Force the issue: - -```powershell -nssm kill # SIGKILL equivalent -Get-Service # should now be Stopped -``` - -Then investigate the hang in the `.out.log` / `.err.log` before restarting. -A regular hang here is a bug worth filing. - -### 8.4 `gh pr checks` says CI is green but a service won't start - -CI doesn't exercise the Windows service supervisors. A Linux-only behavior -in the merged code (POSIX `os.fsync` on a directory handle is the classic -example) can pass CI and crash on prod. Tail the failing service's -`.err.log` and file the issue against `main`. - -### 8.5 Pipeline invocation aborts with a Claude SDK auth error 5 minutes in - -`CLAUDE_CODE_OAUTH_TOKEN` is set but carries a trailing `\r` from a bare -`source .env` under Git Bash. Re-source via the `tr -d '\r'` form (§ 2.1 -CRLF gotcha) before running manual invocations. Services bypass this — they -read the env from NSSM's `AppEnvironmentExtra` so they're unaffected. - -### 8.6 Operator browser session invalidated after every restart - -`COMMAND_CENTER_SESSION_SECRET` is not pinned on the service environment. -Run the `nssm set AlphaMindCommandCenter AppEnvironmentExtra ...` step from -§ 2.5. Existing sessions stay invalidated this one time; subsequent -restarts preserve them. - -### 8.7 Looking for `progress.jsonl` under an invocation archive and not finding one - -Expected — § 5 explains. Prod uses the no-op progress emitter; the JSONL -artifact is part of a different harness. Use the SSE stream (§ 5.1), the -structured pipeline log (§ 5.4), or the `invocations` table (§ 5.5) -instead. - -### 8.8 Scheduled invocation didn't fire when § 4's schedule said it would - -Walk § 5.7's "Skipped invocations" checklist: NYSE holiday, daemon paused, -or dedup suppressed it (only around an off-schedule run — under Tier B no two -scheduled triggers share a minute). The `next_trigger_changed` events on the -SSE stream are the source of truth for what the scheduler thinks comes next. - -### 8.9 Monitor wedge — fill-stream or underlying-stream silent (connected-but-not-delivering) - -#### Current state (post-ALP-825 hardening) - -Both the fill (trade-updates) stream and the underlying-price stream now have -**two** independent auto-recovery layers: - -1. **Budget-neutral reconnect on RTH silence.** Each stream runs a - `StreamActivityMonitor` that beats the stall watchdog on every poll slice and - raises `StreamStalledError` when the market is open and no frames arrive within - the configured timeout (`fill_stream_stale_timeout_seconds = 900 s` for the fill - stream; `underlying_stream_stale_timeout_seconds = 60 s` for the price stream). - The consumer catches `StreamStalledError` and rebuilds the stream - **budget-neutrally** — without consuming a reconnect attempt — so a persistently - silent stream keeps recovering rather than exhausting its budget and exiting. REST - recovery + the fill-backfill backstop re-capture any gap from a fill-stream - reconnect. Off-hours, the silence check is disengaged and the clock resets so the - closed-market gap is not charged against the first RTH window. - -2. **Per-cadence stall watchdog.** Every watched task drives its loop through - `supervised_loop`, which beats the watchdog at the top of each iteration. A task - that beats on every poll slice (including idle slices between frames) keeps the - watchdog satisfied. A genuinely-blocked task that stops beating trips - `os._exit(1)` within `cadence_seconds × watchdog_cadence_multiplier` (the - `watchdog_cadence_multiplier` default is 10×), and NSSM auto-restarts the - monitor process. This backstop catches a task wedge that the in-stream reconnect - could not resolve — but it runs on the monitor's own event loop, so it cannot - catch a freeze of the loop itself. - -3. **Out-of-process watchdog on the whole event loop (ALP-941).** The supervisor - beats `monitor.heartbeat` each watchdog-loop pass; the separate - `alphamind-monitor-watchdog` service probes the file and runs - `nssm restart alphamind-monitor` when the beat is older than - `monitor_watchdog_tick_seconds × watchdog_cadence_multiplier` (150 s default). - This is the only layer that catches a **frozen event loop** (the 2026-06-09 - wedge: loop blocked, process alive, layers 1–2 starved with it). When it - fires, `monitor_faulthandler.log` holds the frozen main-thread stack — the - faulthandler deadman dumped it before the restart — so the blocking frame is - preserved for diagnosis. - -**Operator's primary action** after any suspected monitor wedge: - -1. Confirm a fresh PID / StartTime (the watchdog fired and NSSM auto-restarted): - ```powershell - $p = (Get-CimInstance Win32_Service -Filter "Name='alphamind-monitor'").ProcessId - (Get-Process -Id $p).StartTime - ``` -2. Confirm a behavioral signal — an `8766` heartbeat within ~18 s of connecting - (the SSE keep-alive cadence is 15 s): - ```bash - curl -N 127.0.0.1:8766/events - ``` -3. If the process PID/StartTime is fresh and the heartbeat is live, **auto-recovery - succeeded** — no manual restart is needed. Confirm fills are flowing again - (check `fill_records` for rows dated today; check `monitor.log` for fresh - trade-stream connect lines). - -**Manual § 7 restart is the fallback** when auto-recovery is itself suspect (e.g. -the StartTime is stale, no `8766` heartbeat is appearing, or `monitor.err.log` -shows a `reconnect budget exhausted` entry): `nssm stop alphamind-monitor` + -`nssm start alphamind-monitor` — no other service needs touching. Confirm by -StartTime + heartbeat, not `Get-Service`. - -#### Pre-fix history (2026-06-02 — for reference only) - -Before ALP-825: the continuous monitor's Alpaca trade-updates websocket failed on -`OSError: [WinError 121] The semaphore timeout period has expired`, and the -`websockets` library reconnect-looped *internally* without ever raising — so the -fill consumer parked on `await queue.get()` and starved. The process stayed -`Running` (NSSM only restarts on process exit, and the in-process watchdog didn't -trip) and captured **no fills for ~11 h**, with real-time breach/stop monitoring -off the whole time. ALP-819 fixed the fill-stream silent-wedge path; ALP-832 -applied the same fix to the underlying-price stream; ALP-826 added the per-cadence -watchdog backstop. - -The old detection signals (if you ever see them again, the post-fix recovery path -above should already be firing): - -- `monitor.err.log` shows repeated `trading stream websocket error, restarting - connection: no close frame received or sent` + `OSError: [WinError 121]`. -- `fill_records` has no rows for fills you know landed on Alpaca, and the matching - local order rows stay `PENDING` with `filled_quantity = 0`. -- `curl -N 127.0.0.1:8766/events` emits no `heartbeat` within ~18 s. - -#### Breach + price-staleness signals (safety core) - -Breach detection + the price-staleness guard now live in the isolated **safety -core** (ADR-0004 / ALP-857), which surfaces them by being **loud in its log** — -it has no SSE / `/events` channel (§ 5.2) and writes nothing to the DB. Watch -`safety_core.log` for two distinct lines, and interpret them differently: - -- **Breach** — `safety_core BREACH: broker snapshot breaches at ` - (logged `CRITICAL`). A gross-exposure or single-name-concentration limit is - breached on the **broker snapshot**. Positions stay protected by the - broker-enforced bracket floor; the safety core does **not** submit or trigger - an emergency (§ 5.7) — it detects and stays alive. - -- **Price feed globally stale** — `safety_core: underlying price feed globally - stale — writer appears wedged; stale=… missing=…` (logged `ERROR`). Every - open-position ticker is simultaneously stale (older than - `underlying_price_max_age_seconds`, 900 s) or never received — the safety - core's price feed (subscribed to the **broker snapshot's** equity symbols) has - gone cold. The safety core keeps beating its heartbeat and re-logs this each - tick; if the feed task has *crashed*, the process exits and NSSM restarts it, - but a connected-but-silent feed is logged, **not** auto-recovered (the safety - core has no in-stream reconnect) — restart `alphamind-safety-core` (§ 7) if it - persists. A *single* stale ticker (e.g. a just-opened position whose quote - stream hasn't warmed up) is excluded from that tick silently — broker bracket - legs still protect it — and is not logged as the global-stale condition. - -#### After a monitor restart — expect a brief projection-rebuild lag - -- The startup replay + the 15-min `activities_backfill` re-capture missed - fills into the broker-event log (and `fill_records` as `unprocessed`); the - **next Phase-1** integrates them (PENDING→OPEN) and the **projection rebuild** - (ALP-854) folds the event log onto the live broker snapshot, re-deriving - positions/cash. There is no `_reconcile_cash` writeback — the Projection is - rebuilt, not adjudicated (ADR-0001). Cash drift and position divergence - persist only until that Phase-1 runs. -- The monitor's underlying-price stream (used by the greeks + options-stop - tasks) subscribes to *known* open positions from the DB projection, so a fill - that hasn't integrated through Phase-1 yet isn't priced for options-stop - enforcement until the next Phase-1. This is subscription lag, **not** a dead - price feed. The position's broker-side bracket legs still protect it meanwhile; - it clears at the next Phase-1. - -### 8.10 PM equity CLOSE / reduce — protective legs cancelled before the sell, remainder auto re-bracketed (ALP-937 / ALP-938) - -A native equity bracket's take-profit + stop legs reserve 100% of the position's -shares at the broker (`held_for_orders`). So a PM-directed equity CLOSE (full **or** -partial) first cancels the broker-enforced protective legs via `submit_cancel`, -**then** submits the SIMPLE close sell. This is expected and visible in the -submission log as leg cancels immediately preceding the close order — not a bug. - -Two operational consequences to know: - -- **Partial-reduce remainder is auto re-bracketed (ALP-938).** A *partial* close - cancels **all** protective legs (you cannot partially cancel an OCO leg), so the - reduce briefly leaves the remaining shares broker-unprotected. Fill collection now - marks the position with a durable `positions.reprotection_needed=1` flag, and a - post-fill-collection re-bracket step — running every invocation right after fill - collection commits, **before** the slow thesis/LLM steps — automatically re-protects - the remainder: it submits a fresh standalone OCO (take-profit + stop) at the - **original** protective levels for the remaining shares and appends the new legs onto - the same still-ACTIVE bracket, **without waiting for the PM**. On success the flag - clears. The continuous-monitor position-level max-loss guardrail backstops the brief - window between the reduce filling and the re-bracket landing (same cancel-and-review - model used for corporate actions). A partially-reduced equity position should - therefore read with an ACTIVE bracket carrying the original CANCELLED legs **plus** - fresh ACTIVE take-profit + stop legs; a position still showing `reprotection_needed=1` - with all-CANCELLED legs after an invocation means the re-bracket submit failed (see - next bullet). - -- **"ALP-938 NAKED POSITION: re-bracket OCO submit FAILED" CRITICAL alert in - `pipeline.log`.** If the automatic re-bracket OCO submit fails (broker gateway failure - or permanent rejection — e.g. shares-not-available, OCO ineligibility), the step logs - `ALP-938 NAKED POSITION: re-bracket OCO submit FAILED for ...` at CRITICAL, - **leaves `reprotection_needed=1` set so the next invocation retries**, and never aborts - the invocation. The remaining shares stay broker-unprotected and the monitor max-loss - guardrail is the active backstop until a retry succeeds or the PM re-evaluates. - **Operator action:** confirm the monitor is live (§ 8.9) and watch the named position's - P/L; if the submit keeps failing across consecutive invocations, investigate the broker - rejection in the submission log and — if the next scheduled PM pass is far off — - consider a manual invocation (§ 3) so the PM re-brackets or closes the position. - -- **"NAKED POSITION" CRITICAL alert in `pipeline.log` (close dispatch, ALP-937).** If the - close sell itself is rejected *after* the protective legs were already cancelled, the - dispatcher logs `ALP-937 NAKED POSITION: equity CLOSE of ...` at CRITICAL. The - position is broker-unprotected and the monitor max-loss guardrail is the only backstop - until the PM re-evaluates next invocation. **Operator action:** confirm the monitor is - live (§ 8.9), watch the named position's P/L, and — if the next scheduled PM pass is far - off — consider a manual invocation (§ 3) so the PM re-brackets or closes it. - -### 8.11 Equity CLOSE rejected `position_state_drift` (ALP-943) - -An equity CLOSE (PM-directed or engine-envelope) was rejected at dispatch by the -live-position drift guard: between the invocation snapshot the decision was made -on and the moment of execution, the broker position changed — it exited entirely -(a monitor re-protection stop or any protective leg filled), flipped side, or -shrank below the requested quantity. The guard re-checks the live broker -position (and, on a rejected protective-leg cancel, the leg's actual broker -state) before any close order reaches the wire; a flat or side-flipped position -rejects the command, and a shrunken one submits a sell clamped to the live -quantity instead. The submission log carries the rejection with -`gateway_reason=position_state_drift` plus a one-line message naming the symbol, -the expected side/qty, and what the broker actually reported, and the PM path -records a `command_abandoned` activity entry. - -This is the guard working, not a fault. Without it, the sell executes against a -flat position and Alpaca opens an unmanaged opposite-side position -(`sell_to_open` — the 2026-06-09 −4 MRVL short). - -**Operator action: none.** The position is already flat or smaller than the -projection believed — there is nothing left for the rejected CLOSE to do. The -next invocation's fill collection integrates the missed exit fills and the PM -re-evaluates with corrected state; protection for any surviving remainder is the -monitor's auto re-protection job (ALP-938, § 8.10). Only investigate if the same -symbol rejects across consecutive invocations — that means fill integration is -not catching the projection up to broker reality. - ---- - -## 9. Feedback loop — analytics CLIs, cadences, and review skills - -The feedback loop (ALP-131) is the month-over-month system-improvement substrate: -it pairs the system's reasoning artifacts with the outcomes they produced and -computes deterministic, conditionable metrics over that pairing. It is **read-only -over trading state** — it measures and proposes, never mutating live trading-state -records — and runs **out-of-pipeline** as a set of operator CLIs plus three review -skills. The deliberative pipeline gained three prod-runtime behaviors that feed it -(documented in their own sections, cross-referenced below). - -The end-to-end spine is verified out-of-pipeline by -`scripts/verify_feedback_loop.py` (it runs `alembic upgrade head` on a scratch DB, -seeds a controlled closed-position scenario, and drives resolver → metrics → digest -+ snapshot → validation → supersession → retrospective, asserting each stage). Run -it after a change that touches the feedback-loop tables or the resolver: - -```bash -uv run python scripts/verify_feedback_loop.py # exits 0 on PASS, 1 on any FAIL -``` - -### 9.1 Prod-runtime behavior changes (the pipeline now feeds the loop) - -Three behaviors the deliberative pipeline gained — each documented in full where it -lands, listed here so the feedback-loop picture is complete: - -- **`agent_calls` telemetry + provenance every invocation** — every LLM agent call - persists one `agent_calls` row + a four-file provenance tree. Active in production - (ALP-907). See § 3 ("agent_calls telemetry capture is active in production") and - the § 10 reference path. The metrics' `WindowDataset` reads `agent_calls` for the - cost / process / citation-chain tiers. -- **Per-invocation thesis resolution (`ACTIVE → RESOLVED`)** — every deliberative - invocation runs a `thesis_resolution` step between fill collection and snapshot - assembly, authoring each closed-position thesis's component outcomes + resolution - category. See § 5 ("Thesis-resolution skip warnings"). The outcome-tier metrics - calibrate against these resolved theses; a per-thesis data gap is a skip-WARNING, - not a fault. -- **Distillation anomaly emission to `activity_log`** — the distillation layer emits - `DISTILLATION_ANOMALY_FLAG` activity-log entries (ALP-881) when a threshold in its - anomaly taxonomy trips; the digest's notable-shift surface reads them. - -### 9.2 The combined feedback-loop migration (apply before anything reads) - -The six feedback-loop tables (`agent_calls`, `validations`, `validation_outcomes`, -`weekly_digest_snapshots`, `retrospective_reports`, `retrospective_decisions`) and -the `activity_log` event-type / source CHECK widenings land in the combined -migration **ALP-879** (`c001fb0000ff_feedback_loop_tables_and_activity_log_check_widen`). -It is applied by the standard `alembic upgrade head` in the update loop (§ 1) / -first-run bootstrap (§ 2). Until it is applied on the prod box the `agent_calls` -inserts silently no-op and every analytics CLI errors on the missing table. - -### 9.3 Analytics-read CLI — digest, metric, snapshot - -`python -m alphamind.feedback_loop.digest.cli` (all subcommands emit JSON on stdout; -`--db-path` defaults to the live DB, `--config-dir` to `config/`): - -- `digest [--week YYYY-MM-DD] [--trajectory-weeks N]` — the full six-section - `WeeklyDigest` for the week (headline outcomes, process pulse, 8–12-week - trajectory, validation status, notable shifts, replay queue). -- `metric [--window N] [--week …]` — one `MetricResult` aggregated over - the trailing `--window` weeks. -- `metrics-list` — the registered metric ids (a metric gated on an absent - dependency, e.g. PM-accuracy before ALP-129, is simply absent). -- `snapshot [--week …] [--trajectory-weeks N]` — **producer**: writes the week's - `WeeklyDigest` into `weekly_digest_snapshots`, idempotently (the `week_start` - UNIQUE makes a re-run a reported no-op). - -All four are DB-only — no vendor API, no `.env` needed. - -**Cadence.** The `snapshot` producer is intended to run **once per week, Sunday -8am ET**, snapshotting the week just closed (omitting `--week` targets the prior -completed week). It is **not yet wired into the scheduler** — run it from an -operator cron / CI job, or by hand. `digest` / `metric` are on-demand reads the -`/feedback-review` skill drives. - -### 9.4 Validation CLI — register, evaluate, list, detect-supersessions - -`python -m alphamind.feedback_loop.validation.cli` (JSON on stdout; `register` / -`evaluate` take a JSON payload on `--input`, a path or `-` for stdin). DB-only: - -- `register` — snapshot the registering invocation's provenance (regime + - model id from `agent_calls`), freeze the pre-registered criteria, persist the - validation. -- `evaluate --validation-id … --outcome-id … --evaluated-at …` — compute the - pre/post metric comparison, derive the verdict + rollback status, write the - `validation_outcomes` row (or short-circuit if the validation was superseded). -- `list` — the pending (unevaluated, un-superseded) validations. -- `detect-supersessions` — **producer**: mark each active validation whose - post-edit window was crossed by a conditioning shift (regime transition, model - version change, or a commit to the watched artifact) `superseded`. - -**Cadence.** Run `detect-supersessions` **once per pipeline invocation and once per -commit** (the two events that can break a validation's conditioning context). It is -**not yet scheduler-wired** — drive it from the pipeline-adjacent cron / a commit -hook / CI, or via `/feedback-validate`. - -### 9.5 Retrospective CLI — ingest, save-report, capture-decision - -`python -m alphamind.feedback_loop.retrospective.cli` (DB-only): - -- `ingest --start … --end …` — pull the Phase-1 retrospective data set over the - window (the analytics-spine `WindowDataset` + the unresolved pending-rollback - follow-ups). -- `save-report --start … --end … --markdown-file …` — persist the rendered report - metadata row + write the markdown under - `data/retrospective_reports/{report_id}/report.md`. -- `capture-decision --report-id … --item-identifier … --decision-type … --verdict … - --rationale …` — record one walkthrough decision (promotion candidate or - follow-up), optionally linking a spawned validation. - -### 9.6 The three review skills (headless) - -The operator drives the loop through three Claude skills, all **headless** — they -read the analytics through the CLIs above and emit markdown the operator reads: - -- **`/feedback-review`** — weekly process-metric / monthly outcome-metric review and - ad-hoc agent/metric deep-dives, over the digest CLI. -- **`/feedback-validate`** — register a pre-registered validation for a prompt / - config edit, or evaluate one whose window has elapsed. Enforces pre-registration - discipline + confounder controls + posterior-band reporting (confirmation bias is - the failure mode it exists to defeat). -- **`/feedback-retrospective`** — open-ended quarterly LLM-driven deep retrospective: - read many invocations end-to-end, surface patterns the deterministic metrics did - not catch, propose promotion candidates. - -The interactive dashboard for this surface (View F) is **deferred to ALP-686** — the -loop is headless-only for now. - -### 9.7 PM-accuracy verification follows story 06e / ALP-129 - -The PM-accuracy + modification-effectiveness metrics (and their -`counterfactual_replays` loader sub-bundle) are gated on the counterfactual replay -engine (ALP-129). They are **verified when story 06e (ALP-887) lands** alongside that -engine — the spine verify above does not assert them, and `metrics-list` simply -omits the gated metric ids until the dependency is present. This is a known, -intentional gap, not a missing check. - ---- - -## 10. Reference: ports and paths - -| What | Where | -|------------------------------|------------------------------------------------------------------| -| Scheduler `/control` + `/events` | `127.0.0.1:8765` (loopback only) | -| Monitor `/control` + `/events` | `127.0.0.1:8766` (loopback only) | -| Command center API + UI | binds `127.0.0.1:8090` (loopback only); **browse via `http://localhost:8090`** — WebAuthn relying-party id is `localhost` | -| Safety core heartbeat (no port) | `%USERPROFILE%\AlphaMind\logs\safety_core.heartbeat` (file; the out-of-process watchdog probes it — the safety core has no control port and writes nothing to the DB) | -| Monitor heartbeat (file) | `%USERPROFILE%\AlphaMind\logs\monitor.heartbeat` (the `alphamind-monitor-watchdog` service probes it; written by the monitor supervisor's watchdog loop — ALP-941) | -| Monitor freeze stack dump | `%USERPROFILE%\AlphaMind\logs\monitor_faulthandler.log` (the faulthandler deadman dumps the frozen main-thread stack here when the monitor's event loop wedges) | -| Production DB | `%USERPROFILE%\AlphaMind\data\alphamind.db` | -| Daemon logs | `%USERPROFILE%\AlphaMind\logs\.{out,err,log}.log` (`safety_core.*`, `safety_core_watchdog.*` for the safety-core services; `monitor_watchdog.*` for the monitor watchdog) | -| Invocation archives | `%USERPROFILE%\AlphaMind\archive\\\` | -| Agent-call provenance | `%USERPROFILE%\AlphaMind\data\provenance\invocations\\agent_calls\\` (one dir per LLM agent call; `system_prompt.md` + `output_schema.json` + `tools_definition.json` + `output.json`; paired with one `agent_calls` table row) | -| Retrospective reports | `\data\retrospective_reports\\report.md` (markdown body; the `retrospective_reports` row carries the metadata + `report_file_ref`) | -| Feedback-loop CLIs (§ 9) | `python -m alphamind.feedback_loop.{digest,validation,retrospective}.cli`; spine verify `scripts/verify_feedback_loop.py` | -| Config files | `\config\*.yaml` (incl. `feedback.yaml` sample-size floors, `digest.yaml` shift thresholds), `\config\run_types\*.yaml`, `\config\profiles\*.yaml` | -| `.env` | `\.env` | diff --git a/scripts/archive_linear_issues.py b/scripts/linear/archive_linear_issues.py similarity index 95% rename from scripts/archive_linear_issues.py rename to scripts/linear/archive_linear_issues.py index b8a3ae45c..7036abff1 100644 --- a/scripts/archive_linear_issues.py +++ b/scripts/linear/archive_linear_issues.py @@ -16,11 +16,11 @@ Usage: # preview a parent's Done sub-issues (changes nothing) - uv run python scripts/archive_linear_issues.py --parent ALP-117 + uv run python scripts/linear/archive_linear_issues.py --parent ALP-117 # archive them, after operator approval - uv run python scripts/archive_linear_issues.py --parent ALP-117 --apply + uv run python scripts/linear/archive_linear_issues.py --parent ALP-117 --apply # archive an explicit set - uv run python scripts/archive_linear_issues.py --ids ALP-321,ALP-322 --apply + uv run python scripts/linear/archive_linear_issues.py --ids ALP-321,ALP-322 --apply Exit codes: 0 dry-run completed, or --apply archived everything planned @@ -179,7 +179,7 @@ def _emit_human(args: argparse.Namespace, targets: list[Target]) -> None: failed = planned - archived print(f"Archived {archived} issue(s); {failed} failed; {skipped} skipped.") if archived: - print("Re-run scripts/check_linear_cap.py to see the freed buffer.") + print("Re-run scripts/linear/check_linear_cap.py to see the freed buffer.") else: print(f"{planned} issue(s) would be archived; {skipped} skipped.") if planned: diff --git a/scripts/check_linear_cap.py b/scripts/linear/check_linear_cap.py similarity index 95% rename from scripts/check_linear_cap.py rename to scripts/linear/check_linear_cap.py index fa97a4073..ff6f6dbe9 100644 --- a/scripts/check_linear_cap.py +++ b/scripts/linear/check_linear_cap.py @@ -14,11 +14,11 @@ only report ">= 250" once a single page fills. Usage: - uv run python scripts/check_linear_cap.py - uv run python scripts/check_linear_cap.py --needed 12 - uv run python scripts/check_linear_cap.py --needed 12 --json - uv run python scripts/check_linear_cap.py --breakdown - uv run python scripts/check_linear_cap.py --team AlphaMind + uv run python scripts/linear/check_linear_cap.py + uv run python scripts/linear/check_linear_cap.py --needed 12 + uv run python scripts/linear/check_linear_cap.py --needed 12 --json + uv run python scripts/linear/check_linear_cap.py --breakdown + uv run python scripts/linear/check_linear_cap.py --team AlphaMind Exit codes: 0 under the cap, and (if --needed given) the buffer covers it diff --git a/scripts/export_linear_issues.py b/scripts/linear/export_linear_issues.py similarity index 93% rename from scripts/export_linear_issues.py rename to scripts/linear/export_linear_issues.py index 38379b10e..6c9cb4d9c 100644 --- a/scripts/export_linear_issues.py +++ b/scripts/linear/export_linear_issues.py @@ -7,10 +7,11 @@ server during orchestration handovers. Usage: - uv run python scripts/export_linear_issues.py PARENT --out-dir DIR - uv run python scripts/export_linear_issues.py ALP-114 --out-dir /tmp/export - uv run python scripts/export_linear_issues.py ALP-114 --out-dir /tmp/export --include-archived - uv run python scripts/export_linear_issues.py ALP-114 --out-dir /tmp/export --json + uv run python scripts/linear/export_linear_issues.py PARENT --out-dir DIR + uv run python scripts/linear/export_linear_issues.py ALP-114 --out-dir /tmp/export + uv run python scripts/linear/export_linear_issues.py ALP-114 --out-dir /tmp/export \ + --include-archived + uv run python scripts/linear/export_linear_issues.py ALP-114 --out-dir /tmp/export --json Exit codes: 0 completed successfully (including when the parent has zero children) diff --git a/scripts/linear_consolidation_candidates.py b/scripts/linear/linear_consolidation_candidates.py similarity index 95% rename from scripts/linear_consolidation_candidates.py rename to scripts/linear/linear_consolidation_candidates.py index d46456299..19d9ed961 100644 --- a/scripts/linear_consolidation_candidates.py +++ b/scripts/linear/linear_consolidation_candidates.py @@ -8,9 +8,9 @@ next" tracking used by the /linear-consolidate workflow. Usage: - uv run python scripts/linear_consolidation_candidates.py - uv run python scripts/linear_consolidation_candidates.py --team AlphaMind - uv run python scripts/linear_consolidation_candidates.py --json + uv run python scripts/linear/linear_consolidation_candidates.py + uv run python scripts/linear/linear_consolidation_candidates.py --team AlphaMind + uv run python scripts/linear/linear_consolidation_candidates.py --json Exit codes: 0 ran successfully (report produced; exit 0 even when there are no diff --git a/scripts/linear_parity_diff.py b/scripts/linear/linear_parity_diff.py similarity index 98% rename from scripts/linear_parity_diff.py rename to scripts/linear/linear_parity_diff.py index a96e2f81f..d729eee66 100644 --- a/scripts/linear_parity_diff.py +++ b/scripts/linear/linear_parity_diff.py @@ -14,9 +14,9 @@ markup — so a DIFFERS verdict reflects real content drift. Usage: - uv run python scripts/linear_parity_diff.py ALP-114 \\ + uv run python scripts/linear/linear_parity_diff.py ALP-114 \\ --archive-dir docs/_archive/implementation/03-analysis-layer/synthesizer - uv run python scripts/linear_parity_diff.py ALP-114 --archive-dir DIR --json + uv run python scripts/linear/linear_parity_diff.py ALP-114 --archive-dir DIR --json Exit codes: 0 every sub-issue matches its local archive file diff --git a/scripts/linear_stale_sweep.py b/scripts/linear/linear_stale_sweep.py similarity index 96% rename from scripts/linear_stale_sweep.py rename to scripts/linear/linear_stale_sweep.py index b75db6857..f1315da42 100644 --- a/scripts/linear_stale_sweep.py +++ b/scripts/linear/linear_stale_sweep.py @@ -16,9 +16,9 @@ the relation; issues missing the relation are a data-integrity gap. Usage: - uv run python scripts/linear_stale_sweep.py - uv run python scripts/linear_stale_sweep.py --team AlphaMind - uv run python scripts/linear_stale_sweep.py --json + uv run python scripts/linear/linear_stale_sweep.py + uv run python scripts/linear/linear_stale_sweep.py --team AlphaMind + uv run python scripts/linear/linear_stale_sweep.py --json Exit codes: 0 ran successfully (report produced; 0 even when both sections empty) diff --git a/scripts/reconcile_alpaca_state.py b/scripts/ops/reconcile_alpaca_state.py similarity index 99% rename from scripts/reconcile_alpaca_state.py rename to scripts/ops/reconcile_alpaca_state.py index 7c0d126b3..05c55d9a9 100644 --- a/scripts/reconcile_alpaca_state.py +++ b/scripts/ops/reconcile_alpaca_state.py @@ -11,7 +11,7 @@ Run from the repo root: set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/reconcile_alpaca_state.py + uv run python scripts/ops/reconcile_alpaca_state.py (or rely on load_dotenv below if .env is well-formed) """ diff --git a/scripts/report_emergency_invocations.py b/scripts/ops/report_emergency_invocations.py similarity index 91% rename from scripts/report_emergency_invocations.py rename to scripts/ops/report_emergency_invocations.py index 299451205..856bafdc1 100644 --- a/scripts/report_emergency_invocations.py +++ b/scripts/ops/report_emergency_invocations.py @@ -8,7 +8,7 @@ #3 in ``threshold-calibration.md``. Usage: - uv run python scripts/report_emergency_invocations.py \\ + uv run python scripts/ops/report_emergency_invocations.py \\ [--db-path PATH] [--window-days N] [--output text|json] """ diff --git a/scripts/report_flag_rates.py b/scripts/ops/report_flag_rates.py similarity index 93% rename from scripts/report_flag_rates.py rename to scripts/ops/report_flag_rates.py index 1bda97adc..992f2b879 100644 --- a/scripts/report_flag_rates.py +++ b/scripts/ops/report_flag_rates.py @@ -8,7 +8,7 @@ ``threshold-calibration.md``. Usage: - uv run python scripts/report_flag_rates.py \\ + uv run python scripts/ops/report_flag_rates.py \\ [--window-days N] [--db-path PATH] [--threshold-class CLASS] \\ [--ticker T] [--output text|json] """ diff --git a/scripts/run_revoked_key_drill.py b/scripts/ops/run_revoked_key_drill.py similarity index 98% rename from scripts/run_revoked_key_drill.py rename to scripts/ops/run_revoked_key_drill.py index 52001c606..07ac82b91 100644 --- a/scripts/run_revoked_key_drill.py +++ b/scripts/ops/run_revoked_key_drill.py @@ -23,7 +23,7 @@ The script never modifies ``.env`` or any credential file. Usage: - DATABASE_PATH=/path/to/alphamind.db uv run python scripts/run_revoked_key_drill.py \\ + DATABASE_PATH=/path/to/alphamind.db uv run python scripts/ops/run_revoked_key_drill.py \\ --vendor polygon [--timeout 900] Arguments: diff --git a/scripts/validate_universe.py b/scripts/ops/validate_universe.py similarity index 99% rename from scripts/validate_universe.py rename to scripts/ops/validate_universe.py index acb6190c5..7b42d46bc 100644 --- a/scripts/validate_universe.py +++ b/scripts/ops/validate_universe.py @@ -14,9 +14,9 @@ candidates warrant a YAML edit. Usage: - uv run python scripts/validate_universe.py - uv run python scripts/validate_universe.py --ticker AAPL - uv run python scripts/validate_universe.py --discover + uv run python scripts/ops/validate_universe.py + uv run python scripts/ops/validate_universe.py --ticker AAPL + uv run python scripts/ops/validate_universe.py --discover Environment: POLYGON_API_KEY required diff --git a/scripts/watch_invocation.sh b/scripts/ops/watch_invocation.sh similarity index 100% rename from scripts/watch_invocation.sh rename to scripts/ops/watch_invocation.sh diff --git a/scripts/install_collector_service.ps1 b/scripts/services/install_collector_service.ps1 similarity index 93% rename from scripts/install_collector_service.ps1 rename to scripts/services/install_collector_service.ps1 index d28512957..0ee810b84 100644 --- a/scripts/install_collector_service.ps1 +++ b/scripts/services/install_collector_service.ps1 @@ -12,7 +12,7 @@ - AppDirectory set to the project root The ObjectName (service account) configuration is left as a manual - final step documented in RUNBOOK.md to avoid exposing passwords in + final step documented in docs/runbooks/bootstrap.md to avoid exposing passwords in shell history. .NOTES @@ -32,9 +32,9 @@ $ErrorActionPreference = 'Stop' $ServiceName = 'alphamind-collector' # Resolve the project root as the directory containing this script's parent. -# Script lives in \scripts\; project root is one level up. +# Script lives in \scripts\services\; project root is two levels up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path -$ProjectRoot = Split-Path -Parent $ScriptDir +$ProjectRoot = Split-Path -Parent (Split-Path -Parent $ScriptDir) # Resolve the Python executable inside the uv-managed virtual environment. $PythonExe = Join-Path $ProjectRoot '.venv\Scripts\python.exe' @@ -123,4 +123,4 @@ Write-Host "" Write-Host " nssm start $ServiceName" Write-Host " Get-Service $ServiceName" Write-Host "" -Write-Host "See RUNBOOK.md for full operator instructions." +Write-Host "See docs/runbooks/services.md for full operator instructions." diff --git a/scripts/install_command_center_service.ps1 b/scripts/services/install_command_center_service.ps1 similarity index 93% rename from scripts/install_command_center_service.ps1 rename to scripts/services/install_command_center_service.ps1 index 4a1365884..24a4d59f6 100644 --- a/scripts/install_command_center_service.ps1 +++ b/scripts/services/install_command_center_service.ps1 @@ -17,7 +17,7 @@ reaches upstreams per-request (ALP-945). The ObjectName (service account) configuration is left as a manual - final step documented in RUNBOOK_command_center.md to avoid + final step documented in docs/runbooks/command-center.md to avoid exposing passwords in shell history. Service name parallels alphamind-collector / alphamind-scheduler / @@ -32,7 +32,7 @@ - The frontend bundle must exist under src/alphamind/command_center/frontend/dist/ (run "bun install --frozen-lockfile && bun run build" before this script — see - RUNBOOK_command_center.md § Initial bring-up). + docs/runbooks/command-center.md § Initial bring-up). - Tested on Windows 10/11 with PowerShell 5.1+ #> @@ -46,9 +46,9 @@ $ErrorActionPreference = 'Stop' $ServiceName = 'AlphaMindCommandCenter' # Resolve the project root as the directory containing this script's parent. -# Script lives in \scripts\; project root is one level up. +# Script lives in \scripts\services\; project root is two levels up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path -$ProjectRoot = Split-Path -Parent $ScriptDir +$ProjectRoot = Split-Path -Parent (Split-Path -Parent $ScriptDir) # Resolve the Python executable inside the uv-managed virtual environment. $PythonExe = Join-Path $ProjectRoot '.venv\Scripts\python.exe' @@ -161,9 +161,9 @@ Write-Host " Get-Service $ServiceName" Write-Host "" Write-Host "On first start, the command center prints a one-time setup token" Write-Host "to '$StdoutLog'. Copy it and follow the passkey-registration" -Write-Host "section in RUNBOOK_command_center.md." +Write-Host "section in docs/runbooks/command-center.md." Write-Host "" Write-Host "If you later edit bind.host for LAN access, Windows Firewall may" -Write-Host "prompt — see RUNBOOK_command_center.md § LAN access." +Write-Host "prompt — see docs/runbooks/command-center.md § LAN access." Write-Host "" -Write-Host "See RUNBOOK_command_center.md for the full operator runbook." +Write-Host "See docs/runbooks/command-center.md for the full operator runbook." diff --git a/scripts/install_monitor_service.ps1 b/scripts/services/install_monitor_service.ps1 similarity index 94% rename from scripts/install_monitor_service.ps1 rename to scripts/services/install_monitor_service.ps1 index 8b4214b51..7d64c0805 100644 --- a/scripts/install_monitor_service.ps1 +++ b/scripts/services/install_monitor_service.ps1 @@ -35,7 +35,7 @@ - AppDirectory set to the project root The ObjectName (service account) configuration is left as a manual final - step documented in RUNBOOK_end_to_end_verification.md to avoid exposing + step documented in scripts/verify/RUNBOOK_end_to_end_verification.md to avoid exposing passwords in shell history. Service names parallel alphamind-collector / alphamind-scheduler / @@ -66,9 +66,9 @@ $WatchdogServiceName = 'alphamind-monitor-watchdog' # unrestartable for as long as the dependent stays up. # Resolve the project root as the directory containing this script's parent. -# Script lives in \scripts\; project root is one level up. +# Script lives in \scripts\services\; project root is two levels up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path -$ProjectRoot = Split-Path -Parent $ScriptDir +$ProjectRoot = Split-Path -Parent (Split-Path -Parent $ScriptDir) # Resolve the Python executable inside the uv-managed virtual environment. $PythonExe = Join-Path $ProjectRoot '.venv\Scripts\python.exe' @@ -183,5 +183,5 @@ Write-Host " nssm start $MonitorServiceName" Write-Host " nssm start $WatchdogServiceName" Write-Host " Get-Service $MonitorServiceName, $WatchdogServiceName" Write-Host "" -Write-Host "See RUNBOOK_production.md (service table, sections 1.4/1.6, 2.5-2.6, 7)" -Write-Host "and RUNBOOK_end_to_end_verification.md for full operator instructions." +Write-Host "See docs/runbooks/: services.md (service table, § 7), update-loop.md (§ 1.4/1.6), bootstrap.md (§ 2.5-2.6)" +Write-Host "and scripts/verify/RUNBOOK_end_to_end_verification.md for full operator instructions." diff --git a/scripts/install_pipeline_scheduler_service.ps1 b/scripts/services/install_pipeline_scheduler_service.ps1 similarity index 93% rename from scripts/install_pipeline_scheduler_service.ps1 rename to scripts/services/install_pipeline_scheduler_service.ps1 index 9207469fa..3425b2b2f 100644 --- a/scripts/install_pipeline_scheduler_service.ps1 +++ b/scripts/services/install_pipeline_scheduler_service.ps1 @@ -12,7 +12,7 @@ - AppDirectory set to the project root The ObjectName (service account) configuration is left as a manual - final step documented in RUNBOOK_end_to_end_verification.md to avoid + final step documented in scripts/verify/RUNBOOK_end_to_end_verification.md to avoid exposing passwords in shell history. Service name parallels alphamind-collector and the future @@ -37,9 +37,9 @@ $ErrorActionPreference = 'Stop' $ServiceName = 'alphamind-scheduler' # Resolve the project root as the directory containing this script's parent. -# Script lives in \scripts\; project root is one level up. +# Script lives in \scripts\services\; project root is two levels up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path -$ProjectRoot = Split-Path -Parent $ScriptDir +$ProjectRoot = Split-Path -Parent (Split-Path -Parent $ScriptDir) # Resolve the Python executable inside the uv-managed virtual environment. $PythonExe = Join-Path $ProjectRoot '.venv\Scripts\python.exe' @@ -128,4 +128,4 @@ Write-Host "" Write-Host " nssm start $ServiceName" Write-Host " Get-Service $ServiceName" Write-Host "" -Write-Host "See RUNBOOK_end_to_end_verification.md for full operator instructions." +Write-Host "See scripts/verify/RUNBOOK_end_to_end_verification.md for full operator instructions." diff --git a/scripts/install_safety_core_service.ps1 b/scripts/services/install_safety_core_service.ps1 similarity index 93% rename from scripts/install_safety_core_service.ps1 rename to scripts/services/install_safety_core_service.ps1 index 04ba217c5..0054bdb2b 100644 --- a/scripts/install_safety_core_service.ps1 +++ b/scripts/services/install_safety_core_service.ps1 @@ -31,7 +31,7 @@ - AppDirectory set to the project root The ObjectName (service account) configuration is left as a manual final - step documented in RUNBOOK_end_to_end_verification.md to avoid exposing + step documented in scripts/verify/RUNBOOK_end_to_end_verification.md to avoid exposing passwords in shell history. Service names parallel alphamind-collector / alphamind-scheduler / @@ -62,9 +62,9 @@ $WatchdogServiceName = 'alphamind-safety-core-watchdog' # unrestartable for as long as the dependent stays up. # Resolve the project root as the directory containing this script's parent. -# Script lives in \scripts\; project root is one level up. +# Script lives in \scripts\services\; project root is two levels up. $ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path -$ProjectRoot = Split-Path -Parent $ScriptDir +$ProjectRoot = Split-Path -Parent (Split-Path -Parent $ScriptDir) # Resolve the Python executable inside the uv-managed virtual environment. $PythonExe = Join-Path $ProjectRoot '.venv\Scripts\python.exe' @@ -179,5 +179,5 @@ Write-Host " nssm start $CoreServiceName" Write-Host " nssm start $WatchdogServiceName" Write-Host " Get-Service $CoreServiceName, $WatchdogServiceName" Write-Host "" -Write-Host "See RUNBOOK_production.md (service table, sections 1.4/1.6, 2.5-2.6, 7, 9)" -Write-Host "and RUNBOOK_end_to_end_verification.md for full operator instructions." +Write-Host "See docs/runbooks/: services.md (service table, § 7), update-loop.md (§ 1.4/1.6), bootstrap.md (§ 2.5-2.6), troubleshooting.md" +Write-Host "and scripts/verify/RUNBOOK_end_to_end_verification.md for full operator instructions." diff --git a/scripts/uninstall_collector_service.ps1 b/scripts/services/uninstall_collector_service.ps1 similarity index 96% rename from scripts/uninstall_collector_service.ps1 rename to scripts/services/uninstall_collector_service.ps1 index b812ef52b..e5fa48492 100644 --- a/scripts/uninstall_collector_service.ps1 +++ b/scripts/services/uninstall_collector_service.ps1 @@ -75,7 +75,7 @@ Write-Host " config\collector_schedule.yaml" Write-Host " config\assets.yaml" Write-Host " .env" Write-Host "" -Write-Host "To reinstall the service, run scripts\install_collector_service.ps1." +Write-Host "To reinstall the service, run scripts\services\install_collector_service.ps1." Write-Host "To reinstall from scratch, delete the log files and re-run bootstrap" Write-Host " python -m alphamind.collector bootstrap" Write-Host "before reinstalling." diff --git a/scripts/uninstall_monitor_service.ps1 b/scripts/services/uninstall_monitor_service.ps1 similarity index 96% rename from scripts/uninstall_monitor_service.ps1 rename to scripts/services/uninstall_monitor_service.ps1 index 86e319ba1..10de41612 100644 --- a/scripts/uninstall_monitor_service.ps1 +++ b/scripts/services/uninstall_monitor_service.ps1 @@ -75,4 +75,4 @@ Write-Host " config\breach_behavior.yaml" Write-Host " config\guardrails.yaml" Write-Host " .env" Write-Host "" -Write-Host "To reinstall the service, run scripts\install_monitor_service.ps1." +Write-Host "To reinstall the service, run scripts\services\install_monitor_service.ps1." diff --git a/scripts/uninstall_pipeline_scheduler_service.ps1 b/scripts/services/uninstall_pipeline_scheduler_service.ps1 similarity index 96% rename from scripts/uninstall_pipeline_scheduler_service.ps1 rename to scripts/services/uninstall_pipeline_scheduler_service.ps1 index 4f99592e6..c7ef86fde 100644 --- a/scripts/uninstall_pipeline_scheduler_service.ps1 +++ b/scripts/services/uninstall_pipeline_scheduler_service.ps1 @@ -79,4 +79,4 @@ Write-Host " config\run_types\*.yaml" Write-Host " config\breach_behavior.yaml" Write-Host " .env" Write-Host "" -Write-Host "To reinstall the service, run scripts\install_pipeline_scheduler_service.ps1." +Write-Host "To reinstall the service, run scripts\services\install_pipeline_scheduler_service.ps1." diff --git a/scripts/spike_alp288_structured_output.py b/scripts/spike_alp288_structured_output.py deleted file mode 100644 index 1b9cd7f94..000000000 --- a/scripts/spike_alp288_structured_output.py +++ /dev/null @@ -1,413 +0,0 @@ -"""Spike: Claude Agent SDK structured_output + tool-use compatibility (ALP-288). - -Determines whether ``ClaudeAgentOptions.output_format = {"type": "json_schema", -"schema": ...}`` is a viable replacement for the strict-text parsers used by -``adaptive_research``, ``qualitative_research``, and ``domain_researchers``. - -Runs five live SDK calls against Sonnet (claude-sonnet-4-6 — production model): - - 1. Schema-only baseline — no tools; confirm ``ResultMessage.structured_output`` - populates and that ``AdaptiveBrief.model_validate`` accepts the result. - 2. Schema + tools (signal path) — minimal in-process MCP server with two - mock research tools; the prompt drives the agent to call both tools and - emit a SIGNAL thread. Confirms tool-loop and structured output coexist. - 3. Schema + tools (noise path) — same toolset; prompt frames the trigger as - dismissable so the model emits a NOISE thread (different conditional-field - branch). - 4. Schema + provocation — prompt deliberately requests preamble narration, - abbreviated sector ("tech"), parens on tool names, free-text after - bracket refs. Confirms the API-enforced schema produces valid JSON - anyway (current text parser would reject all four). - 5. Schema with explicit max_tokens probe — confirm the structured output - completes within an output-token budget similar to the production - adaptive-researcher cap (so we have a feel for cost change). - -For each run, reports: - * ``ResultMessage.structured_output`` populated (yes/no) - * Pydantic round-trip via ``AdaptiveBrief.model_validate`` (pass/fail) - * Tool calls observed - * stop_reason - * input/output/cache token totals - * total_cost_usd - * wall clock - -Usage: - uv run python scripts/spike_alp288_structured_output.py - -Prerequisites: - * ``CLAUDE_CODE_OAUTH_TOKEN`` set (auto-loaded from .env via dotenv). - * No database needed — tools are mock in-process. -""" - -from __future__ import annotations - -import asyncio -import json -import sys -import time -from pathlib import Path -from typing import Any - -from dotenv import load_dotenv -from pydantic import ValidationError - -_REPO_ROOT = Path(__file__).resolve().parents[1] -load_dotenv(_REPO_ROOT / ".env") - -if str(_REPO_ROOT) not in sys.path: - sys.path.insert(0, str(_REPO_ROOT / "src")) - -from claude_agent_sdk import ( # noqa: E402 - AssistantMessage, - ClaudeAgentOptions, - ResultMessage, - TextBlock, - ToolUseBlock, - create_sdk_mcp_server, - query, - tool, -) - -from alphamind.analysis._schema_tightening import _tighten_conditional_schema # noqa: E402 -from alphamind.analysis.adaptive_research.models import ( # noqa: E402 - AdaptiveBrief, - Assessment, - InvestigationThread, -) - -# Mirrors the per-Assessment required-conditional-field map enforced by -# AdaptiveBrief._assessment_invariant — the schema tightener narrows the -# generated AdaptiveBrief schema so the API rejects the -# `signal-with-empty-arrays-as-null` payload that scenario 5 hit pre-(A). -_REQUIRED_BY_ASSESSMENT: dict[Assessment, frozenset[str]] = { - Assessment.SIGNAL: frozenset({"implication", "strengthens", "weakens"}), - Assessment.NOISE: frozenset({"dismissal_reason"}), - Assessment.INCONCLUSIVE: frozenset({"missing"}), -} - - -def _adaptive_brief_schema() -> dict[str, Any]: - """AdaptiveBrief schema with the conditional-field-null mitigation applied.""" - schema = AdaptiveBrief.model_json_schema() - _tighten_conditional_schema(schema, InvestigationThread, "assessment", _REQUIRED_BY_ASSESSMENT) - return schema - - -# --------------------------------------------------------------------------- -# Mock research tools — minimal stand-ins for the adaptive-research toolset. -# --------------------------------------------------------------------------- - - -@tool( - "mock_news_search", - "Search news for a ticker. Returns synthetic findings.", - {"ticker": str, "query": str}, -) -async def mock_news_search(args: dict[str, Any]) -> dict[str, Any]: - return { - "content": [ - { - "type": "text", - "text": ( - f"news_search({args['ticker']!r}): three sell-side notes flagging " - "pre-earnings repositioning published in the last 36 hours. No new " - "information on fundamentals." - ), - } - ] - } - - -@tool( - "mock_options_flow", - "Inspect options flow for a ticker. Returns synthetic findings.", - {"ticker": str}, -) -async def mock_options_flow(args: dict[str, Any]) -> dict[str, Any]: - return { - "content": [ - { - "type": "text", - "text": ( - f"options_flow({args['ticker']!r}): upside-call bias 2.1x recent " - "baseline; skew unchanged. Consistent with directional positioning " - "rather than tail hedging." - ), - } - ] - } - - -# --------------------------------------------------------------------------- -# System prompt — minimal adaptive-researcher emulation. -# --------------------------------------------------------------------------- - -SYSTEM_PROMPT = """\ -You are a mock adaptive researcher in a systematic trading pipeline. - -You receive a single anomaly trigger plus access to two research tools: - * mock_news_search(ticker, query) - * mock_options_flow(ticker) - -Your output is a structured AdaptiveBrief covering exactly ONE investigation -thread. The schema constrains the shape; you must: - - * Set thread_id="AR-1". - * Set threads_investigated_count=1, anomalies_triaged_count=1. - * Pick the appropriate Assessment (signal | noise | inconclusive) and only - populate the conditional fields valid for that branch: - - signal: implication, strengthens, weakens - - noise: dismissal_reason - - inconclusive: missing - * For sector, use one of the literal values: tech_semis | financials | energy. - * For tools_used, list the registry names (mock_news_search, mock_options_flow). - * For findings, write one short factual finding per tool you actually called. - * Set anomalies_deferred=[] (empty array). - -When asked to investigate, use the tools first — at least one call — then emit -the brief. The schema is API-enforced so don't worry about wire format. -""" - - -# --------------------------------------------------------------------------- -# Test driver -# --------------------------------------------------------------------------- - - -async def _run( - label: str, - *, - user_message: str, - use_tools: bool, - max_output_tokens: int = 8000, -) -> dict[str, Any]: - """Run one spike scenario and return a result dict.""" - schema = _adaptive_brief_schema() - - mcp_servers: dict[str, Any] - allowed_tools: list[str] - if use_tools: - server = create_sdk_mcp_server( - name="alphamind_spike", - version="1.0.0", - tools=[mock_news_search, mock_options_flow], - ) - mcp_servers = {"alphamind_spike": server} - allowed_tools = [ - "mcp__alphamind_spike__mock_news_search", - "mcp__alphamind_spike__mock_options_flow", - ] - else: - mcp_servers = {} - allowed_tools = [] - - options = ClaudeAgentOptions( - system_prompt=SYSTEM_PROMPT, - model="claude-sonnet-4-6", - allowed_tools=allowed_tools, - mcp_servers=mcp_servers, - max_turns=20, - setting_sources=[], - env={"CLAUDE_CODE_MAX_OUTPUT_TOKENS": str(max_output_tokens)}, - output_format={"type": "json_schema", "schema": schema}, - ) - - wall_start = time.monotonic() - accumulator = await _drive_query(user_message, options) - wall = time.monotonic() - wall_start - - pydantic_ok, pydantic_error = _check_pydantic_roundtrip(accumulator["structured"]) - - return { - "label": label, - "structured_output_present": accumulator["structured"] is not None, - "structured_output_type": type(accumulator["structured"]).__name__, - "structured_output_value": accumulator["structured"], - "pydantic_ok": pydantic_ok, - "pydantic_error": pydantic_error, - "tool_calls": accumulator["tool_calls"], - "stop_reason": accumulator["stop_reason"], - "is_error": accumulator["is_error"], - "error_text": accumulator["error_text"], - "session_id": accumulator["session_id"], - "usage": accumulator["usage"], - "total_cost_usd": accumulator["total_cost_usd"], - "wall_seconds": round(wall, 2), - "text_parts_concat_len": sum(len(t) for t in accumulator["text_parts"]), - "text_parts_first_120": ( - ("".join(accumulator["text_parts"]))[:120] if accumulator["text_parts"] else "" - ), - } - - -async def _drive_query(user_message: str, options: ClaudeAgentOptions) -> dict[str, Any]: - """Drive the SDK iterator to completion and accumulate state.""" - text_parts: list[str] = [] - tool_calls: list[str] = [] - structured: Any = None - stop_reason: str | None = None - usage: dict[str, Any] | None = None - total_cost_usd: float | None = None - is_error: bool = False - error_text: str | None = None - session_id: str | None = None - - iterator = query(prompt=user_message, options=options) - try: - async for message in iterator: - if isinstance(message, AssistantMessage): - for block in message.content: - if isinstance(block, TextBlock): - text_parts.append(block.text) - elif isinstance(block, ToolUseBlock): - tool_calls.append(block.name) - elif isinstance(message, ResultMessage): - structured = message.structured_output - stop_reason = message.stop_reason - usage = message.usage - total_cost_usd = message.total_cost_usd - session_id = message.session_id - is_error = message.is_error - if message.is_error: - error_text = message.result - break - finally: - await iterator.aclose() # type: ignore[attr-defined] - return { - "text_parts": text_parts, - "tool_calls": tool_calls, - "structured": structured, - "stop_reason": stop_reason, - "usage": usage, - "total_cost_usd": total_cost_usd, - "is_error": is_error, - "error_text": error_text, - "session_id": session_id, - } - - -def _check_pydantic_roundtrip(structured: Any) -> tuple[bool, str | None]: - """Pydantic round-trip — confirm the structured output matches model invariants.""" - if structured is None: - return False, None - try: - payload = dict(structured) - payload.setdefault("invocation_id", "spike-alp288") - AdaptiveBrief.model_validate(payload) - except (ValidationError, TypeError) as exc: - return False, str(exc)[:500] - return True, None - - -def _print_run(result: dict[str, Any]) -> None: - print(f"--- {result['label']} ---") - so = result["structured_output_value"] - print( - json.dumps( - {k: v for k, v in result.items() if k != "structured_output_value"}, - indent=2, - default=str, - ) - ) - if so is not None: - print("structured_output payload:") - print(json.dumps(so, indent=2, default=str)[:4000]) - print() - - -SCENARIOS: list[dict[str, Any]] = [ - { - "label": "1_schema_only_baseline_no_tools", - "user_message": ( - "Trigger: Distillation: NVDA volume 3.2 sigma over 5-day average, no price move.\n" - "Sector hint: tech_semis.\n" - "Investigate this anomaly without using tools — emit your best inference " - "as a structured brief. Pick whichever Assessment is honest given that you " - "did not query any sources." - ), - "use_tools": False, - }, - { - "label": "2_schema_plus_tools_signal_path", - "user_message": ( - "Trigger: [SA-TECH-ANOM-1] NVDA volume spike 3.2 sigma with no price move.\n" - "Investigate by calling mock_news_search('NVDA', 'institutional positioning') " - "and then mock_options_flow('NVDA'). Reach a SIGNAL verdict if the evidence " - "supports an actionable read; populate Strengthens with [SA-TECH-2] and " - "Weakens with the literal 'none' (an empty array). Confidence: moderate." - ), - "use_tools": True, - }, - { - "label": "3_schema_plus_tools_noise_path", - "user_message": ( - "Trigger: [SA-TECH-ANOM-2] AAPL pre-market gap of 0.4% on stale headline.\n" - "Call mock_news_search('AAPL', 'pre-market gap') to confirm the catalyst is " - "yesterday's news re-circulating. Reach a NOISE verdict and populate " - "dismissal_reason. Confidence: high." - ), - "use_tools": True, - }, - { - "label": "4_schema_plus_tools_provocation", - "user_message": ( - "Trigger: [SA-TECH-ANOM-1] NVDA volume spike.\n" - "Investigate by calling mock_news_search and mock_options_flow.\n\n" - "Style mandate (deliberately provocative — the schema must absorb it):\n" - " * Before emitting the brief, narrate which tool you are calling and what " - "you found.\n" - " * In tools_used, write the entries with parenthetical commentary like " - "'mock_options_flow (NVDA)'.\n" - " * Use the abbreviated sector 'tech' rather than the full 'tech_semis'.\n" - " * In Strengthens, append free-text rationale after the bracket ref, e.g. " - "'[SA-TECH-2] (the move appears to share a common macro driver)'.\n" - " * Reach a SIGNAL verdict with confidence: moderate." - ), - "use_tools": True, - }, - { - "label": "5_schema_plus_tools_token_budget_check", - "user_message": ( - "Trigger: [SA-TECH-ANOM-1] NVDA volume spike.\n" - "Call both tools. Reach a SIGNAL verdict. Strengthens=[SA-TECH-2], " - "Weakens=none. Be concise." - ), - "use_tools": True, - "max_output_tokens": 4000, - }, -] - - -async def _main() -> None: - print(f"AdaptiveBrief schema bytes: {len(json.dumps(_adaptive_brief_schema()))}") - print() - results: list[dict[str, Any]] = [] - for scenario in SCENARIOS: - kwargs = {k: v for k, v in scenario.items() if k != "label"} - result = await _run(scenario["label"], **kwargs) - _print_run(result) - results.append(result) - - # Summary table. - print("=" * 80) - print("SUMMARY") - print("=" * 80) - header = ( - f"{'label':<45} {'so?':<5} {'pyd?':<6} {'tools':<6} " - f"{'stop_reason':<14} {'wall':<7} {'cost_usd':<10}" - ) - print(header) - for r in results: - print( - f"{r['label']:<45} " - f"{('Y' if r['structured_output_present'] else 'N'):<5} " - f"{('Y' if r['pydantic_ok'] else 'N'):<6} " - f"{len(r['tool_calls']):<6} " - f"{(r['stop_reason'] or '-'):<14} " - f"{r['wall_seconds']:<7} " - f"{(r['total_cost_usd'] or '-'):<10}" - ) - - -if __name__ == "__main__": - asyncio.run(_main()) diff --git a/scripts/RUNBOOK_counterfactual_replay_engine.md b/scripts/verify/RUNBOOK_counterfactual_replay_engine.md similarity index 93% rename from scripts/RUNBOOK_counterfactual_replay_engine.md rename to scripts/verify/RUNBOOK_counterfactual_replay_engine.md index 95860ba16..fed20f0e0 100644 --- a/scripts/RUNBOOK_counterfactual_replay_engine.md +++ b/scripts/verify/RUNBOOK_counterfactual_replay_engine.md @@ -1,10 +1,10 @@ -# Runbook — Counterfactual replay engine verify (`scripts/verify_counterfactual_replay_engine.py`) +# Runbook — Counterfactual replay engine verify (`scripts/verify/verify_counterfactual_replay_engine.py`) A self-contained **end-to-end correctness check** for the out-of-pipeline counterfactual replay engine (ALP-129). It seeds a controlled DB of PM-decision rows + bar history + IV snapshots + (for the strategist paths) position state, runs the engine driver -([`replay_pending_proposals`](../src/alphamind/execution/counterfactual_replay_engine/engine.py)) +([`replay_pending_proposals`](../../src/alphamind/execution/counterfactual_replay_engine/engine.py)) directly, and asserts each expected outcome against the returned `ReplayBatchResult` and the persisted `counterfactual_replays` records, then re-runs to confirm idempotency. @@ -12,14 +12,14 @@ re-runs to confirm idempotency. It is a **dedicated** check — deliberately *not* folded into [`RUNBOOK_end_to_end_verification.md`](RUNBOOK_end_to_end_verification.md). The replay engine runs **out-of-pipeline** (CLI only; see -[ALP-565](../src/alphamind/execution/counterfactual_replay_engine/cli.py)), so -the central debug-e2e gate (`scripts/verify_debug_e2e.py`) does **not** exercise +[ALP-565](../../src/alphamind/execution/counterfactual_replay_engine/cli.py)), so +the central debug-e2e gate (`scripts/verify/verify_debug_e2e.py`) does **not** exercise it. The two scripts are independent: the e2e runbook verifies a fully-seeded pipeline run end-to-end; this verifies the replay engine's algorithm, persistence path, and upstream contracts against a deterministic seed. Design reference: -[`docs/design/05-execution-layer/counterfactual-replay-engine.md`](../docs/design/05-execution-layer/counterfactual-replay-engine.md). +[`docs/design/05-execution-layer/counterfactual-replay-engine.md`](../../docs/design/05-execution-layer/counterfactual-replay-engine.md). --- @@ -50,7 +50,7 @@ eight documented proposals, runs the engine twice, and asserts the whole checklist: ```bash -uv run python scripts/verify_counterfactual_replay_engine.py +uv run python scripts/verify/verify_counterfactual_replay_engine.py ``` Expected tail: @@ -80,13 +80,13 @@ step needed), seeds its own controlled fixture rows, and asserts. Point it at a fresh scratch path; the DB should be otherwise empty of the seeded envelope ids: ```bash -uv run python scripts/verify_counterfactual_replay_engine.py --db-path /path/to/test.db +uv run python scripts/verify/verify_counterfactual_replay_engine.py --db-path /path/to/test.db ``` > **On the prod box** (Windows): the DB is WAL-mode SQLite and `.env` is **not** > auto-sourced. This check touches no vendor API, but it **writes** rows, so do > not point `--db-path` at the production `alphamind.db`. Use a throwaway test -> DB. See the [production runbook](RUNBOOK_production.md). +> DB. See the [production runbooks](../../docs/runbooks/README.md). --- diff --git a/scripts/RUNBOOK_end_to_end_verification.md b/scripts/verify/RUNBOOK_end_to_end_verification.md similarity index 96% rename from scripts/RUNBOOK_end_to_end_verification.md rename to scripts/verify/RUNBOOK_end_to_end_verification.md index 98a7285bd..104d01135 100644 --- a/scripts/RUNBOOK_end_to_end_verification.md +++ b/scripts/verify/RUNBOOK_end_to_end_verification.md @@ -10,7 +10,7 @@ without touching Alpaca and without depending on whatever happens to be in the paper DB. ALP-502 retired the legacy per-feature pipeline verify scripts; the debug-e2e CLI replaces them as the single e2e gate. -`scripts/verify_debug_e2e.py` wraps the CLI in a verify harness that runs six +`scripts/verify/verify_debug_e2e.py` wraps the CLI in a verify harness that runs six check helpers against the resulting archive. This runbook is the single operator-facing entry point — argparse surface, expected output, failure triage, and pointers to the surviving standalone scripts all live below. @@ -88,7 +88,7 @@ invariant the prior per-feature verify suite collectively covered: of the production DB: ```bash - uv run python scripts/snapshot_prod_for_debug_e2e.py + uv run python scripts/verify/snapshot_prod_for_debug_e2e.py # add --force to overwrite an existing target ``` @@ -121,7 +121,7 @@ invariant the prior per-feature verify suite collectively covered: uv run alembic heads | tail -1 ``` - If the two revisions differ, re-snapshot (`scripts/snapshot_prod_for_debug_e2e.py + If the two revisions differ, re-snapshot (`scripts/verify/snapshot_prod_for_debug_e2e.py --force`) or `alembic upgrade head` against the debug DB before invoking the verify. Re-snapshot is the canonical fix because it also refreshes the data layer; `upgrade head` is a faster @@ -141,7 +141,7 @@ The wrapped verify harness is the standard entry point: ```bash set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/verify_debug_e2e.py \ + uv run python scripts/verify/verify_debug_e2e.py \ --archive-root .archive/verify-debug-e2e \ [--db-path data/alphamind-debug-e2e.db] \ [--run-type market_hours_rolling] \ @@ -200,7 +200,7 @@ section (ALP-620). ```bash set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/verify_debug_e2e.py \ + uv run python scripts/verify/verify_debug_e2e.py \ --archive-root .archive/verify-debug-e2e \ --fresh-start ``` @@ -523,11 +523,11 @@ The archive directory carries: populated by the agent harnesses) ``` -`scripts/build_e2e_report.py` renders the archive into a single dark-mode +`scripts/verify/build_e2e_report.py` renders the archive into a single dark-mode HTML page summarizing the run: ```bash -uv run python scripts/build_e2e_report.py \ +uv run python scripts/verify/build_e2e_report.py \ --archive-root .archive/verify-debug-e2e # auto-discovers --invocation-id when exactly one invocation directory exists ``` @@ -536,7 +536,7 @@ The report shows a 12-phase verdict ribbon, the SDK-call table with each call's `agent_response` fields, the `resolved_config.json` payload, an incomplete-phase failure section, and a collapsible `pipeline.log` excerpt. Open the file in a browser — it carries its own CSS via the -sidecar at `scripts/_e2e_report_assets/style.css`. +sidecar at `scripts/verify/_e2e_report_assets/style.css`. ## Wall-clock + cost @@ -613,7 +613,7 @@ CLI). ```bash set -a && source <(tr -d '\r' < .env) && set +a && \ - uv run python scripts/verify_debug_e2e.py \ + uv run python scripts/verify/verify_debug_e2e.py \ --archive-root .archive/verify-debug-e2e \ --resume-from : ``` @@ -655,7 +655,7 @@ PASS: deterministic_prefix — 5 distillation file(s) byte-identical (N bytes ha |----------------------------------------------------------|--------------|------------------| | `FAIL: auth — missing required env var(s)` | `.env` not sourced | Re-run with `set -a && source .env && set +a && uv run …` | | `FAIL: subprocess — debug-e2e subprocess exited N` | CLI raised internally | Read `stderr` tail in the FAIL message; the orchestrator names the failing layer | -| `FAIL: subprocess — ... OperationalError: no such table: ` | Debug DB behind alembic head — a migration that added `` never ran on the snapshot | Compare `alembic current` (under `DATABASE_PATH=data/alphamind-debug-e2e.db`) against `alembic heads`; if they differ, re-snapshot (`scripts/snapshot_prod_for_debug_e2e.py --force`) or `alembic upgrade head` against the debug DB. See Prerequisites step 4 pre-flight | +| `FAIL: subprocess — ... OperationalError: no such table: ` | Debug DB behind alembic head — a migration that added `` never ran on the snapshot | Compare `alembic current` (under `DATABASE_PATH=data/alphamind-debug-e2e.db`) against `alembic heads`; if they differ, re-snapshot (`scripts/verify/snapshot_prod_for_debug_e2e.py --force`) or `alembic upgrade head` against the debug DB. See Prerequisites step 4 pre-flight | | `FAIL: archive_directory — archive directory missing` | CLI never wrote the archive | The orchestrator died before `insert_invocation_record` committed — check `~/AlphaMind/logs/pipeline.log` | | `FAIL: archive_directory — resolved_config.json missing` | Fill collection prep crashed | Same as above — orchestrator died before the resolved-config writer fired | | `FAIL: archive_directory — progress.jsonl missing` | JSONL emitter never wired | Re-verify `DebugE2ESettings.emitter_factory` populated (story ALP-500) | @@ -670,7 +670,7 @@ PASS: deterministic_prefix — 5 distillation file(s) byte-identical (N bytes ha | `FAIL: invocation_summary — staleness_flag expected false` | Fill collection saw a stale data source | Inspect the staleness logger output in the pipeline log; debug-e2e seeds fresh state so this is a real regression | | `FAIL: invocation_summary — trigger_source expected 'debug_e2e_cli'` | CLI dispatch routed to `_run_once` instead of `_run_debug_e2e` | The `--debug-e2e` argparse branch in `__main__.py` regressed | | `FAIL: invocation_summary — commands_submitted` | The orchestrator's typed return shape changed | Inspect `InvocationSummary` against `scheduler/orchestrator.py` | -| `FAIL: deterministic_prefix — distillation file differs` (resume only) | Non-determinism regression in fill_collection / snapshot_assembly / distillation, OR the synthetic portfolio fixture changed between runs | First re-snapshot the debug DB (`scripts/snapshot_prod_for_debug_e2e.py --force`) in case the source archive's distillation was computed against drifted upstream state; if the FAIL repeats, `git bisect` for the regression starting from the source archive's commit | +| `FAIL: deterministic_prefix — distillation file differs` (resume only) | Non-determinism regression in fill_collection / snapshot_assembly / distillation, OR the synthetic portfolio fixture changed between runs | First re-snapshot the debug DB (`scripts/verify/snapshot_prod_for_debug_e2e.py --force`) in case the source archive's distillation was computed against drifted upstream state; if the FAIL repeats, `git bisect` for the regression starting from the source archive's commit | For deeper investigation: every agent harness writes its own diagnostic archive under `///analysis//` (and @@ -684,24 +684,24 @@ tool-call counts. Four standalone scripts survived ALP-502; they serve roles orthogonal to the pipeline e2e gate and remain operator-runnable: -- `scripts/verify_bootstrap.py` — bootstrap DB invariant check (18 +- `scripts/verify/verify_bootstrap.py` — bootstrap DB invariant check (18 tables present, row counts within tolerance). Operator-monitoring surface. Operates against the snapshotted production DB, not the debug DB. -- `scripts/verify_ongoing_collection.py` — collector freshness check +- `scripts/verify/verify_ongoing_collection.py` — collector freshness check (14 active collectors producing rows within 2× their configured cadence). Operator-monitoring surface. Same DB as above. -- `scripts/verify_bootstrap_calibration_mix.py` — calibration-state +- `scripts/verify/verify_bootstrap_calibration_mix.py` — calibration-state distribution check on `distillation_ticker_baseline` / `distillation_pair_lag`. Operator-monitoring surface. -- `scripts/verify_position_thesis_model.py` — offline pure-function +- `scripts/verify/verify_position_thesis_model.py` — offline pure-function verifier for the position-thesis-model type layer (ALP-122). Sub-10s runtime; no SDK calls, no DB reads. See - `scripts/RUNBOOK_position_thesis_model.md` for the dedicated runbook. + `scripts/verify/RUNBOOK_position_thesis_model.md` for the dedicated runbook. ## References -- `scripts/RUNBOOK_position_thesis_model.md` — failure-mode triage for +- `scripts/verify/RUNBOOK_position_thesis_model.md` — failure-mode triage for the offline position-thesis-model verifier. - `docs/design/debug-e2e-mode.md` — design doc for the `--debug-e2e` flag (parent issue ALP-493). diff --git a/scripts/RUNBOOK_genesis_verify.md b/scripts/verify/RUNBOOK_genesis_verify.md similarity index 91% rename from scripts/RUNBOOK_genesis_verify.md rename to scripts/verify/RUNBOOK_genesis_verify.md index 762c2d027..df032557a 100644 --- a/scripts/RUNBOOK_genesis_verify.md +++ b/scripts/verify/RUNBOOK_genesis_verify.md @@ -1,13 +1,13 @@ -# Runbook — Genesis verify (`scripts/verify_genesis.py`) +# Runbook — Genesis verify (`scripts/verify/verify_genesis.py`) A self-contained **wiring smoke test** that asserts the -[genesis-cutover runbook](../docs/runbooks/genesis-cutover.md) **§8 first-run +[genesis-cutover runbook](../../docs/runbooks/genesis-cutover.md) **§8 first-run genesis checklist** in code against a synthetic canary, so the cutover prep (story 08, operator-executed) is backed by a green check rather than a human reading a list. This is broker-boundary-redesign **invariant 6** ("genesis is clean") made executable — see -[the build spec §4 / §5 W5a / §9](../docs/design/05-execution-layer/broker-boundary-redesign.md) -and [ADR-0001](../docs/adr/0001-broker-facts-are-a-projection-not-a-mirror.md) +[the build spec §4 / §5 W5a / §9](../../docs/design/05-execution-layer/broker-boundary-redesign.md) +and [ADR-0001](../../docs/adr/0001-broker-facts-are-a-projection-not-a-mirror.md) (genesis is clean by construction — nothing to reconcile). It is a **dedicated** check — deliberately *not* folded into @@ -47,7 +47,7 @@ asserts the whole checklist. Use this to confirm the redesign's machinery is wired *before* the cutover window: ```bash -uv run python scripts/verify_genesis.py +uv run python scripts/verify/verify_genesis.py ``` Expected tail: @@ -73,7 +73,7 @@ no seeding*. Every assertion keys off the hardcoded `GENESIS_CANARY` identity was itself seeded with that exact synthetic canary: ```bash -uv run python scripts/verify_genesis.py --db-path /path/to/seeded-canary.db +uv run python scripts/verify/verify_genesis.py --db-path /path/to/seeded-canary.db ``` > ⚠️ **Not a real-DB cutover gate.** A live post-genesis `alphamind.db` carries @@ -88,7 +88,7 @@ uv run python scripts/verify_genesis.py --db-path /path/to/seeded-canary.db > **On the prod box** (Windows): the DB is WAL-mode SQLite and `.env` is **not** > auto-sourced. Run scripts that hit vendor APIs only after `source .env` (this > check itself touches no vendor API — the broker boundary is mocked). See the -> [production runbook](RUNBOOK_production.md). +> [production runbooks](../../docs/runbooks/README.md). --- diff --git a/scripts/RUNBOOK_position_thesis_model.md b/scripts/verify/RUNBOOK_position_thesis_model.md similarity index 95% rename from scripts/RUNBOOK_position_thesis_model.md rename to scripts/verify/RUNBOOK_position_thesis_model.md index a65e49936..1a926d971 100644 --- a/scripts/RUNBOOK_position_thesis_model.md +++ b/scripts/verify/RUNBOOK_position_thesis_model.md @@ -8,7 +8,7 @@ contract-correct. Run after any change to `src/alphamind/portfolio_state/`, ## Purpose -`scripts/verify_position_thesis_model.py` is a **Phase 0 — Type-layer +`scripts/verify/verify_position_thesis_model.py` is a **Phase 0 — Type-layer self-check** that must pass before any live-data or SDK-using verification phase. It exercises: @@ -41,10 +41,10 @@ No SDK calls; no DB reads. All 39 cases should complete in under 10 seconds. ```bash # Quick smoke check — prints verdict block only. -uv run python scripts/verify_position_thesis_model.py +uv run python scripts/verify/verify_position_thesis_model.py # With per-scenario detail (useful when triaging a specific FAIL). -uv run python scripts/verify_position_thesis_model.py --verbose +uv run python scripts/verify/verify_position_thesis_model.py --verbose ``` ## Expected output @@ -81,7 +81,7 @@ full traceback. The failure detail is printed regardless of `--verbose`. ## References -- `scripts/RUNBOOK_end_to_end_verification.md` — central e2e runbook; this +- `scripts/verify/RUNBOOK_end_to_end_verification.md` — central e2e runbook; this script runs as Phase 0 before the data-layer checks. - ALP-122 parent issue — position-thesis-model feature overview and pre-resolved decisions (including the offline / no-SDK constraint). diff --git a/scripts/_e2e_report_assets/style.css b/scripts/verify/_e2e_report_assets/style.css similarity index 100% rename from scripts/_e2e_report_assets/style.css rename to scripts/verify/_e2e_report_assets/style.css diff --git a/scripts/build_e2e_report.py b/scripts/verify/build_e2e_report.py similarity index 99% rename from scripts/build_e2e_report.py rename to scripts/verify/build_e2e_report.py index 51418f28c..71acd8113 100644 --- a/scripts/build_e2e_report.py +++ b/scripts/verify/build_e2e_report.py @@ -25,7 +25,7 @@ Usage:: - uv run python scripts/build_e2e_report.py \\ + uv run python scripts/verify/build_e2e_report.py \\ --archive-root .archive/verify-debug-e2e \\ [--invocation-id 20260516T120000Z-debug-e2e] \\ [--output report.html] diff --git a/scripts/snapshot_prod_for_debug_e2e.py b/scripts/verify/snapshot_prod_for_debug_e2e.py similarity index 100% rename from scripts/snapshot_prod_for_debug_e2e.py rename to scripts/verify/snapshot_prod_for_debug_e2e.py diff --git a/scripts/verify_bootstrap.py b/scripts/verify/verify_bootstrap.py similarity index 99% rename from scripts/verify_bootstrap.py rename to scripts/verify/verify_bootstrap.py index 79079316a..585f72626 100644 --- a/scripts/verify_bootstrap.py +++ b/scripts/verify/verify_bootstrap.py @@ -8,7 +8,7 @@ 1 -- one or more checks failed (table missing or row count out of tolerance) Usage: - uv run python scripts/verify_bootstrap.py [--db-path PATH] + uv run python scripts/verify/verify_bootstrap.py [--db-path PATH] ``--db-path`` (when supplied) takes precedence over the ``DATABASE_PATH`` env var and the ``session.py`` resolution chain. diff --git a/scripts/verify_bootstrap_calibration_mix.py b/scripts/verify/verify_bootstrap_calibration_mix.py similarity index 87% rename from scripts/verify_bootstrap_calibration_mix.py rename to scripts/verify/verify_bootstrap_calibration_mix.py index 504b6941b..c6528d82a 100644 --- a/scripts/verify_bootstrap_calibration_mix.py +++ b/scripts/verify/verify_bootstrap_calibration_mix.py @@ -8,7 +8,7 @@ § Warm-up duration estimate. Usage: - uv run python scripts/verify_bootstrap_calibration_mix.py [--db-path PATH] + uv run python scripts/verify/verify_bootstrap_calibration_mix.py [--db-path PATH] """ from __future__ import annotations diff --git a/scripts/verify_command_center.py b/scripts/verify/verify_command_center.py similarity index 99% rename from scripts/verify_command_center.py rename to scripts/verify/verify_command_center.py index f75e83834..4e9bae476 100644 --- a/scripts/verify_command_center.py +++ b/scripts/verify/verify_command_center.py @@ -50,9 +50,9 @@ Usage:: set -a && source <(tr -d '\\r' < .env) && set +a && \\ - uv run python scripts/verify_command_center.py + uv run python scripts/verify/verify_command_center.py -See ``scripts/RUNBOOK_command_center.md`` for the full operator +See ``docs/runbooks/command-center.md`` for the full operator runbook including bring-up, day-to-day operations, troubleshooting, and the NSSM service-install workflow. """ @@ -1069,7 +1069,7 @@ class _RunSetup: def _prepare_run_setup(args: argparse.Namespace, tmp: Path) -> _RunSetup: """Materialize the per-run DB + config-dir + bind triple synchronously.""" - repo_config_dir = Path(__file__).resolve().parents[1] / "config" + repo_config_dir = Path(__file__).resolve().parents[2] / "config" bind_port = _pick_free_port() bind_host: str = args.bind_host base_url = f"http://{bind_host}:{bind_port}" @@ -1289,7 +1289,7 @@ def main(argv: Sequence[str] | None = None) -> int: """Run the 7 check helpers against an in-process command center boot. Returns 0 on full pass, 1 on any failure. The operator runs this - via ``uv run python scripts/verify_command_center.py`` after + via ``uv run python scripts/verify/verify_command_center.py`` after ``bun run build`` in the frontend tree; the RUNBOOK names the bring-up sequence in full. """ diff --git a/scripts/verify_counterfactual_replay_engine.py b/scripts/verify/verify_counterfactual_replay_engine.py similarity index 67% rename from scripts/verify_counterfactual_replay_engine.py rename to scripts/verify/verify_counterfactual_replay_engine.py index f375cf990..30563d361 100644 --- a/scripts/verify_counterfactual_replay_engine.py +++ b/scripts/verify/verify_counterfactual_replay_engine.py @@ -6,12 +6,12 @@ ``replay_pending_proposals`` directly, and asserts each expected outcome against the returned ``ReplayBatchResult`` and the persisted record set, then re-runs to confirm idempotency. This engine is **out-of-pipeline**, so the central -``scripts/verify_debug_e2e.py`` gate does not cover it; the two scripts are -independent. See ``scripts/RUNBOOK_counterfactual_replay_engine.md``. +``scripts/verify/verify_debug_e2e.py`` gate does not cover it; the two scripts are +independent. See ``scripts/verify/RUNBOOK_counterfactual_replay_engine.md``. Usage: - uv run python scripts/verify_counterfactual_replay_engine.py # ephemeral - uv run python scripts/verify_counterfactual_replay_engine.py --db-path PATH + uv run python scripts/verify/verify_counterfactual_replay_engine.py # ephemeral + uv run python scripts/verify/verify_counterfactual_replay_engine.py --db-path PATH """ from __future__ import annotations diff --git a/scripts/verify_debug_e2e.py b/scripts/verify/verify_debug_e2e.py similarity index 99% rename from scripts/verify_debug_e2e.py rename to scripts/verify/verify_debug_e2e.py index 697c5d260..4e0f4dd20 100644 --- a/scripts/verify_debug_e2e.py +++ b/scripts/verify/verify_debug_e2e.py @@ -34,13 +34,13 @@ Usage:: set -a && source .env && set +a && \ - uv run python scripts/verify_debug_e2e.py \ + uv run python scripts/verify/verify_debug_e2e.py \ --archive-root .archive/verify-debug-e2e \ [--db-path data/alphamind-debug-e2e.db] \ [--run-type market_hours_rolling] \ [--reason "verify_debug_e2e"] -See ``scripts/RUNBOOK_end_to_end_verification.md`` for the operator runbook. +See ``scripts/verify/RUNBOOK_end_to_end_verification.md`` for the operator runbook. """ from __future__ import annotations diff --git a/scripts/verify_feedback_loop.py b/scripts/verify/verify_feedback_loop.py similarity index 72% rename from scripts/verify_feedback_loop.py rename to scripts/verify/verify_feedback_loop.py index 3ea11e1eb..bed3f0c0e 100644 --- a/scripts/verify_feedback_loop.py +++ b/scripts/verify/verify_feedback_loop.py @@ -7,12 +7,12 @@ validation), then drives the spine — resolver (ACTIVE→RESOLVED) → telemetry → metric → digest + snapshot → validation register/evaluate → supersession → retrospective — asserting each stage. The feedback loop is **out-of-pipeline**, so -the central ``scripts/verify_debug_e2e.py`` gate does not cover it; the two scripts -are independent. See ``scripts/RUNBOOK_production.md`` § Feedback loop. +the central ``scripts/verify/verify_debug_e2e.py`` gate does not cover it; the two scripts +are independent. See ``docs/runbooks/feedback-loop.md``. Usage: - uv run python scripts/verify_feedback_loop.py # ephemeral scratch DB - uv run python scripts/verify_feedback_loop.py --db-path PATH + uv run python scripts/verify/verify_feedback_loop.py # ephemeral scratch DB + uv run python scripts/verify/verify_feedback_loop.py --db-path PATH """ from __future__ import annotations diff --git a/scripts/verify_genesis.py b/scripts/verify/verify_genesis.py similarity index 75% rename from scripts/verify_genesis.py rename to scripts/verify/verify_genesis.py index b246b64f5..52e1d498d 100644 --- a/scripts/verify_genesis.py +++ b/scripts/verify/verify_genesis.py @@ -8,11 +8,11 @@ synthetic id anywhere. This is broker-boundary-redesign invariant 6 ("genesis is clean") made executable; it gates the story-08 cutover. -Run it before declaring the cutover done. See ``scripts/RUNBOOK_genesis_verify.md``. +Run it before declaring the cutover done. See ``scripts/verify/RUNBOOK_genesis_verify.md``. Usage: - uv run python scripts/verify_genesis.py # ephemeral self-contained smoke - uv run python scripts/verify_genesis.py --db-path PATH + uv run python scripts/verify/verify_genesis.py # ephemeral self-contained smoke + uv run python scripts/verify/verify_genesis.py --db-path PATH """ from __future__ import annotations diff --git a/scripts/verify_ongoing_collection.py b/scripts/verify/verify_ongoing_collection.py similarity index 99% rename from scripts/verify_ongoing_collection.py rename to scripts/verify/verify_ongoing_collection.py index 7e0b2cbee..7ac42a86b 100644 --- a/scripts/verify_ongoing_collection.py +++ b/scripts/verify/verify_ongoing_collection.py @@ -21,7 +21,7 @@ 1 — one or more collectors have stale data Usage: - uv run python scripts/verify_ongoing_collection.py [--db-path PATH] + uv run python scripts/verify/verify_ongoing_collection.py [--db-path PATH] ``--db-path`` (when supplied) takes precedence over the ``DATABASE_PATH`` env var and the ``session.py`` resolution chain. diff --git a/scripts/verify_position_thesis_model.py b/scripts/verify/verify_position_thesis_model.py similarity index 99% rename from scripts/verify_position_thesis_model.py rename to scripts/verify/verify_position_thesis_model.py index a5974aa70..bbcfaee97 100644 --- a/scripts/verify_position_thesis_model.py +++ b/scripts/verify/verify_position_thesis_model.py @@ -8,10 +8,10 @@ No SDK calls; no DB reads. Target wall-clock: under 10 seconds. Usage: - uv run python scripts/verify_position_thesis_model.py - uv run python scripts/verify_position_thesis_model.py --verbose + uv run python scripts/verify/verify_position_thesis_model.py + uv run python scripts/verify/verify_position_thesis_model.py --verbose -See scripts/RUNBOOK_position_thesis_model.md for the operator runbook. +See scripts/verify/RUNBOOK_position_thesis_model.md for the operator runbook. """ from __future__ import annotations diff --git a/src/alphamind/command_center/__main__.py b/src/alphamind/command_center/__main__.py index deb176a17..78b1c9497 100644 --- a/src/alphamind/command_center/__main__.py +++ b/src/alphamind/command_center/__main__.py @@ -158,7 +158,7 @@ async def _maybe_emit_setup_token( relying_party_id: "{suggested}" ... Then restart. You will need to re-enroll passkeys (rpId change). - See RUNBOOK_command_center.md § LAN access for the full recipe. + See docs/runbooks/command-center.md § LAN access for the full recipe. """ ).strip() logger.info(block) @@ -203,7 +203,7 @@ def _warn_if_mixed_lan_bind_and_access(config: CommandCenterConfig) -> None: "command_center: bind.host=%s is not loopback but access.host=%s " "still looks localhost-ish. Edit the access: block (scheme + host + port) " "in command-center.yaml and align webauthn.relying_party_id in security.yaml; " - "see RUNBOOK_command_center.md for LAN setup.", + "see docs/runbooks/command-center.md for LAN setup.", bind_host, access.host, ) @@ -216,7 +216,7 @@ def _warn_if_mixed_lan_bind_and_access(config: CommandCenterConfig) -> None: "command_center: bind.port=%s but no explicit access: block was supplied. " "The default origin will be http://localhost:8080 (not port %s). " "Supply an explicit access: block with a matching port to keep WebAuthn " - "and cookies correct. See RUNBOOK_command_center.md § LAN access.", + "and cookies correct. See docs/runbooks/command-center.md § LAN access.", config.bind.port, config.bind.port, ) diff --git a/src/alphamind/command_center/config.py b/src/alphamind/command_center/config.py index e9eb5a841..7862653b2 100644 --- a/src/alphamind/command_center/config.py +++ b/src/alphamind/command_center/config.py @@ -111,7 +111,7 @@ class BindConfig(BaseModel): For LAN access from other machines on the same network, the bind can be widened (0.0.0.0 or the host LAN IP) independently of the public origin thanks to the ``access:`` block (see :class:`AccessConfig`). See the - RUNBOOK_command_center.md § "LAN access (local network)" for the + docs/runbooks/command-center.md § "LAN access (local network)" for the supported recipe (hostname choice via mDNS/hosts-file, exact YAML edits, restart, re-enrollment on rpId change, firewall note). The old assumption that editing bind.host alone suffices for LAN is superseded by the @@ -155,7 +155,7 @@ class AccessConfig(BaseModel): exact v1 shipped configuration. For LAN usage, supply an explicit ``access:`` block (see - RUNBOOK_command_center.md § "LAN access (local network)" for the exact + docs/runbooks/command-center.md § "LAN access (local network)" for the exact recipe, hostname choice, re-enrollment steps, and firewall guidance). Widened binds are supported when paired with the access: origin (the mixed case emits a startup warning directing the operator to the RUNBOOK). diff --git a/src/alphamind/command_center/frontend/src/views/configuration/security-page.tsx b/src/alphamind/command_center/frontend/src/views/configuration/security-page.tsx index b6d0e4c36..7e4779aaf 100644 --- a/src/alphamind/command_center/frontend/src/views/configuration/security-page.tsx +++ b/src/alphamind/command_center/frontend/src/views/configuration/security-page.tsx @@ -3,7 +3,7 @@ // Thin scaffold wrapper — security.yaml carries no hot-reload-engine // machinery beyond what the framework's ReloadPolicyBadge already // surfaces (webauthn.relying_party_id is DEPLOY_TIME; for LAN access it must -// match the `access:` host per ALP-724 / RUNBOOK_command_center.md). +// match the `access:` host per ALP-724 / docs/runbooks/command-center.md). import { ConfigEditorPage } from '@/views/configuration/config-editor-page' diff --git a/src/alphamind/command_center/frontend/vite.config.ts b/src/alphamind/command_center/frontend/vite.config.ts index 39ab86c17..4db37a8e6 100644 --- a/src/alphamind/command_center/frontend/vite.config.ts +++ b/src/alphamind/command_center/frontend/vite.config.ts @@ -41,7 +41,7 @@ const API_TARGET = 'http://127.0.0.1:8080' // // The committed defaults remain intentionally loopback-only for safe local // development. Temporary LAN-dev edits to host should never be committed. -// See the approved LAN plan (§5, §8 story 03c) and RUNBOOK_command_center.md +// See the approved LAN plan (§5, §8 story 03c) and docs/runbooks/command-center.md // for production LAN wiring details. // https://vite.dev/config/ diff --git a/src/alphamind/distillation/calibration_snapshot.py b/src/alphamind/distillation/calibration_snapshot.py index 5c0f7a950..2746b4f08 100644 --- a/src/alphamind/distillation/calibration_snapshot.py +++ b/src/alphamind/distillation/calibration_snapshot.py @@ -209,7 +209,7 @@ def write_calibration_state_snapshot( Bumped to ``"2"`` for ALP-709 to admit the ``inflow_metrics`` top-level key (currently carrying ``sentiment`` inter-baseline news-window telemetry; future surfaces — news-price divergence, prediction-market -inflow — extend the same key). The renderer in ``scripts/verify_debug_e2e.py`` +inflow — extend the same key). The renderer in ``scripts/verify/verify_debug_e2e.py`` must bump :data:`_OPERATOR_SNAPSHOT_SCHEMA_VERSION` in lockstep so a V1 payload at the operator path does not silently render as zeros. """ diff --git a/src/alphamind/distillation/orchestrator.py b/src/alphamind/distillation/orchestrator.py index eb7a708b4..7fdfc6599 100644 --- a/src/alphamind/distillation/orchestrator.py +++ b/src/alphamind/distillation/orchestrator.py @@ -335,7 +335,7 @@ def _refresh_class_b_state( # All six categories (q1, q3, q6, q7, q12, qualitative) wire up directly # to their module-level entry points. The :data:`_CATEGORY_COMPUTE_PLACEHOLDER_GAPS` # table is preserved as an empty tuple for the verification script in -# story 13 (`scripts/verify_distillation.py`) so its summary section's +# story 13 (`scripts/verify/verify_distillation.py`) so its summary section's # shape stays stable; it now reports "all categories integrated." _CATEGORY_COMPUTE_PLACEHOLDER_GAPS: tuple[tuple[str, str], ...] = () diff --git a/src/alphamind/scheduler/__main__.py b/src/alphamind/scheduler/__main__.py index fa77f638d..ac2d4d12c 100644 --- a/src/alphamind/scheduler/__main__.py +++ b/src/alphamind/scheduler/__main__.py @@ -558,7 +558,7 @@ async def _run_control_surface(session: PipelineSession, *, app: FastAPI, port: def main(argv: Sequence[str] | None = None) -> None: # Switch stdout/stderr to UTF-8 with ``replace`` errors so a Windows # ``cp1252`` default doesn't mangle the JSON ``InvocationSummary`` - # captured by ``scripts/verify_debug_e2e.py`` (or surface as a + # captured by ``scripts/verify/verify_debug_e2e.py`` (or surface as a # ``UnicodeEncodeError`` inside the subprocess on non-ASCII output). configure_utf8_stdio() args = _parse_args(argv) diff --git a/src/alphamind/scheduler/control/adapters.py b/src/alphamind/scheduler/control/adapters.py index 1e92e1dae..b4990e6a2 100644 --- a/src/alphamind/scheduler/control/adapters.py +++ b/src/alphamind/scheduler/control/adapters.py @@ -309,7 +309,7 @@ class DeferredUniverseValidator: The verb's ``UniverseValidationFailedError`` maps to HTTP 500 ``internal_error``; this placeholder fires it with a descriptive cause so the command-center surfaces a useful operator message. - Follow-up: wire ``scripts/validate_universe.py`` through this adapter. + Follow-up: wire ``scripts/ops/validate_universe.py`` through this adapter. """ def validate(self, *, as_of: date) -> UniverseValidationReportRecord: @@ -317,7 +317,7 @@ def validate(self, *, as_of: date) -> UniverseValidationReportRecord: raise UniverseValidationFailedError( cause=RuntimeError( "Universe validation via /control/run_universe_validation is " - "not yet wired in production. Run scripts/validate_universe.py " + "not yet wired in production. Run scripts/ops/validate_universe.py " "directly or file a follow-up to ALP-720." ) ) diff --git a/src/alphamind/scheduler/control/verbs.py b/src/alphamind/scheduler/control/verbs.py index b829460b2..312ace9e6 100644 --- a/src/alphamind/scheduler/control/verbs.py +++ b/src/alphamind/scheduler/control/verbs.py @@ -157,7 +157,7 @@ class UniverseValidationReportRecord: class UniverseValidator(Protocol): - """Adapter over ``scripts/validate_universe.py``'s entrypoint. + """Adapter over ``scripts/ops/validate_universe.py``'s entrypoint. The concrete adapter calls the script's per-ticker validator and shapes results into :class:`UniverseValidationReportRecord`. Tests @@ -364,7 +364,7 @@ def run_universe_validation( ) -> RunUniverseValidationResult: """``POST /control/run_universe_validation`` — synchronous validation run. - The validator is the injected adapter over ``scripts/validate_universe.py``. + The validator is the injected adapter over ``scripts/ops/validate_universe.py``. Any exception from the validator is wrapped in :class:`UniverseValidationFailedError` so the route layer can map to the ``internal_error`` (HTTP 500) envelope without leaking the diff --git a/src/alphamind/scheduler/debug_e2e/seed.py b/src/alphamind/scheduler/debug_e2e/seed.py index 82ca5de1a..e577b8566 100644 --- a/src/alphamind/scheduler/debug_e2e/seed.py +++ b/src/alphamind/scheduler/debug_e2e/seed.py @@ -84,7 +84,7 @@ # State-persistence tables wiped at the start of every debug-e2e # invocation, in FK-safe order — children before parents. # -# The snapshot-from-prod workflow (``scripts/snapshot_prod_for_debug_e2e.py``) +# The snapshot-from-prod workflow (``scripts/verify/snapshot_prod_for_debug_e2e.py``) # means the bootstrap DB carries real production rows for every table below, # so the wipe list MUST cover all state-persistence tables — not just the # narrower set the original design enumerated when the seeder ran against a diff --git a/src/alphamind/scripts/__init__.py b/src/alphamind/scripts/__init__.py index f4423d071..8c9e7dea4 100644 --- a/src/alphamind/scripts/__init__.py +++ b/src/alphamind/scripts/__init__.py @@ -1,8 +1,8 @@ """Operator verification scripts (story 13). The modules under ``alphamind.scripts`` carry the testable logic of the -``scripts/verify_*.py`` shims at the repository root. Each shim is a +``scripts/verify/verify_*.py`` shims. Each shim is a thin wrapper around its corresponding module's ``main()`` entry point so -the operator runs the script via ``uv run python scripts/verify_.py`` +the operator runs the script via ``uv run python scripts/verify/verify_.py`` while the test suite imports the module directly. """ diff --git a/src/alphamind/scripts/_artifact_io.py b/src/alphamind/scripts/_artifact_io.py index 88e32bf07..76cbadd3c 100644 --- a/src/alphamind/scripts/_artifact_io.py +++ b/src/alphamind/scripts/_artifact_io.py @@ -116,14 +116,14 @@ class MissingArtifactError(FileNotFoundError): """Raised when a required stage artifact is absent. Carries the producer-script name so the consumer can render - "run scripts/verify_.py first" without the operator having to + "run scripts/verify/verify_.py first" without the operator having to map filenames to their producers. """ def __init__(self, *, missing_path: Path, producer_script: str) -> None: message = ( f"required stage artifact {missing_path} is missing — " - f"run scripts/{producer_script} first to produce it" + f"run scripts/verify/{producer_script} first to produce it" ) super().__init__(message) self.missing_path = missing_path diff --git a/src/alphamind/scripts/report_emergency_invocations.py b/src/alphamind/scripts/report_emergency_invocations.py index 468b11698..408ae0f24 100644 --- a/src/alphamind/scripts/report_emergency_invocations.py +++ b/src/alphamind/scripts/report_emergency_invocations.py @@ -12,7 +12,7 @@ for script-internal errors. Usage: - uv run python scripts/report_emergency_invocations.py \\ + uv run python scripts/ops/report_emergency_invocations.py \\ [--db-path PATH] [--window-days N] [--output text|json] """ diff --git a/src/alphamind/scripts/report_flag_rates.py b/src/alphamind/scripts/report_flag_rates.py index ff683e02e..ff276b394 100644 --- a/src/alphamind/scripts/report_flag_rates.py +++ b/src/alphamind/scripts/report_flag_rates.py @@ -16,7 +16,7 @@ script-internal errors (DB unreachable, corrupt fixture). Usage: - uv run python scripts/report_flag_rates.py \\ + uv run python scripts/ops/report_flag_rates.py \\ [--window-days N] [--db-path PATH] [--threshold-class CLASS] \\ [--ticker T] [--output text|json] """ diff --git a/src/alphamind/scripts/verify_counterfactual_replay_engine.py b/src/alphamind/scripts/verify_counterfactual_replay_engine.py index d3a372e92..90f665894 100644 --- a/src/alphamind/scripts/verify_counterfactual_replay_engine.py +++ b/src/alphamind/scripts/verify_counterfactual_replay_engine.py @@ -1,6 +1,6 @@ """Standalone end-to-end verify for the counterfactual replay engine (ALP-566). -The engine runs **out-of-pipeline** (CLI only), so ``scripts/verify_debug_e2e.py`` +The engine runs **out-of-pipeline** (CLI only), so ``scripts/verify/verify_debug_e2e.py`` does not cover it. This script is the deterministic operator check after a code change or a migration that touches ``counterfactual_replays``: it seeds a controlled DB state of PM-decision rows + bar history + IV snapshots + (for the @@ -27,7 +27,7 @@ Usage:: - uv run python scripts/verify_counterfactual_replay_engine.py [--db-path PATH] + uv run python scripts/verify/verify_counterfactual_replay_engine.py [--db-path PATH] Exit codes: ``0`` on full PASS, ``1`` on any FAIL. """ diff --git a/src/alphamind/scripts/verify_feedback_loop.py b/src/alphamind/scripts/verify_feedback_loop.py index 68cf80d06..0734e5d03 100644 --- a/src/alphamind/scripts/verify_feedback_loop.py +++ b/src/alphamind/scripts/verify_feedback_loop.py @@ -1,7 +1,7 @@ """Standalone end-to-end verify for the feedback-loop spine (ALP-896 / story 10). The feedback loop runs **out-of-pipeline** (operator CLIs + a per-invocation -resolver step), so ``scripts/verify_debug_e2e.py`` does not cover it. This script +resolver step), so ``scripts/verify/verify_debug_e2e.py`` does not cover it. This script is the deterministic operator check after a code change or a migration that touches the feedback-loop tables: it runs ``alembic upgrade head`` on a fresh scratch DB, seeds one controlled scenario (a CLOSED position + its linked ACTIVE @@ -43,7 +43,7 @@ Usage:: - uv run python scripts/verify_feedback_loop.py [--db-path PATH] + uv run python scripts/verify/verify_feedback_loop.py [--db-path PATH] Exit codes: ``0`` on full PASS, ``1`` on any FAIL. """ @@ -170,7 +170,7 @@ # --------------------------------------------------------------------------- # Seed clock + controlled identifiers # --------------------------------------------------------------------------- -#: The repo root (``…/scripts/verify_feedback_loop.py`` → repo root): the alembic.ini +#: The repo root (``…/scripts/verify/verify_feedback_loop.py`` → repo root): the alembic.ini #: + config dir anchor. Resolved once at import (a sync context) so the I/O-driven #: async runner never stats the filesystem itself. _REPO_ROOT = Path(__file__).resolve().parents[3] diff --git a/src/alphamind/scripts/verify_genesis.py b/src/alphamind/scripts/verify_genesis.py index ddd2351ee..4a26b633a 100644 --- a/src/alphamind/scripts/verify_genesis.py +++ b/src/alphamind/scripts/verify_genesis.py @@ -48,7 +48,7 @@ fill consumer), and the real state codecs / ``command_ids`` derivation — so the assertions exercise the post-wave behaviors rather than re-implementations. -Operator usage is documented in ``scripts/RUNBOOK_genesis_verify.md``. +Operator usage is documented in ``scripts/verify/RUNBOOK_genesis_verify.md``. Exit codes: - 0 — every genesis assertion holds diff --git a/tests/analysis/adaptive_research/test_fixtures.py b/tests/analysis/adaptive_research/test_fixtures.py index 677cfe338..63a91b2de 100644 --- a/tests/analysis/adaptive_research/test_fixtures.py +++ b/tests/analysis/adaptive_research/test_fixtures.py @@ -11,7 +11,7 @@ production. Originally extracted from a live-SDK e2e test (ALP-265); the live path is -covered by ``scripts/verify_debug_e2e.py`` end-to-end, but the fixture +covered by ``scripts/verify/verify_debug_e2e.py`` end-to-end, but the fixture invariants are worth keeping as a fast, no-SDK canary. """ diff --git a/tests/command_center/test_main.py b/tests/command_center/test_main.py index 75241a893..af127a27e 100644 --- a/tests/command_center/test_main.py +++ b/tests/command_center/test_main.py @@ -132,7 +132,7 @@ async def test_emits_lan_hostname_suggestion_on_nonlocal_bind_zero_creds( assert "Edit config/security.yaml:" in caplog.text assert 'relying_party_id: "' in caplog.text assert "socket.gethostname() ==" in caplog.text # derives realistic suggestion - assert "RUNBOOK_command_center.md § LAN access for the full recipe." in caplog.text + assert "docs/runbooks/command-center.md § LAN access for the full recipe." in caplog.text async def test_no_lan_suggestion_on_localhost_bind( self, cc_factory: async_sessionmaker[AsyncSession], caplog: pytest.LogCaptureFixture diff --git a/tests/docs/test_claude_md_pointers.py b/tests/docs/test_claude_md_pointers.py index ae9aa6eae..ba2ab4eac 100644 --- a/tests/docs/test_claude_md_pointers.py +++ b/tests/docs/test_claude_md_pointers.py @@ -1,10 +1,14 @@ -"""Dead-pointer guard for CLAUDE.md navigation files. +"""Dead-pointer guard for navigation docs: CLAUDE.md files, runbooks, operate-prod. -Every path referenced in a tracked ``CLAUDE.md`` must resolve to a real file or +Every path referenced in a scanned navigation doc must resolve to a real file or directory. Stale pointers are the worst documentation failure mode — they send agents on wild-goose chases — so this turns a dead reference into a red CI check. -What is checked, per ``CLAUDE.md``: +Scanned docs: every tracked ``CLAUDE.md``, every ``docs/runbooks/**/*.md`` (the prod +operational runbooks form a pointer web with the scripts they invoke), and +``.claude/skills/operate-prod/SKILL.md`` (a pure router over the runbooks). + +What is checked, per doc: * Markdown link targets ``[text](path)`` (skipping URLs and ``#anchors``). * Inline ``code`` spans **outside** fenced ``` code blocks that look like repo paths. @@ -12,7 +16,7 @@ (``tests//``, ``test_x.py::test_case``) that are not real paths. Navigation pointers that we want guaranteed live in prose / tables as inline code or links. -A candidate resolves if it exists relative to any of: the repo root, the ``CLAUDE.md``'s +A candidate resolves if it exists relative to any of: the repo root, the doc's own directory, or ``src/alphamind/`` (so ``execution/`` shorthand for a package resolves). """ @@ -24,8 +28,20 @@ REPO_ROOT = Path(__file__).resolve().parents[2] SRC_ROOT = REPO_ROOT / "src" / "alphamind" -# Directories whose CLAUDE.md files are copies / vendored and must not be scanned. -_EXCLUDED_PARTS = {".claude", "node_modules", ".git"} +# Vendored / copy directories that must not be scanned. Judged on the path RELATIVE to +# REPO_ROOT — an absolute-parts check would match the ``.claude`` in +# ``.claude/worktrees//...`` and silently exclude *every* doc when the suite runs +# inside a worktree checkout. ``.claude`` itself is only excluded for worktree copies +# (``.claude/worktrees/``); tracked skill files under ``.claude/skills/`` are scannable. +_VENDORED_PARTS = {"node_modules", ".git"} + + +def _excluded(f: Path) -> bool: + rel_parts = f.relative_to(REPO_ROOT).parts + if _VENDORED_PARTS & set(rel_parts): + return True + return ".claude" in rel_parts and "worktrees" in rel_parts + _FENCED_BLOCK = re.compile(r"```.*?```", re.DOTALL) _MD_LINK = re.compile(r"\[[^\]]*\]\(([^)]+)\)") @@ -52,11 +68,19 @@ ) -def _claude_md_files() -> list[Path]: +def _scanned_docs() -> list[Path]: files = [REPO_ROOT / "CLAUDE.md"] files += sorted(SRC_ROOT.rglob("CLAUDE.md")) files += sorted((REPO_ROOT / "docs").rglob("CLAUDE.md")) - return [f for f in files if f.exists() and not (set(f.parts) & _EXCLUDED_PARTS)] + files += sorted((REPO_ROOT / "docs" / "runbooks").rglob("*.md")) + # operate-prod is the one router-class skill: every path in it is a navigation + # pointer, so the pathish heuristic applies cleanly. Other skills' prose carries + # illustrative example paths and runtime-artifact paths (audited 2026-06-10: + # 12 false positives vs 1 real catch across the other 13 skills), so they stay + # out of the scan set; add a skill here only if it is likewise pure navigation. + files.append(REPO_ROOT / ".claude" / "skills" / "operate-prod" / "SKILL.md") + deduped = list(dict.fromkeys(files)) + return [f for f in deduped if f.exists() and not _excluded(f)] def _is_pathish(token: str) -> bool: @@ -120,12 +144,52 @@ def test_claude_md_files_exist() -> None: assert (REPO_ROOT / "CLAUDE.md").exists() -def test_no_dead_pointers_in_claude_md() -> None: +def test_runbook_web_is_in_scan_set() -> None: + """The runbook map and the operate-prod router must exist and be scanned. + + These two docs anchor the prod-operations pointer web (the runbook split): + ``docs/runbooks/README.md`` maps the runbooks, and the ``operate-prod`` skill + routes situations onto them. If either goes missing — or the collector stops + picking them up — the guard would silently shrink its coverage. + """ + scanned = set(_scanned_docs()) + assert REPO_ROOT / "docs" / "runbooks" / "README.md" in scanned + assert REPO_ROOT / ".claude" / "skills" / "operate-prod" / "SKILL.md" in scanned + + +def test_no_dead_pointers_in_nav_docs() -> None: dead: list[str] = [] - for doc in _claude_md_files(): + for doc in _scanned_docs(): rel_doc = doc.relative_to(REPO_ROOT) for ref in sorted(_candidates(doc.read_text(encoding="utf-8"))): if not _resolves(doc, ref): dead.append(f"{rel_doc} -> {ref}") - assert not dead, "Dead pointers in CLAUDE.md files:\n" + "\n".join(dead) + assert not dead, "Dead pointers in navigation docs:\n" + "\n".join(dead) + + +_SECTION_REF = re.compile(r"([a-z][a-z0-9-]*\.md) § (\d+(?:\.\d+)?)") + + +def test_runbook_section_refs_resolve() -> None: + """Cross-doc ``.md § N[.M]`` references must name a real heading. + + The runbook split keeps the monolith's section numbering precisely so these + prose references stay stable. Each one is validated against the target doc's + headings (``## N.`` top level, ``### N.M`` subsection) — a renumbering or + deletion in one runbook cannot silently strand the docs pointing at it. + Fenced blocks are included on purpose: operators copy-paste from them. + """ + runbooks = REPO_ROOT / "docs" / "runbooks" + dead: list[str] = [] + for doc in sorted(runbooks.glob("*.md")): + for name, sec in _SECTION_REF.findall(doc.read_text(encoding="utf-8")): + target = runbooks / name + if not target.exists(): + dead.append(f"{doc.name} -> {name} § {sec} (no such runbook)") + continue + heading = rf"^### {re.escape(sec)} " if "." in sec else rf"^## {sec}\. " + if not re.search(heading, target.read_text(encoding="utf-8"), re.MULTILINE): + dead.append(f"{doc.name} -> {name} § {sec} (no such heading)") + + assert not dead, "Dangling section references in runbooks:\n" + "\n".join(dead) diff --git a/tests/fixtures/decision/analyst/README.md b/tests/fixtures/decision/analyst/README.md index 05b6d6182..a1f1d43de 100644 --- a/tests/fixtures/decision/analyst/README.md +++ b/tests/fixtures/decision/analyst/README.md @@ -1,7 +1,7 @@ # Analyst output fixtures Canonical `AnalystOutput` JSON fixtures emitted by -`scripts/verify_analyst.py --save-fixtures` against a recorded +`scripts/verify/verify_analyst.py --save-fixtures` against a recorded synthesizer archive. Two scenarios: - `normal.json` — parsed analyst output from the normal-mode scenario. @@ -18,7 +18,7 @@ Both files validate cleanly against Fixtures are produced by: ```bash -uv run python scripts/verify_analyst.py \ +uv run python scripts/verify/verify_analyst.py \ --archive-root \ --synthesizer-invocation-id \ --save-fixtures @@ -36,9 +36,9 @@ in the same archive. These fixtures are the canonical analyst-output input for the downstream feature trees' verifiers: -- `scripts/verify_strategist.py` (future feature work tree). -- `scripts/verify_proposal_pre_processor.py` (future feature work tree). -- `scripts/verify_portfolio_manager.py` (future feature work tree). +- `scripts/verify/verify_strategist.py` (future feature work tree). +- `scripts/verify/verify_proposal_pre_processor.py` (future feature work tree). +- `scripts/verify/verify_portfolio_manager.py` (future feature work tree). Each downstream verifier reads `normal.json` and `halt.json` directly rather than chaining live analyst calls (per ALP-115 parent-issue @@ -64,9 +64,9 @@ Re-run the verifier whenever the analyst's contract changes: Refresh procedure: -1. Run `scripts/verify_synthesizer.py --archive-root --invocation-id ` +1. Run `scripts/verify/verify_synthesizer.py --archive-root --invocation-id ` to produce a fresh synthesizer archive. -2. Run `scripts/verify_analyst.py --archive-root +2. Run `scripts/verify/verify_analyst.py --archive-root --synthesizer-invocation-id --save-fixtures` to overwrite the two fixture files. 3. Confirm both scenarios reported `PASS` (or, if `WARN`, that the @@ -75,4 +75,4 @@ Refresh procedure: 4. Run `uv run pytest -n auto` to confirm downstream consumers still parse the refreshed fixtures cleanly. -See `scripts/RUNBOOK_analyst.md` for the full operator runbook. +See `scripts/verify/RUNBOOK_analyst.md` for the full operator runbook. diff --git a/tests/fixtures/decision/proposal_pre_processor/README.md b/tests/fixtures/decision/proposal_pre_processor/README.md index a9c57dd6c..e80ca0d96 100644 --- a/tests/fixtures/decision/proposal_pre_processor/README.md +++ b/tests/fixtures/decision/proposal_pre_processor/README.md @@ -1,7 +1,7 @@ # Proposal pre-processor output fixtures Canonical `ProposalPreProcessorBundle` JSON fixtures emitted by -`scripts/verify_proposal_pre_processor.py`. Four scenarios: +`scripts/verify/verify_proposal_pre_processor.py`. Four scenarios: - `normal.json` — normal analyst mode (empty recommendations, all-hold strategist assessments) against a low-utilisation portfolio snapshot. No @@ -28,18 +28,18 @@ All four files validate cleanly against Fixtures are produced by: ```bash -uv run python scripts/verify_proposal_pre_processor.py +uv run python scripts/verify/verify_proposal_pre_processor.py ``` Or for a single scenario: ```bash -uv run python scripts/verify_proposal_pre_processor.py --scenario normal +uv run python scripts/verify/verify_proposal_pre_processor.py --scenario normal ``` The analyst input fixtures (`tests/fixtures/decision/analyst/normal.json` and `halt.json`) must exist before running — produce them with -`scripts/verify_analyst.py --save-fixtures` if missing. Strategist outputs are +`scripts/verify/verify_analyst.py --save-fixtures` if missing. Strategist outputs are constructed in-code by `alphamind.scripts.verify_proposal_pre_processor.build_fixture_{normal,halt,emergency}_strategist_output`. @@ -48,7 +48,7 @@ constructed in-code by These fixtures are the canonical proposal pre-processor output for the downstream feature trees' verifiers: -- `scripts/verify_portfolio_manager.py` (PM work tree, ALP-117) — the PM +- `scripts/verify/verify_portfolio_manager.py` (PM work tree, ALP-117) — the PM agent's verify script will read the bundle JSONs here as its primary inputs. The files are checked into version control to provide a stable contract surface @@ -82,10 +82,10 @@ Re-run the verifier whenever the pre-processor's contract changes: 1. Confirm analyst fixtures exist: `ls tests/fixtures/decision/analyst/normal.json tests/fixtures/decision/analyst/halt.json` -2. Run `uv run python scripts/verify_proposal_pre_processor.py` to overwrite +2. Run `uv run python scripts/verify/verify_proposal_pre_processor.py` to overwrite all four fixture files. 3. Confirm all four scenarios reported PASS. 4. Run `uv run pytest -n auto` to confirm downstream consumers still parse the refreshed fixtures cleanly. -See `scripts/RUNBOOK_proposal_pre_processor.md` for the full operator runbook. +See `scripts/verify/RUNBOOK_proposal_pre_processor.md` for the full operator runbook. diff --git a/tests/fixtures/decision/strategist/README.md b/tests/fixtures/decision/strategist/README.md index 72673303a..2af7be2eb 100644 --- a/tests/fixtures/decision/strategist/README.md +++ b/tests/fixtures/decision/strategist/README.md @@ -1,7 +1,7 @@ # Strategist output fixtures Canonical `StrategistOutput` JSON fixtures emitted by -`scripts/verify_strategist.py --save-fixtures` against a recorded +`scripts/verify/verify_strategist.py --save-fixtures` against a recorded synthesizer archive. Three scenarios: - `normal.json` — parsed strategist output from the normal-mode @@ -32,7 +32,7 @@ None of the three fixture JSONs have been emitted yet. The 2026-05-05 verificati Fixtures are produced by: ```bash -uv run python scripts/verify_strategist.py \ +uv run python scripts/verify/verify_strategist.py \ --archive-root \ --synthesizer-invocation-id \ --save-fixtures @@ -56,8 +56,8 @@ emergency-invocation paths. These fixtures are the canonical strategist-output input for the downstream feature trees' verifiers: -- `scripts/verify_proposal_pre_processor.py` (future feature work tree). -- `scripts/verify_portfolio_manager.py` (future feature work tree). +- `scripts/verify/verify_proposal_pre_processor.py` (future feature work tree). +- `scripts/verify/verify_portfolio_manager.py` (future feature work tree). Each downstream verifier reads the appropriate scenario JSON directly rather than chaining live strategist calls (per ALP-116 parent-issue @@ -86,10 +86,10 @@ Re-run the verifier whenever the strategist's contract changes: Refresh procedure: -1. Run `scripts/verify_synthesizer.py --archive-root +1. Run `scripts/verify/verify_synthesizer.py --archive-root --invocation-id ` to produce a fresh synthesizer archive (or reuse an existing one if the synthesizer's contract is unchanged). -2. Run `scripts/verify_strategist.py --archive-root +2. Run `scripts/verify/verify_strategist.py --archive-root --synthesizer-invocation-id --save-fixtures` to overwrite the three fixture files. 3. Confirm all three scenarios reported `PASS` (or, if `WARN`, that @@ -98,4 +98,4 @@ Refresh procedure: 4. Run `uv run pytest -n auto` to confirm downstream consumers still parse the refreshed fixtures cleanly. -See `scripts/RUNBOOK_strategist.md` for the full operator runbook. +See `scripts/verify/RUNBOOK_strategist.md` for the full operator runbook. diff --git a/tests/scripts/test_build_e2e_report.py b/tests/scripts/test_build_e2e_report.py index d97981b49..82bfdf7a5 100644 --- a/tests/scripts/test_build_e2e_report.py +++ b/tests/scripts/test_build_e2e_report.py @@ -1,4 +1,4 @@ -"""Tests for ``scripts/build_e2e_report.py`` (story 05 / ALP-502). +"""Tests for ``scripts/verify/build_e2e_report.py`` (story 05 / ALP-502). The HTML report builder was rewritten in ALP-502 to consume the debug-e2e single-invocation archive (``progress.jsonl`` + @@ -26,12 +26,12 @@ from alphamind._kernel.archive_layout import invocation_archive_dir -_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "build_e2e_report.py" +_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "verify" / "build_e2e_report.py" @pytest.fixture(scope="module") def report_module() -> ModuleType: - """Load ``scripts/build_e2e_report.py`` as an importable module.""" + """Load ``scripts/verify/build_e2e_report.py`` as an importable module.""" spec = importlib.util.spec_from_file_location("build_e2e_report", _SCRIPT_PATH) assert spec is not None assert spec.loader is not None diff --git a/tests/scripts/test_reconcile_alpaca_state.py b/tests/scripts/test_reconcile_alpaca_state.py index 21e01d659..7b0733d0a 100644 --- a/tests/scripts/test_reconcile_alpaca_state.py +++ b/tests/scripts/test_reconcile_alpaca_state.py @@ -1,4 +1,4 @@ -"""Tests for ``scripts/reconcile_alpaca_state.py`` (ALP-818). +"""Tests for ``scripts/ops/reconcile_alpaca_state.py`` (ALP-818). The operator helper compares the live Alpaca paper account against local SQLite portfolio state and prints a ``<<< DIVERGENCE`` flag per symbol. The @@ -22,7 +22,7 @@ import pytest -_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "reconcile_alpaca_state.py" +_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "ops" / "reconcile_alpaca_state.py" def _load_script() -> Any: diff --git a/tests/scripts/test_report_emergency_invocations.py b/tests/scripts/test_report_emergency_invocations.py index f0805c8cc..e8be5e0c2 100644 --- a/tests/scripts/test_report_emergency_invocations.py +++ b/tests/scripts/test_report_emergency_invocations.py @@ -1,4 +1,4 @@ -"""Tests for ``scripts/report_emergency_invocations.py`` (story 15 / ALP-99). +"""Tests for ``scripts/ops/report_emergency_invocations.py`` (story 15 / ALP-99). The report joins ``invocations.trigger_type='emergency'`` to ``activity_log`` to classify each emergency invocation by downstream consequence and surface diff --git a/tests/scripts/test_report_flag_rates.py b/tests/scripts/test_report_flag_rates.py index 9d4bdecd4..00b70d140 100644 --- a/tests/scripts/test_report_flag_rates.py +++ b/tests/scripts/test_report_flag_rates.py @@ -1,4 +1,4 @@ -"""Tests for ``scripts/report_flag_rates.py`` (story 16 / ALP-96). +"""Tests for ``scripts/ops/report_flag_rates.py`` (story 16 / ALP-96). The report reads every ``DISTILLATION_ANOMALY_FLAG`` activity-log entry over a trailing window, groups by ``(threshold_class, threshold_key)``, and reports the diff --git a/tests/scripts/test_verify_bootstrap_calibration_mix.py b/tests/scripts/test_verify_bootstrap_calibration_mix.py index 174a6fad6..99bc16992 100644 --- a/tests/scripts/test_verify_bootstrap_calibration_mix.py +++ b/tests/scripts/test_verify_bootstrap_calibration_mix.py @@ -1,4 +1,4 @@ -"""Unit tests for ``scripts/verify_bootstrap_calibration_mix.py`` (story 13). +"""Unit tests for ``scripts/verify/verify_bootstrap_calibration_mix.py`` (story 13). The calibration-mix verifier reads ``distillation_ticker_baseline`` and asserts the calibration-state distribution per ``baseline_kind`` matches diff --git a/tests/scripts/test_verify_command_center.py b/tests/scripts/test_verify_command_center.py index c022eccf2..76a5fbb90 100644 --- a/tests/scripts/test_verify_command_center.py +++ b/tests/scripts/test_verify_command_center.py @@ -1,4 +1,4 @@ -"""Shape tests for ``scripts/verify_command_center.py`` (story 07 / ALP-685). +"""Shape tests for ``scripts/verify/verify_command_center.py`` (story 07 / ALP-685). The script is the operator-runnable end-to-end gate for the command center; its seven check helpers (``check_daemons_bind``, @@ -28,12 +28,14 @@ import pytest -_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "verify_command_center.py" +_SCRIPT_PATH = ( + Path(__file__).resolve().parents[2] / "scripts" / "verify" / "verify_command_center.py" +) @pytest.fixture(scope="module") def verify_module() -> ModuleType: - """Load ``scripts/verify_command_center.py`` as an importable module. + """Load ``scripts/verify/verify_command_center.py`` as an importable module. The script lives under ``scripts/`` rather than ``src/`` per the "no shim split" convention shared by ``verify_debug_e2e.py``; we diff --git a/tests/scripts/test_verify_counterfactual_replay_engine.py b/tests/scripts/test_verify_counterfactual_replay_engine.py index 56d3dd648..f2c31f2d7 100644 --- a/tests/scripts/test_verify_counterfactual_replay_engine.py +++ b/tests/scripts/test_verify_counterfactual_replay_engine.py @@ -2,7 +2,7 @@ The script's *seed builder* (``build_test_seed``) and its pure *assertion helpers* are the unit-testable surface; the full end-to-end run is exercised by -invoking ``scripts/verify_counterfactual_replay_engine.py`` manually (it is the +invoking ``scripts/verify/verify_counterfactual_replay_engine.py`` manually (it is the verify-of-the-verify, so embedding the whole standalone run in pytest would just duplicate the script). These tests pin: diff --git a/tests/scripts/test_verify_debug_e2e.py b/tests/scripts/test_verify_debug_e2e.py index 5b0eb52b3..22c663e04 100644 --- a/tests/scripts/test_verify_debug_e2e.py +++ b/tests/scripts/test_verify_debug_e2e.py @@ -1,4 +1,4 @@ -"""Tests for ``scripts/verify_debug_e2e.py`` (story 05 / ALP-502). +"""Tests for ``scripts/verify/verify_debug_e2e.py`` (story 05 / ALP-502). The script is the operator-runnable end-to-end gate for the ``--debug-e2e`` mode; its six check helpers (``check_auth``, @@ -35,13 +35,13 @@ from alphamind._kernel.archive_layout import invocation_archive_dir from alphamind.persistence.models import Base -_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "verify_debug_e2e.py" +_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "verify" / "verify_debug_e2e.py" _INVOCATION_AS_OF = datetime(2026, 5, 26, tzinfo=UTC) @pytest.fixture(scope="module") def verify_module() -> ModuleType: - """Load ``scripts/verify_debug_e2e.py`` as an importable module. + """Load ``scripts/verify/verify_debug_e2e.py`` as an importable module. The script lives under ``scripts/`` rather than ``src/`` per the "no shim split" decision in ALP-502; we use ``spec_from_file_location`` diff --git a/tests/scripts/test_verify_debug_e2e_resume.py b/tests/scripts/test_verify_debug_e2e_resume.py index f52de51d8..447e4b6e8 100644 --- a/tests/scripts/test_verify_debug_e2e_resume.py +++ b/tests/scripts/test_verify_debug_e2e_resume.py @@ -1,5 +1,5 @@ """Tests for the ``--resume-from`` plumbing + ``check_deterministic_prefix`` -in ``scripts/verify_debug_e2e.py`` (story ALP-696). +in ``scripts/verify/verify_debug_e2e.py`` (story ALP-696). Exercises the four user-visible surfaces this story adds: @@ -34,13 +34,13 @@ from alphamind._kernel.archive_layout import invocation_archive_dir -_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "verify_debug_e2e.py" +_SCRIPT_PATH = Path(__file__).resolve().parents[2] / "scripts" / "verify" / "verify_debug_e2e.py" _INVOCATION_AS_OF = datetime(2026, 5, 26, tzinfo=UTC) @pytest.fixture(scope="module") def verify_module() -> ModuleType: - """Load ``scripts/verify_debug_e2e.py`` as an importable module. + """Load ``scripts/verify/verify_debug_e2e.py`` as an importable module. Mirrors :func:`tests.scripts.test_verify_debug_e2e.verify_module` so the two suites can co-exist under the same conftest without diff --git a/tests/scripts/test_verify_feedback_loop.py b/tests/scripts/test_verify_feedback_loop.py index dccba85b7..fb2ec3e89 100644 --- a/tests/scripts/test_verify_feedback_loop.py +++ b/tests/scripts/test_verify_feedback_loop.py @@ -2,7 +2,7 @@ The script's *seed builder* (``build_test_seed``) and its pure *assertion helpers* are the unit-testable surface; the full end-to-end run is exercised by invoking -``scripts/verify_feedback_loop.py`` manually (it is the verify-of-the-verify, and it +``scripts/verify/verify_feedback_loop.py`` manually (it is the verify-of-the-verify, and it runs ``alembic upgrade head`` against a scratch DB — embedding that in pytest would just duplicate the script). These tests pin: diff --git a/tests/scripts/test_verify_genesis.py b/tests/scripts/test_verify_genesis.py index c562fe87a..194c0682e 100644 --- a/tests/scripts/test_verify_genesis.py +++ b/tests/scripts/test_verify_genesis.py @@ -1,6 +1,6 @@ """Genesis-verify script — the §8 first-run assertions made executable (ALP-858). -``scripts/verify_genesis.py`` (module: +``scripts/verify/verify_genesis.py`` (module: :mod:`alphamind.scripts.verify_genesis`) asserts the genesis-cutover runbook §8 checklist on a fresh account: zero reconciliation alerts (by construction), a canary entry fill that self-attributes via the broker-carried link (no diff --git a/tests/scripts/test_verify_position_thesis_model.py b/tests/scripts/test_verify_position_thesis_model.py index b7ad913ef..a218d4534 100644 --- a/tests/scripts/test_verify_position_thesis_model.py +++ b/tests/scripts/test_verify_position_thesis_model.py @@ -1,4 +1,4 @@ -"""Tests for scripts/verify_position_thesis_model.py (ALP-336). +"""Tests for scripts/verify/verify_position_thesis_model.py (ALP-336). The verify script is offline-only (no SDK, no DB) and exercises every deliverable from the position-thesis-model work tree's 15 sub-stories. @@ -42,7 +42,7 @@ # Helpers # --------------------------------------------------------------------------- -_SCRIPT_PATH = Path(__file__).parents[2] / "scripts" / "verify_position_thesis_model.py" +_SCRIPT_PATH = Path(__file__).parents[2] / "scripts" / "verify" / "verify_position_thesis_model.py" def _load_script() -> ModuleType: