Support source-based AutoML installation and development branches

.agents/skills/cosmos-automl/SKILL.md

@@ -6,10 +6,31 @@ description: Optimize Cosmos Framework reasoner SFT hyperparameters or evaluatio
 # Cosmos Framework AutoML
 
 Read [docs/automl.md](../../../docs/automl.md) before planning a search. It owns
-the installation/release prerequisites, manifest contract, result semantics
+the installation prerequisites, manifest contract, result semantics
 and current limits. Use the native `cosmos-automl` CLI; do not call a TAO runner
-or introduce model/runtime selection. The engine is an optional wheel, not a
-source-checkout dependency.
+or introduce model/runtime selection. Install the optional engine from the
+`packages/automl-core` subdirectory of `NVIDIA-TAO/tao-automl`; no TAO application,
+SDK, persistent source checkout, or publicly published wheel is required.
+
+Before a new installation, ask: **"Use AutoML core from `tao-automl/main`, or
+do you want a development branch from that repository? If so, which branch?"**
+Use `main` unless the user selects another branch; if they already supplied a
+branch, confirm that selection without asking again. Follow the
+[source-install instructions](../../../docs/automl.md#installation-from-source),
+including the named-branch override. Do not substitute a development branch
+when the selected source is unavailable. Record the resolved commit, installed
+package version and `engine_build`; a branch name alone is not a build identity.
+For an existing run, inspect and preserve its original environment/build instead
+of reinstalling the latest branch head. Source selection does not authorize
+upgrading a live controller or migrating a stopped experiment.
+
+Follow the [fresh-machine checklist](../../../docs/automl.md#fresh-machine-reproducibility-checklist):
+record Framework/core commits, the actual installed inventory, data/model
+identities and reviewed recipe/budgets. Preflight the evaluator and checkpoint
+export as well as training. Prefer local POSIX workspaces for local runs; shared
+storage needs concurrent atomic-read/replacement and locking checks. Report
+dependency warnings accurately; a successful training smoke is not a clean
+full-environment dependency check.
 
 Reuse **cosmos3-setup** for environment preparation and
 **cosmos3-post-training** for a prepared reasoner recipe and dataset/checkpoint
@@ -59,11 +80,19 @@ Keep this one skill entrypoint for both workflows.
 - Supervise the native CPU allocation before the brain starts; trial retries
   cannot handle bootstrap/import/prolog failures. Follow the SLURM guide's
   [startup contract](../../../docs/automl-slurm.md#before-the-search-controller-starts):
+  when startup recovery is required, explicitly configure `AUTOML_STARTUP_HELPER`,
+  its registered attempt base and the original deadline. Leaving the helper unset
+  preserves the direct-CLI launch path, but does not supply bootstrap validation
+  or pre-controller recovery; do not claim those checks ran. For the supervised path,
   arm the cross-node observer automatically before submission, preserve startup
   receipts, and mark the boundary before any algorithm/API/job fan-out. Only
   confirmed pre-fan-out infrastructure/observer failures may use approved CPU
   startup retries. Unknown submission outcomes block replacement. A missing
   experiment ledger is not success; report unlaunched modes as `NOT_STARTED`.
+  Use the guide's persistent observer service, verify `ready` before submission,
+  and register every acknowledged CPU job with its remaining allocation cap.
+  One-shot ticks or chat polling do not cover later requeue generations. The
+  watcher never replaces unknown submissions or extends the original deadline.
 - If the CPU allocation can expire first, review the SLURM guide's opt-in
   `--controller-max-allocations`, allocation seconds and grace in both plan and
   run. CPU handoff requeues the same CPU job and adopts existing GPU jobs; it
@@ -131,8 +160,16 @@ Keep this one skill entrypoint for both workflows.
   A `scheduler_unavailable` observation is not a failed trial: the controller
   retries read-only SLURM queries within the original deadline. Do not wrap a
   transient status error in blanket cancellation or restart a paid request.
+  Timed-out read-only preflight checks also wait within the original limits;
+  inspect `_scheduler_check.json`. A fully sealed, unsubmitted candidate can
+  continue across handoff; an attempted submission still requires reconciliation.
   If cancellation is unconfirmed, reconcile native jobs and worker receipts;
   neither a stale ledger nor a stop marker proves that GPUs have been released.
+  A local venv controller can exit while its worker continues. Reconcile both
+  before restarting; never upgrade its environment while workers are active.
+  Same-host venv cold migration is a separately approved, terminal-worker-only
+  operation preserving results, inventory, remaining budgets and deadline; see
+  [local recovery](../../../docs/automl.md#results-monitoring-and-recovery).
 - Hand off `best.json`, the winning native TOML, DCP and held-out metrics, or
   report that no trial succeeded. Do not call validation improvement final test
   performance or describe fixture checks as GPU end-to-end verification.
@@ -145,8 +182,9 @@ searches also passed bounded GPU smokes. Six of nine full-data numerical/populat
 searches completed; Bayesian, Hyperband and DEHB did not complete their approved
 budgets. Live infrastructure recovery and Hyperband-ES curve pruning remain
 unverified; smoke success is not production-scale or model-quality validation.
-Keep the core at `1.0.0.dev0` during the port; local builds are not releases.
-Use wheel hashes and `engine_build` to distinguish builds, not version bumps.
+Use the core package version required by Framework's extras (currently `0.1.0`);
+source installs and local builds are not releases. Use resolved Git commits,
+wheel hashes when applicable, and `engine_build` to distinguish builds, not version bumps.
 Preserve exact old wheels/manifests for paused searches; changed builds fail resume
 identity checks even when package versions match.
 Route ordinary training/evaluation to the existing skills, and do not infer

.agents/skills/cosmos3-setup/SKILL.md

@@ -51,6 +51,11 @@ The canonical setup reference is `docs/setup.md`. The README (`README.md` § Set
 - **CUDA compatibility**: compare the host driver with the selected CUDA build's requirements. `nvidia-smi` reports driver capability, not the installed toolkit; its CUDA version need not equal `torch.version.cuda`.
 - **`HF_HOME`**: controls where checkpoints are cached (default: `~/.cache/huggingface`). Set this if disk space is tight or you want a shared cache.
 - **Conflicting env vars**: stale `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` env vars can silently override CLI auth. Check with `printenv | grep HF_`.
+- **AutoML is a separate source install**: `uv sync --all-extras` does not install
+  the core engine. For AutoML setup, follow `docs/automl.md#installation-from-source`
+  and the `cosmos-automl` skill: ask whether to use `NVIDIA-TAO/tao-automl`'s `main`
+  branch or a user-selected development branch before a new installation. Keep
+  the recorded source revision for existing experiments; do not upgrade on resume.
 
 ## Related skills
 

.claude/skills/cosmos3-setup/SKILL.md

@@ -51,6 +51,11 @@ The canonical setup reference is `docs/setup.md`. The README (`README.md` § Set
 - **CUDA version alignment**: the major CUDA version from `nvidia-smi` must match `torch.version.cuda`. Mismatches cause cryptic shared-library errors.
 - **`HF_HOME`**: controls where checkpoints are cached (default: `~/.cache/huggingface`). Set this if disk space is tight or you want a shared cache.
 - **Conflicting env vars**: stale `HF_TOKEN` or `HUGGING_FACE_HUB_TOKEN` env vars can silently override CLI auth. Check with `printenv | grep HF_`.
+- **AutoML is a separate source install**: `uv sync --all-extras` does not install
+  the core engine. For AutoML setup, follow `docs/automl.md#installation-from-source`
+  and the `cosmos-automl` skill: ask whether to use `NVIDIA-TAO/tao-automl`'s `main`
+  branch or a user-selected development branch before a new installation. Keep
+  the recorded source revision for existing experiments; do not upgrade on resume.
 
 ## Related skills
 

.github/workflows/automl-helpers.yml

@@ -0,0 +1,29 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: OpenMDW-1.1
+
+name: AutoML helpers
+on:
+  pull_request:
+  push:
+    branches: [main]
+    paths:
+      - 'tools/cosmos_workflows/**'
+      - 'examples/automl/**'
+      - '.github/workflows/automl-helpers.yml'
+      - '.pytest.toml'
+permissions:
+  contents: read
+jobs:
+  cpu-regression:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: '3.12'
+      - run: python -m pip install pytest jsonschema
+      - name: Test execution helpers without GPU dependencies or scheduler access
+        run: >-
+          python -m pytest -o addopts= --confcutdir=tools/cosmos_workflows
+          -o 'python_files=test_*.py *_test.py' tools/cosmos_workflows

README.md

@@ -143,14 +143,14 @@ invokes automatically when the user's request matches its description.
 Skills live in [`.agents/skills/`](./.agents/skills) (canonical) and are
 mirrored under [`.claude/skills/`](./.claude/skills) for Claude Code:
 
-| Skill                                                                            | When it activates                                                                                          |
-| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
-| [`cosmos3-setup`](./.agents/skills/cosmos3-setup/SKILL.md)                       | Installation, environment setup, checkpoint downloads, Docker.                                             |
-| [`cosmos3-codebase-nav`](./.agents/skills/cosmos3-codebase-nav/SKILL.md)         | "Where is X" / "where do I change parameter Y" questions across `cosmos_framework/`.                       |
-| [`cosmos3-inference`](./.agents/skills/cosmos3-inference/SKILL.md)               | Running offline or online inference, parallelism, sampling parameters.                                     |
-| [`cosmos3-post-training`](./.agents/skills/cosmos3-post-training/SKILL.md)       | SFT post-training end-to-end: data prep, DCP conversion, launch, export.                                   |
-| [`cosmos-automl`](./.agents/skills/cosmos-automl/SKILL.md)                       | Reasoner hyperparameter search and GEPA prompt optimization; [setup and release status](./docs/automl.md). |
-| [`cosmos3-env-troubleshoot`](./.agents/skills/cosmos3-env-troubleshoot/SKILL.md) | Diagnosing install/runtime errors (ImportError, CUDA, Docker, checkpoint failures).                        |
+| Skill                                                                            | When it activates                                                                                                                        |
+| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| [`cosmos3-setup`](./.agents/skills/cosmos3-setup/SKILL.md)                       | Installation, environment setup, checkpoint downloads, Docker.                                                                           |
+| [`cosmos3-codebase-nav`](./.agents/skills/cosmos3-codebase-nav/SKILL.md)         | "Where is X" / "where do I change parameter Y" questions across `cosmos_framework/`.                                                     |
+| [`cosmos3-inference`](./.agents/skills/cosmos3-inference/SKILL.md)               | Running offline or online inference, parallelism, sampling parameters.                                                                   |
+| [`cosmos3-post-training`](./.agents/skills/cosmos3-post-training/SKILL.md)       | SFT post-training end-to-end: data prep, DCP conversion, launch, export.                                                                 |
+| [`cosmos-automl`](./.agents/skills/cosmos-automl/SKILL.md)                       | Reasoner hyperparameter search and GEPA prompt optimization; [main/development-branch setup](./docs/automl.md#installation-from-source). |
+| [`cosmos3-env-troubleshoot`](./.agents/skills/cosmos3-env-troubleshoot/SKILL.md) | Diagnosing install/runtime errors (ImportError, CUDA, Docker, checkpoint failures).                                                      |
 
 See [`AGENTS.md`](./AGENTS.md) for the canonical repo map that agents load
 first; the skills above are referenced from there.

cosmos_framework/scripts/automl.py

@@ -12,10 +12,14 @@ def main() -> None:
         if exc.name not in {"automl_core", "automl_core.cli"}:
             raise
         raise SystemExit(
-            "AutoML requires the separately supplied nvidia-automl-core 1.0.0.dev0 wheel. "
-            "Install that wheel into this Python environment first, then install "
-            "cosmos-framework[automl,reasoner-eval]. See docs/automl.md for the local-wheel "
-            "installation workflow (including uv run --no-sync)."
+            "AutoML requires nvidia-automl-core==0.1.0. Install packages/automl-core from "
+            "NVIDIA-TAO/tao-automl (main by default, or your selected development branch) "
+            "into this Python environment, then install cosmos-framework[automl,reasoner-eval]. "
+            "Git access to the selected repository/revision is required; if unavailable, "
+            "use a reviewed compatible core wheel supplied by your project maintainer. "
+            "Record the resolved commit and engine_build, and reuse them for existing runs. "
+            "See docs/automl.md#installation-from-source for source-install commands, "
+            "compatible versions, and uv run --no-sync. No published wheel is required."
         ) from exc
     if len(sys.argv) > 1 and sys.argv[1] == "prompts":
         from cosmos_framework.automl.prompts import load

docs/automl-algorithms.md

@@ -1,6 +1,7 @@
 # AutoML algorithm contracts
 
-Install the local core 1.0.0.dev0 wheel as described in [AutoML](automl.md). Choose
+Install the core from `tao-automl/main` or a selected development branch as
+described in [AutoML setup](automl.md#installation-from-source). Choose
 `search.algorithm` in the same native reasoner manifest; there are no competing
 launchers or target selectors. All modes use explicit objective direction,
 declared parameter domains, recorded native jobs and bounded sequential execution.

docs/automl-platforms.md

@@ -1,6 +1,6 @@
 # AutoML on Kubernetes, Brev and Lepton
 
-These are execution adapters in the optional `nvidia-automl-core` wheel, used by
+These are execution adapters in the optional `nvidia-automl-core` package, used by
 the existing `cosmos-automl` and `cosmos-automl prompts` commands. They do not
 call a TAO runner, require a source checkout, or introduce another search loop.
 All twelve SFT modes and GEPA use the same workload, scoring and promotion

docs/automl-prompts.md

@@ -6,15 +6,24 @@ weights and is not an SFT search algorithm. The reusable implementation lives
 in `automl_core`; Framework supplies native annotations, evaluation and metrics.
 Users need no TAO source checkout or SDK.
 
-Install the supplied core 1.0.0.dev0 wheel and Framework's `automl-prompts` extra:
+Follow [source installation](automl.md#installation-from-source), using `main` by
+default or an explicitly selected development branch. For a pip-managed,
+prepared Framework environment without a previous core installation:
 
 ```bash
-python -m pip install '/path/to/nvidia_automl_core-1.0.0.dev0-py3-none-any.whl[prompts]'
-python -m pip install -e '.[automl-prompts,reasoner-eval]'
+AUTOML_CORE_REF=main  # Or your development branch in NVIDIA-TAO/tao-automl.
+AUTOML_CORE_SOURCE="nvidia-automl-core @ git+https://github.com/NVIDIA-TAO/tao-automl.git@${AUTOML_CORE_REF}#subdirectory=packages/automl-core"
+python -m pip install "$AUTOML_CORE_SOURCE" -e '.[automl-prompts,reasoner-eval]'
+python -m pip check
 cosmos-automl prompts --help
 ```
 
-The controller extra pins `gepa==0.1.2`. The workload image needs the matching
+For a `uv`-managed environment, use the main guide's post-sync source install
+and `uv run --no-sync cosmos-automl prompts --help`; `--all-extras` includes
+Framework's GEPA dependency. Preserve the resolved source commit on resume.
+
+The Framework controller extra pins `gepa==0.1.2` directly; it does not depend on
+the selected core branch declaring a `prompts` extra. The workload image needs the matching
 Framework prompt evaluator and its normal GPU inference dependencies, not GEPA
 or an LLM credential. Existing training-only images are not implicitly upgraded.