Covered continuous-unwind shorts/longs — hardening + parity (cont. of #2764)

docs/derivatives/IMPLEMENTATION_PLAN.md

@@ -0,0 +1,180 @@
+# Implementation plan — covered continuous-unwind shorts
+
+Companion to `DESIGN.md`. Goal: land the spec's **shorts-first** launch with the smallest faithful
+diff, reusing `SubnetMovingPrice` (pEMA), the swap engine (fees/weights), recycle, and the dereg
+hook. Long paths are written symmetric but flag-gated off.
+
+Code is **not** written here — this is the build order, the exact files to touch, and the test/gate
+plan.
+
+---
+
+## Phase 0 — scaffolding (no behavior change)
+
+| File | Change | ~LOC |
+|---|---|---|
+| `pallets/subtensor/src/derivatives/mod.rs` | **new** module tree: `pub mod short; pub mod decay; pub mod settle; pub mod types;` | 10 |
+| `pallets/subtensor/src/lib.rs` | `pub mod derivatives;`; add storage items (`ShortPositions`, `ShortAggregate`, governance `StorageValue`s + `type_value` defaults) | ~70 |
+| `pallets/subtensor/src/derivatives/types.rs` | `ShortPosition`, `ShortAgg` structs (+ `freeze_struct`, derives) | ~40 |
+
+No migration (new empty maps default cleanly; confirmed against repo convention). No
+`STORAGE_VERSION` bump.
+
+---
+
+## Phase 1 — math core (pure, unit-testable, no extrinsics)
+
+All in `derivatives/short.rs` as `impl<T: Config> Pallet<T>` helpers. These are the spec
+closed-forms (Appendix A.1) used for quoting/sizing.
+
+| Function | Spec | Returns |
+|---|---|---|
+| `short_t_ref(netuid)` | §3.1, §4 | `min(SubnetTAO, pEMA·A_live)` |
+| `solve_collateral(p, t_ref, lambda, s)` | §4.2 | `(C, N)` via quadratic; reject `N ≤ 0` |
+| `lambda_eff(...)` | §4.1 | effective LTV; reject `≤ 0` |
+| `solve_phi(n, t_live)` | §4.3 | `ϕ = (1 − √(1 − 4N/T))/2`; reject `4N > T` |
+| `decay_factor_g(u)` | §6.2 | per-block `g` from `d_day(u)` |
+| `materialize(pos, agg)` | §6.3 | `f = exp(-(Ω−Ω_entry))`, scale `r,e,b` |
+
+Each gets a focused unit test asserting the spec's worked examples (§1.7–1.8: `C=100`, `N=37.5`,
+`ϕ≈0.039`, `Q≈3900`, `E=39`).
+
+---
+
+## Phase 2 — reserve legs (the risky part, isolate + test)
+
+`derivatives/settle.rs`: the three pool-touching primitives, each a thin wrapper over existing
+reserve helpers + the swap engine.
+
+| Function | Net reserve effect | Built from |
+|---|---|---|
+| `open_remove_sell_back(netuid, n, e, q)` | `SubnetTAO -= (N+E)`; book `Q` debt | `decrease_provided_tao_reserve`; engine quote to confirm realized `N` |
+| `restoration_zap(netuid, dU)` | `SubnetTAO += dU` (price drifts up) | `increase_provided_tao_reserve` (escalate to min-swap form only if sim demands) |
+| `settlement_zap(netuid, alpha_in, tao_in)` | balanced add of repaid `Q` + escrow | engine min-swap (§8.5) + `increase_provided_*` |
+
+**Gate for this phase:** the §3.5 conservation test — open → N blocks of decay → close (or default)
+returns exactly the TAO removed plus posted floor, minus equity. Run on a Balancer pool with
+non-default weights, not just 0.5/0.5.
+
+---
+
+## Phase 3 — extrinsics
+
+| File | Change | ~LOC |
+|---|---|---|
+| `derivatives/short.rs` | `do_open_short`, `do_top_up_short`, `do_close_short`, `do_default_short` (ensure_signed, validate, materialize, mutate, emit) | ~220 |
+| `macros/dispatches.rs` | 4 thin wrappers, `call_index` 139–142, placeholder `DbWeight` weights | ~50 |
+| `macros/events.rs` | 5 event variants | ~25 |
+| `macros/errors.rs` | ~8 error variants | ~12 |
+
+Validation order in `do_open_short` (spec §8.1): side flag → `SubnetMechanism==1` → solve `C,N` →
+reject `N≤0` / `4N>T` / `S+B>κ_S·T_ref` → solve `ϕ,Q,E` → realize legs → store/merge → bump
+aggregate. Same-block stacked opens read the progressively updated `b_sigma` (spec §5.2.1) for free,
+because each open re-reads `ShortAggregate`.
+
+---
+
+## Phase 4 — per-block decay hook
+
+| File | Change | ~LOC |
+|---|---|---|
+| `derivatives/decay.rs` | `run_derivatives_decay()` — iterate subnets with `b_sigma>0`, O(1) tick each (§6.4), call `restoration_zap` | ~70 |
+| `coinbase/block_step.rs` | one call after `run_coinbase`, before `update_moving_prices` | ~2 |
+
+---
+
+## Phase 5 — terminal dereg settlement
+
+| File | Change | ~LOC |
+|---|---|---|
+| `derivatives/settle.rs` | `settle_shorts_on_dereg(netuid)` — for each short: materialize, `K_D=max(K_spot,last, K_EMA)` (slippage-aware CPMM buyback, not scalar `Q·pEMA`), pay `equity`, `recycle_tao(liability_cover)`, extinguish `Q`, clear | ~90 |
+| `coinbase/root.rs` (`do_dissolve_network`) | call `settle_shorts_on_dereg(netuid)` before `destroy_alpha_in_out_stakes` | ~2 |
+
+`K_spot,last(Q)` = `sim_swap(GetAlphaForTao, …)` cost to buy `Q` at the final executable state;
+`pEMA` = `get_moving_alpha_price`. Buckets stay disjoint (liability-cover recycled outside terminal
+distribution — same rule as default), so no terminal fixed-point (spec §11.3).
+
+---
+
+## Phase 6 — runtime API
+
+| File | Change | ~LOC |
+|---|---|---|
+| `rpc_info/derivatives_info.rs` | **new** `ShortOpenQuote`, `ShortPositionInfo` DTOs + `quote_open_short`, `get_short_position` | ~110 |
+| `rpc_info/mod.rs` | `pub mod derivatives_info;` | 1 |
+| `runtime-api/src/lib.rs` | new trait `DerivativesRuntimeApi` (2 methods) + DTO imports | ~20 |
+| `runtime/src/lib.rs` | `impl DerivativesRuntimeApi for Runtime` in `impl_runtime_apis!` | ~12 |
+
+JSON-RPC (`pallets/subtensor/rpc`, `node/src/rpc.rs`) only if external clients need it — deferred.
+
+---
+
+## Phase 7 — governance wiring
+
+| File | Change | ~LOC |
+|---|---|---|
+| `utils/misc.rs` | `set_*` for each param (put + event), `get_*` readers | ~60 |
+| `admin-utils/src/lib.rs` | sudo/owner extrinsics: `sudo_set_shorts_enabled`, `…_short_kappa`, `…_short_base_ltv`, `…_decay_bounds`, `…_short_dust` | ~90 |
+
+`ShortsEnabled` stays `false` until the trading-games gate passes.
+
+---
+
+## Phase 8 — tests & trading-games gate (spec §14.5)
+
+`pallets/subtensor/src/tests/derivatives.rs` (+ eco-tests for adversarial sims). The spec makes these
+the launch gate, not optional:
+
+1. **Conservation** (§3.5) on weighted pools.
+2. **Same-block stacked opens** cannot bypass `S+B ≤ κ_S·T_ref` (§5.2.1).
+3. **Worked examples** (§1.7–1.8, §15) reproduce exactly.
+4. **Dust/escrow bound** `E/R ≤ 1/(1−ϕ_cap)` holds through top-ups/partials (§7.3).
+5. **Short-driven dereg**: no free terminal extraction; payout bounded by `K_D(Q)` (§10.7).
+6. **Flow neutrality**: assert `SubnetTaoFlow` unchanged across every derivative leg (§4.5).
+7. **Decay schedule**: 365-day remaining-fraction table (§14.3) within tolerance.
+
+Only after 1–7 pass on a mainnet-like replica does governance flip `ShortsEnabled` and begin ramping
+`κ_S` (spec §5.1, §14.6).
+
+---
+
+## Diff estimate
+
+| Area | Files touched | New files | ~LOC |
+|---|---|---|---|
+| Storage + types | `lib.rs` | `derivatives/{mod,types}.rs` | ~120 |
+| Math core | — | `derivatives/short.rs` (part) | ~120 |
+| Reserve legs | — | `derivatives/settle.rs` (part) | ~140 |
+| Extrinsics + FRAME surface | `dispatches.rs`, `events.rs`, `errors.rs` | — | ~90 |
+| Decay hook | `coinbase/block_step.rs` | `derivatives/decay.rs` | ~72 |
+| Dereg hook | `coinbase/root.rs` | — | ~92 |
+| Runtime API | `runtime-api/src/lib.rs`, `runtime/src/lib.rs`, `rpc_info/mod.rs` | `rpc_info/derivatives_info.rs` | ~143 |
+| Governance | `utils/misc.rs`, `admin-utils/src/lib.rs` | — | ~150 |
+| **Total (excl. tests)** | **~10 edited** | **~6 new** | **~1,000** |
+
+No on-chain migration. No `STORAGE_VERSION` bump. Reuses pEMA, swap engine, recycle, and dereg
+plumbing rather than re-implementing them — which is where the line-count is kept down.
+
+---
+
+## Build / sanity commands
+
+```bash
+# compile the pallet only (fast loop)
+cargo check -p pallet-subtensor
+
+# pallet tests
+cargo test -p pallet-subtensor derivatives
+
+# full runtime build (after runtime-api wiring)
+cargo check -p node-subtensor-runtime
+```
+
+## Open decisions for the author
+
+1. **Position granularity**: merged-per-`(coldkey,netuid)` (chosen, minimal) vs. multi-position with
+   an id index. Merge is spec-sanctioned (§8.6); revisit only if UX needs distinct lots.
+2. **Restoration realization**: net `SubnetTAO +=` (chosen) vs. explicit min-swap zap. Start with the
+   net form; escalate only if the conservation test on weighted pools fails.
+3. **`hotkey` association**: carry it for identity/precompile parity, or drop it and key purely on
+   coldkey. Carrying it is cheap and keeps consistency with staking.

docs/derivatives/QA_REPORT.md

@@ -0,0 +1,173 @@
+# Derivatives (covered shorts/longs) — QA & Test Report
+
+Scope: the pool-borrowing covered shorts/longs feature on the Alpha/TAO CPMM
+(continuation of #2764). Feature is governance-gated **off** by default
+(`ShortsEnabled`/`LongsEnabled = false`). This report records the QA performed on
+the branch and an overall score.
+
+**Overall QA & test score: 9/10.** All CI-grade gates green, comprehensive unit
+coverage on both sides, four independent adversarial review rounds passed, and a
+full on-chain lifecycle exercised on a live local chain. Weight benchmarks are now
+implemented and wired (extrinsics + the O(N) decay/dereg hooks); the remaining
+pre-mainnet items are operational gates, not code-correctness gaps: regenerating
+the weight constants on CI reference hardware, and the adversarial trading-games
+matrix (incl. the EMA-slowness safety margin) before any `κ` ramp or
+`ShortsEnabled` flip.
+
+---
+
+## 1. Build (local laptop, aarch64 macOS)
+
+| Target | Result |
+|---|---|
+| `cargo check -p pallet-subtensor` | ✅ clean |
+| `cargo check -p node-subtensor-runtime` (native, `SKIP_WASM_BUILD=1`) | ✅ clean |
+| **wasm runtime** (`cargo build -p node-subtensor-runtime`) | ✅ built (1m04s) |
+| **full node** (`cargo build -p node-subtensor`) | ✅ 325 MB binary |
+| **localnet release** (`--workspace --profile=release --features fast-runtime`) | ✅ 9m28s |
+
+macOS note: the wasm build needs a WebAssembly-capable LLVM (Apple clang lacks the
+target). Fix: `brew install llvm` and build with
+`CC_wasm32v1_none=/opt/homebrew/opt/llvm/bin/clang
+AR_wasm32v1_none=…/llvm-ar CFLAGS_wasm32v1_none="-DZSTD_DISABLE_ASM=1"`.
+
+## 2. Static analysis / style gates
+
+| Gate | Result |
+|---|---|
+| `cargo fmt --check` (all feature files) | ✅ clean |
+| `cargo clippy -p pallet-subtensor --lib` | ✅ no warnings in feature code (only an unrelated `trie-db` dependency future-incompat note) |
+
+## 3. Tests
+
+- **Derivatives suite: 74 tests, 0 failed.**
+- **Full pallet suite: `cargo test -p pallet-subtensor --lib` → 1258 passed / 0 failed / 9 ignored** (no regressions in adjacent staking / networks / weights / swap-hotkey / registration suites).
+
+Coverage by area (both short and long sides unless noted):
+
+- **Open**: quote↔open match; reject paths — disabled, stable-subnet, zero/min-input,
+  capacity (`κ·ref`), low-liquidity (`λ_eff ≤ 0`), cold-EMA fresh subnet; merge +
+  hotkey-mismatch; **execution-bound rejection** (`max_alpha_liability` /
+  `max_tao_liability`, both sides); validate-before-mutate (`assert_noop!`
+  strands-no-funds).
+- **Atomicity**: `open_short_failed_pool_transfer_rolls_back_atomically` — forces the
+  pool→custody leg to fail after the floor moved and asserts full `#[transactional]`
+  rollback (would fail without the attribute).
+- **Top-up / close**: partial + full close, alpha-mint guards, invalid fraction,
+  many-partial drain, close quote consistency.
+- **Default**: dust + grace eligibility, permissionless-default anti-snipe,
+  per-side grace independence, recycle-exactly-the-floor proof.
+- **Decay / dereg**: rate-vs-closed-form, restore, block-step, materialize-never-
+  inflates; full terminal matrix (in-the-money / underwater / cold-EMA) both sides;
+  dereg **full-`do_dissolve_network`-path** long-equity survival through the stake-wipe.
+- **Views/RPC, governance clamps, capacity/anti-split, active-set tracking.**
+- **Invariant proofs**: TAO+alpha conservation across the full mixed lifecycle;
+  **custody ≥ Σ materialized(P+R+E) under decay at the `DecayMax` clamp extreme,
+  checked every tick**; `ShortPositionCount == |ShortPositions[netuid]|` and
+  active-set ⟺ nonzero Σ through churn.
+
+## 4. Adversarial review (three independent lenses)
+
+| Lens | Verdict |
+|---|---|
+| Security skeptic | **CONVINCED** — atomicity (`#[transactional]`) + decay-restore ordering re-derived; no overflow / conservation / desync residue |
+| Architect skeptic | **CONVINCED** — operand-order footgun resolved, dereg ordering contract guarded by an end-to-end test, `pEMA` dependency documented, decay-hook bound noted; M2/M3 now property-tested |
+| Exploiter | **DEFEATED** — no profitable extraction across self-short-to-dereg, sandwich, capacity-split, decay-drift mint, EMA manipulation, cross-side, RPC abuse |
+
+Key hardening landed from review: caller-signed execution bounds; validate-before-mutate;
+`#[transactional]` on all 8 money functions; terminal `K_D = max(K_spot, K_EMA)` as a
+u128 ceiling CPMM buyback (fixes an `I64F64` rao² overflow) + short cold-EMA floor;
+decay restore-then-commit ordering; cancellation-stable `solve_collateral`.
+
+## 5. Precision review & on-chain CPMM audit
+
+- **Precision vs house style**: the only rao² product (terminal buyback) uses u128
+  ceiling math; all other `I64F64` use is price×rao / ratio×rao (rao-scale, no
+  overflow). `solve_collateral` uses the cancellation-stable root. Matches the
+  codebase's "no `I64F64` for rao² products" convention.
+- **Live CPMM audit** (128 dynamic mainnet subnets): no cold/empty/tiny/over-1.0-price
+  anomalies; spot↔EMA drift handled in the safe direction by the `min`/`max`
+  reference design. Documented `pEMA` caveat (`min(spot,1.0)` clamp, ~30d half-life;
+  guarantees hold for price ≤ ~1.0, true for all subnets today).
+
+## 6. Live local-chain end-to-end (3-validator `fast-runtime` localnet)
+
+- Chain produced blocks; runtime metadata contains all derivative extrinsics
+  (incl. the `max_alpha_liability` bound param) and governance setters.
+- **Governance**: `sudo_set_shorts_enabled`, `sudo_set_short_kappa`,
+  `sudo_set_subtoken_enabled`, `sudo_set_longs_enabled` — all applied on-chain.
+- **Full SHORT lifecycle**: `add_stake` (fund pool) → `open_short` (`ShortOpened`;
+  SubnetTAO fell by exactly R+E — conservation) → `top_up_short` (R grew by the
+  exact amount) → `close_short` full (`ShortClosed`; position cleared, escrow +
+  repaid Q returned to pool).
+- **Atomic rollback observed live**: an `open_short` against an unfunded pool failed
+  on the pool leg and rolled back completely (no position, `SubnetTAO` Δ=0, only the
+  tx fee).
+- **LONG side**: enabled; `open_long` correctly rejected on the alpha-depleted pool
+  via both guards (`EffectiveLtvNonPositive`, `AmountTooLow`) — extrinsic + domain
+  checks live (a clean long open needs a pool with healthy alpha reserve).
+
+## 7. Residual / out-of-scope (tracked, not code-correctness gaps)
+
+- **Benchmarked weights** — *now implemented:* FRAME v2 benchmarks exist for all 8
+  extrinsics plus the O(N) hooks (`run_short_decay`/`run_long_decay` with an
+  active-subnet component over `[0,128]`, `settle_shorts_on_dereg`/`settle_longs_on_dereg`
+  with a position component over `[0,1024]`); the 8 dispatches use `T::WeightInfo::*`,
+  `on_initialize` charges the per-block decay at `TotalNetworks`, and the dissolve
+  extrinsics charge terminal settlement at the *actual* per-subnet position count.
+  The remaining step is regenerating the weight constants on CI reference hardware
+  (`--extrinsic '*'`) before mainnet enablement — the harness/wiring is in place.
+- A clean **successful long open on-chain** was subsequently demonstrated on a
+  mainnet-seeded localnet (`open_long` P=1.0α → D liability at spot, full close
+  clears); also covered by unit tests (`long_dereg_in_the_money_pays_bounded_equity`,
+  conservation proofs).
+
+### 7.1 Pre-enablement checklist (must clear before `ShortsEnabled`/`κ` ramp)
+
+These are integration dependencies and operational invariants, not code-correctness
+gaps. They must be re-verified at enablement time because they depend on upstream
+state or governance configuration that can drift after merge.
+
+1. **Adversarial trading-games matrix** on a mainnet-like replica — EMA half-life ×
+   `κ` × pool depth × attacker capital × dereg distance × registration timing ×
+   spot-buy defense. The short-to-dereg safety margin and the one-sided
+   reserve-accounting approximation (intentional divergence from fee/weighted spot
+   execution) are only valid once this passes. **Must include position-count stress**
+   (not just economics): drive a subnet to a high synthetic open-position count within
+   the `κ`/min-input bounds and confirm the immediate dereg sweep stays within block
+   weight/latency.
+2. **`pEMA` dependency is load-bearing.** The safety math assumes
+   `SubnetMovingPrice = EMA(min(spot, 1.0))` with a slow half-life. If upstream
+   changes the `min(·,1.0)` clamp or the half-life, the derivative anti-suppression
+   math must be revalidated before enablement.
+3. **High-price-subnet caveat.** Because `pEMA` is clamped around 1.0, the terminal
+   anti-suppression guarantee is stated only for subnets priced below ~1.0 (true for
+   all mainnet subnets today). Confirm this still holds at enablement.
+4. **Benchmark validation boundaries.** Decay `WeightInfo` is benchmarked over
+   `[0,128]` (= `DefaultSubnetLimit`); `on_initialize` charges `run_*_decay(TotalNetworks)`
+   with the full count (linear, scaled — not under-charged — above 128, but only
+   *validated* to 128). Terminal settlement is benchmarked over `[0,1024]` and the
+   dissolve extrinsics charge `settle_*_on_dereg(actual position count)`; counts above
+   1024 extrapolate the per-position slope (scaled, not under-charged). There is no
+   per-subnet open-position cap (parity with the uncapped alpha-stake unwind); position
+   count is bounded only by the κ capacity limit and min-input. If the subnet count is
+   raised above 128, or subnets routinely carry more than 1024 positions, regenerate the
+   corresponding weights to re-validate linearity. `[0,1024]` is a benchmark calibration
+   range, **not** a consensus-enforced cap — terminal settlement is an immediate
+   enumerate-and-settle sweep (see DESIGN.md §3.4), modelled on native alpha liquidation.
+5. **CI reference-hardware weight regen** (`--extrinsic '*'`) — wiring is in place. The
+   `open_short`/`open_long` weights were regenerated locally after the cap removal (the stale
+   `Short/LongMaxPositions` storage-read annotation is gone); the final constants still need a
+   run on CI reference hardware before enablement.
+6. **Dereg weight/latency monitoring** — on a mainnet-seeded localnet, exercise
+   `dissolve_network` against subnets carrying high synthetic short/long position counts and
+   record settlement weight + wall-clock, confirming the immediate sweep stays within block
+   limits at the counts the `κ`/min-input economics actually permit.
+
+### 7.2 Accepted tradeoffs (intentional, not blockers)
+
+- **Terminal transfer failure = log + sweep, not abort.** A custody-invariant breach
+  during dereg settlement logs loudly and the unpaid value stays in custody to be
+  swept (recycled), rather than aborting the dereg. No value is created/lost and the
+  emitted `equity` reflects only what was actually paid; the position is underpaid
+  rather than the subnet bricked on a dust shortfall. Intentional.