diff --git a/README.md b/README.md index 8d808c7..c87d05d 100644 --- a/README.md +++ b/README.md @@ -1,456 +1,129 @@
-

pool

-

Elastic, load-balanced, self-healing pools of workers for the Wago WebAssembly runtime.

+

pool

+

Elastic, load-balanced, self-healing pools of Wago workers.

-

- CI - Coverage - Go >= 1.24 - Wago >= 0.1.0 -

+Pool is the policy layer above +[github.com/wago-org/workers](https://github.com/wago-org/workers). It groups +workers that run the same guest table entry, routes tagged byte payloads across +them, reconciles target size, and applies bounded restart and autoscaling rules. -
-Table of Contents +Workers owns the privileged instances. Pool has no Wago authorities of its own; +it depends on Workers and calls its typed contract. -- [Overview](#overview) -- [Installation](#installation) -- [Concepts](#concepts) -- [Usage](#usage) -- [API](#api) - - [Creating a pool](#creating-a-pool) - - [Submitting work](#submitting-work) - - [Routing strategies](#routing-strategies) - - [Overflow policies](#overflow-policies) - - [Autoscaling](#autoscaling) - - [Supervision & restart](#supervision--restart) - - [Draining & teardown](#draining--teardown) - - [Metrics & events](#metrics--events) - - [Options & limits](#options--limits) - - [Errors](#errors) -- [How it composes](#how-it-composes) -- [Testing](#testing) -- [Architecture](#architecture) -- [Contributing](#contributing) -- [License](#license) -- [Contact](#contact) +> Pool is experimental (`v0.1.0`). Its API may change before the first stable +> release. -
- -## Overview - -`pool` is an optional [Wago](https://github.com/wago-org/wago) plugin that turns the -low-level worker primitives of [`workers`](https://github.com/wago-org/workers) into -**managed pools**: named groups of workers that all run the same guest table entry, -with a routing strategy in front, supervision behind, and elastic sizing on top. - -It is the policy layer `workers` deliberately leaves out. Where `workers` gives you -`Spawn`, `Send`, `DispatchNext`, `Link`, and `Kill`, `pool` gives you: - -- **Load balancing** across a fleet with six routing strategies — round-robin, - least-loaded, random, power-of-two-choices, consistent-hash, and broadcast. -- **Autoscaling** — a pool grows under load and shrinks back when idle, driven by a - simple, timer-free target: outstanding tasks per worker. The ceiling is computed - from the machine's parallelism by default, so pools fill the CPU without - oversubscribing it. -- **Supervision** — workers that trap, are killed, or return are replaced according - to a restart policy, with crash-loop protection. -- **Backpressure** — full mailboxes spill to other workers, reject, or shed, your - choice. -- **Graceful draining** — stop accepting work and let in-flight tasks finish before - workers are torn down. -- **Observability** — per-pool and per-worker counters, plus a stream of lifecycle - and routing events. - -The whole thing is bounded by construction: every pool has a worker ceiling, and the -service has aggregate caps on pools and total workers — on top of the hard ceiling -enforced by the underlying `workers` service. - -> **Stability:** experimental (`v0.0.0`). The API may change without notice. - -## Installation - -If you have the [`wago`](https://github.com/wago-org/wago) CLI installed: - -```sh -wago pkg install github.com/JairusSW/pool -``` - -or use [`go get`](https://pkg.go.dev/cmd/go#hdr-Get_packages_and_dependencies): +## Install ```sh -go get github.com/JairusSW/pool -``` - -The pool plugin needs **no privileged capabilities of its own** — it composes the -`workers` service, which holds the privileged instance capabilities. A host that uses -pool must therefore also register `workers` and grant it `instance.manage` and -`instance.lifecycle`: - -```json -{ - "$schema": "https://wago.sh/v0/schema.json", - "plugins": { - "JairusSW/pool": "^0.0.0", - "wago-org/workers": "^0.0.0" - } -} +wago add github.com/JairusSW/pool ``` -The reviewed worker authority is recorded in `wago-lock.json`: - -```json -{ - "plugins": { - "wago-org/workers": { - "version": "0.0.0", - "requiredCapabilities": [ - "instance.manage", - "instance.lifecycle" - ], - "capabilities": { - "instance.manage": { "maxInstances": 256 }, - "instance.lifecycle": true - } - }, - "JairusSW/pool": { - "version": "0.0.0" - } - } -} -``` - -## Concepts - -| Term | Meaning | -| --- | --- | -| **Plugin** | `pool.New()` — the `wago.Extension` you register with a runtime. It provides one **service**. | -| **Service** (`*pool.Pools`) | The host-side handle you use to create pools and route work. Obtain it with `Plugin.Service()` or through `pool.ServiceKey`. | -| **Pool** | A named group of workers (`PoolID`) that all run the same table entry, fronted by a routing strategy and kept at a target size. | -| **Worker** | One member of a pool — a `workers` worker (`WorkerID`) driven on its own goroutine. | -| **Task** | A tagged byte payload routed to one worker (or broadcast to all). "Distribute stuff" = submit tasks. | -| **Desired size** | The number of workers a pool is converging toward — set by `MinWorkers`, autoscaling, `Scale`, or supervision. | - -A pool's life: **create → route tasks across members → grow/shrink & self-heal via -`Reconcile` → drain → destroy.** +The resolver adds `github.com/wago-org/workers`, reviews its authorities, and +locks both the package dependency and Pool's exact +`github.com/wago-org/workers/service@1` binding. The generated runtime links +both explicit `/register` catalogs; neither package self-registers from `init`. -## Usage +## Use from a plugin -Register `workers`, register `pool` wired to it, and drive it from your own host -imports — the embedder owns the guest ABI and passes the live caller through: +Pool provides `github.com/JairusSW/pool/service@1`: ```go -package main - -import ( - "log" - - "github.com/wago-org/wago" - "github.com/wago-org/workers" - "github.com/JairusSW/pool" -) - -func main() { - rt := wago.NewRuntime() - defer rt.Close() - - // 1. The workers service holds the privileged instance capabilities. - wk := workers.New(workers.WithLimits(workers.WorkerLimits{MaxLiveWorkers: 256})) - if err := rt.Use(wk, wago.WithPluginGrants( - wago.PluginManagedInstances, // "instance.manage" - wago.PluginInstanceHooks, // "instance.lifecycle" - )); err != nil { - log.Fatal(err) - } - - // 2. The pool service composes it. For programmatic embedding via Use, wire it - // explicitly with WithWorkers; under the manifest/LoadPlugins plan path the - // runtime binds it automatically through workers.ServiceKey. - pl := pool.New(pool.WithWorkers(wk.Service())) - if err := rt.Use(pl); err != nil { - log.Fatal(err) +func Definition() wago.PluginDefinition { + return wago.PluginDefinition{ + // ...ID, version, and provenance... + Requires: []wago.PluginRequirement{{ + ID: pool.PluginID, Version: "^0.1.0", + }}, + Consumes: []wago.ContractRequirement{{ + ID: pool.Contract.ID(), Major: pool.Contract.Major(), + Mode: wago.ContractRequired, + }}, } - pools := pl.Service() - - // 3. Observe pool lifecycle and routing (optional). - pools.OnEvent(func(ev *pool.PoolEvent) { - log.Printf("pool %d event kind=%d worker=%d", ev.Pool, ev.Kind, ev.Worker) - }) - - // ... inside a host import, create a pool from the calling instance and route: - // id, _ := pools.Create(caller, tableIndex, pool.PoolOptions{ - // Strategy: pool.LeastLoaded, MinWorkers: 4, MaxWorkers: 32, TargetPerWorker: 8, - // }) - // pools.Submit(id, tag, payload) // no caller needed } -``` - -`Create`, `Reconcile`, `Scale`, and `SubmitFrom` fork the calling instance, so they -must run **inside a live host import**, where `caller` is the active -`wago.HostModule`. `Submit`, `SubmitKeyed`, `Broadcast`, `Drain`, `Destroy`, and -`Stats` take no caller and may be called from anywhere. -## API - -### Creating a pool +poolRef, err := wagoplugin.Require(reg, pool.Contract) +if err != nil { + return err +} -```go -id, err := pools.Create(caller, tableIndex, pool.PoolOptions{ - Strategy: pool.LeastLoaded, - MinWorkers: 4, - MaxWorkers: 32, - TargetPerWorker: 8, // autoscale: ~8 outstanding tasks per worker - Restart: pool.RestartOnFailure, +err = poolRef.With(func(service pool.Service) error { + id, err := service.Create(caller, tableIndex, pool.PoolOptions{ + Strategy: pool.PowerOfTwo, + MinWorkers: 2, + MaxWorkers: 16, + }) + if err != nil { + return err + } + _, err = service.Submit(id, 42, payload) + return err }) ``` -`Create` spawns `MinWorkers` workers immediately, each running the guest table entry -at `tableIndex` (which must have the Wasm signature `() -> ()`), forked from the -calling instance and linked to it so the pool is torn down when the caller closes. If -it cannot spawn the full minimum, the partial pool is destroyed and the error is -returned. - -### Submitting work - -`Submit` routes one task to a worker using the pool's strategy and returns the -`WorkerID` that accepted it. It copies the payload and never blocks: - -```go -wid, err := pools.Submit(id, tag, payload) -``` - -- `SubmitKeyed(id, key, tag, payload)` routes by an explicit key (independent of the - tag) — this is the routing key for `ConsistentHash`. -- `SubmitFrom(caller, id, tag, payload)` reconciles the pool first (self-healing and - autoscaling from a live caller), then routes. Use it on a guest's hot path to make - the pool grow and recover under load. -- `Broadcast(id, tag, payload)` delivers the task to every worker and returns the - count that accepted it. - -### Routing strategies - -| Strategy | Behavior | -| --- | --- | -| `RoundRobin` | Cycles through workers in order — even spread. **(default)** | -| `LeastLoaded` | Routes to the worker with the smallest mailbox backlog. | -| `Random` | Uniformly random worker. | -| `PowerOfTwo` | Samples two workers, picks the less loaded — near-least-loaded at O(1). | -| `ConsistentHash` | Routes by key via a virtual-node hash ring: same key → same worker, and scaling remaps only a fraction of keys. | -| `Broadcast` | Every task goes to every worker. | - -### Overflow policies - -When a routed task's chosen worker mailbox is full, `OverflowPolicy` decides what -happens: - -| Policy | Behavior | -| --- | --- | -| `OverflowSpill` | Try the remaining workers in preference order. **(default)** | -| `OverflowReject` | Fail fast with `ErrPoolBackpressure`. | -| `OverflowShed` | Drop the task silently (counted as *shed*), returning `(0, nil)`. | - -### Autoscaling - -Set `TargetPerWorker > 0` to make a pool **elastic**. Its desired size is derived — -with no timers — from current load: - -``` -desired = clamp(ceil(totalOutstanding / TargetPerWorker), MinWorkers, MaxWorkers) -``` - -`Reconcile(caller, id)` converges the pool to that target: it spawns workers to make -up a deficit (needs the live caller, since spawning forks it) and kills **idle** -workers when the pool is over target — never dropping queued work. `SubmitFrom` calls -`Reconcile` for you on the hot path, so a busy pool grows itself. +The package edge installs Pool and its Workers dependency transitively. The +contract edge pins the exact reviewed Pool provider the consumer may call. -**Optimal sizing, no oversubscription.** You don't have to guess a ceiling. Leave -`MaxWorkers` at `0` on an elastic pool and it is sized to `OptimalWorkers()` — the -machine's parallelism (`GOMAXPROCS`) — so the pool grows to fill the CPU but never -runs more compute-bound workers than the hardware can execute in parallel (which only -adds context-switching, scheduler contention, and cache pressure). Across pools, the -service-wide `RunnableWorkers` budget (also `OptimalWorkers()` by default) caps how -many workers autoscaling will run at once, so several busy elastic pools can't -collectively oversubscribe the CPU. Each pool's `MinWorkers` floor is always honored; -the budget caps *growth*, never starves a pool. Call `pool.OptimalWorkers()` yourself -to size things explicitly, and set `MaxWorkers`/`RunnableWorkers` higher for -I/O-bound workloads that benefit from more concurrency than cores. +`Create`, `Reconcile`, `Scale`, and `SubmitFrom` require the active +`wago.HostModule`, because growing a pool forks the exact guest caller. +`Submit`, `SubmitKeyed`, `Broadcast`, `Drain`, `Destroy`, `Stats`, and `List` +can be called without a guest caller. -For a **fixed** pool (leave `TargetPerWorker` at 0), the size is what you set: -`MinWorkers` at creation, adjusted by `Scale(caller, id, target)` and supervision. +The service is available only inside `Ref.With`. Wago prevents new calls after +the consumer stops and waits for calls already in flight before Workers stops. -### Supervision & restart +## Routing and supervision -When a worker exits, the pool reacts according to `RestartPolicy` (fixed pools) or -self-heals to the load target (elastic pools): +Strategies: -| Policy | Behavior | -| --- | --- | -| `RestartOnFailure` | Replace a worker that trapped or was killed; let a clean return shrink the pool. **(default)** | -| `RestartAlways` | Replace every exited worker, keeping the pool full even as workers return. | -| `RestartNever` | Never replace; the pool shrinks permanently. | +- `RoundRobin` +- `LeastLoaded` +- `Random` +- `PowerOfTwo` +- `ConsistentHash` +- `Broadcast` -Replacement happens on the next `Reconcile`/`SubmitFrom` (which supplies a live -caller). `MaxRestarts` caps total replacements over a pool's life; exceeding it -**halts** replacement (crash-loop protection) and is reported in `Stats().Halted`. Use -`UnlimitedRestarts` to disable the cap. +Overflow can spill to another worker, reject immediately, or shed a task. +Restart policy can replace failed workers, every exit, or no exits. A bounded +restart counter stops crash loops. Autoscaling derives a target from outstanding +tasks and clamps it to the pool and service ceilings; the default runnable +ceiling is `GOMAXPROCS`. -### Draining & teardown +Pool events use owned subscriptions: ```go -pools.Drain(id) // stop accepting; kill idle workers now, busy ones once they drain -pools.Destroy(id) // kill everything immediately and remove the pool +events, err := service.ObserveEvents(func(event *pool.PoolEvent) { + log.Printf("pool %d event %d", event.Pool, event.Kind) +}) +// In the consuming plugin's Stop callback, through poolRef.With: +err = service.UnsubscribeEvents(events) ``` -`Drain` is graceful: new submits are rejected with `ErrPoolDraining`, workers with an -empty mailbox are killed at once, and each remaining worker is killed the moment its -mailbox empties. `Destroy` is immediate and drops any queued tasks. - -### Metrics & events +Unsubscribing waits for callbacks already in flight. It must not be called +inside its own callback. -`Stats(id)` returns an atomic snapshot — pool size and target, per-worker backlog, and -lifetime counters (submitted, routed, rejected, shed, restarted, exited, failed): +## Configure -```go -st, _ := pools.Stats(id) -log.Printf("live=%d desired=%d routed=%d rejected=%d outstanding=%d", - st.Live, st.Desired, st.Routed, st.Rejected, st.Outstanding) +```sh +wago plugin config github.com/JairusSW/pool \ + '{"maxPools":64,"maxWorkersPerPool":64,"maxTotalWorkers":256,"runnableWorkers":16}' ``` -`OnEvent` streams `PoolEvent`s — `PoolCreated`, `PoolWorkerAdded`, `PoolWorkerRemoved` -(with the exit kind), `PoolScaled`, `PoolTaskRouted`, `PoolTaskRejected`, -`PoolDrained`, `PoolDestroyed` — for logging, metrics, or external orchestration. -Observers add zero cost to the hot path when none are registered. - -### Options & limits - -`PoolOptions` (per pool): - -| Field | Default | Meaning | -| --- | --- | --- | -| `Strategy` | `RoundRobin` | Routing algorithm. | -| `Overflow` | `OverflowSpill` | Full-mailbox behavior. | -| `Restart` | `RestartOnFailure` | Replacement policy for a fixed pool. | -| `MinWorkers` | `1` | Initial size and autoscale floor. | -| `MaxWorkers` | `OptimalWorkers()` if elastic, else `= MinWorkers` | Autoscale ceiling. `0` auto-sizes an elastic pool to the CPU. | -| `TargetPerWorker` | `0` (off) | Outstanding tasks per worker; turns autoscaling on. | -| `MaxRestarts` | `16` | Replacement budget; `UnlimitedRestarts` disables it. | -| `Worker` | `workers` defaults | Per-worker mailbox bounds (`workers.WorkerOptions`). | -| `HashSeed` | derived from `PoolID` | Seed for the ring and random strategies. | -| `RingReplicas` | `64` | Virtual nodes per worker on the `ConsistentHash` ring. | - -`Limits` (whole service, via `pool.WithLimits`): - -| Field | Default | Meaning | -| --- | --- | --- | -| `MaxPools` | `64` | Maximum pools at once. | -| `MaxWorkersPerPool` | `64` | Cap per pool, overriding a larger `MaxWorkers`. | -| `MaxTotalWorkers` | `256` | Hard cap on workers summed across all pools. | -| `RunnableWorkers` | `OptimalWorkers()` | Cap on workers autoscaling will run across all pools, to avoid CPU oversubscription. Growth-only; floors are always honored. | - -`MaxPools`/`MaxWorkersPerPool`/`MaxTotalWorkers` are hard safety bounds; -`RunnableWorkers` is the performance guard that keeps elastic pools from -oversubscribing the CPU. All of these sit above the `workers` service's own -`WorkerLimits`, which is the hard ceiling on live workers. +All four limits must be positive when explicitly set. Configuration is strict: +unknown fields, trailing JSON, and a per-pool ceiling above the total ceiling +are rejected before activation. -### Errors - -All errors are comparable sentinels — match them with `errors.Is`. - -| Error | When | -| --- | --- | -| `ErrPoolsInactive` | Service is nil or not wired to a workers service. | -| `ErrPoolsClosed` | The service has been stopped. | -| `ErrWorkersUnavailable` | The workers service could not be resolved. | -| `ErrPoolNotFound` | No pool with that ID. | -| `ErrPoolEmpty` | The pool has no live workers to route to. | -| `ErrPoolDraining` | The pool is draining or destroyed. | -| `ErrPoolBackpressure` | Every candidate worker was at capacity. | -| `ErrInvalidPoolOptions` | Options are inconsistent (e.g. `MaxWorkers < MinWorkers`). | -| `ErrPoolLimit` | The service's `MaxPools` was reached. | -| `ErrPoolWorkerLimit` | A per-pool or total worker ceiling was reached. | -| `ErrPoolIDExhausted` | The `PoolID` space is exhausted. | - -## How it composes - -`pool` → `workers` → `wago`. The pool service never touches the core instance API -directly; it holds a reference to the `workers` service and expresses everything — -spawning, routing, teardown, supervision — through `Spawn`, `Send`, `DispatchNext`, -`Link`, `Kill`, and the `OnMessage`/`OnExit` observers. This is the plugin-composition -story Wago is built for: one plugin's service becomes another's foundation. - -Two wiring paths are supported: - -- **Manifest / `LoadPlugins`** — the runtime resolves `workers.ServiceKey` and binds it - into the pool plugin automatically. This is the production path. -- **Programmatic `Runtime.Use`** — pass the workers handle explicitly with - `pool.WithWorkers(wk.Service())`, since `Use` commits each plugin independently and - does not run cross-plugin service resolution. - -A generated Wago host does not import this package directly; it blank-imports the -[`register`](./register) subpackage, which activates init-time registration for both -pool and its `workers` dependency: - -```go -import _ "github.com/JairusSW/pool/register" -``` - -## Testing +## Test ```sh go test ./... -go test -race ./... # the concurrency paths are the whole point -go test -short ./... # smaller stress sizes for a quick pass +go test -race ./... ``` -The suite mixes pure-logic unit tests (strategies, the autoscale target, supervision -state machine, the hash ring) with end-to-end integration tests that build real guest -modules via `github.com/wago-org/wago/testutil/wasmtest` — spawning pools, load -balancing across them, poisoning workers and healing, draining, and autoscaling from a -real backlog — plus concurrency stress tests that hammer pools from many producers -while workers churn. - -## Architecture - -- **`pool.go`** — the whole service: the `Plugin` extension, the `Pools` service, - routing, overflow, autoscaling, supervision, draining, and the `OnMessage`/`OnExit` - wiring onto the workers service. -- **`ring.go`** — the virtual-node consistent-hash ring for `ConsistentHash` routing. -- **`register/`** — a blank-import shim that activates registration for pool and its - workers dependency. -- **`wago.json`** — the package manifest. - -Design notes: - -- **One lock, short critical sections.** All pool state is guarded by a single mutex; - the mutex is never held across a `workers` call or an observer callback, so routing - scales and there is no lock-order cycle with the worker goroutines that deliver - `OnMessage`/`OnExit`. -- **Timer-free autoscaling.** Desired size is a pure function of current load, applied - on `Reconcile`. There are no background goroutines to reason about or shut down. -- **Caller-safe healing.** Spawning forks the calling instance, so it only happens - inside a live host import. Supervision records a deficit; the next - `Reconcile`/`SubmitFrom` fills it with the live caller. - -## Contributing - -Contributions are welcome! Please: - -- Run `go test -race ./...` and `go vet ./...` before opening a pull request. -- Keep the layering intact — `pool` builds on the `workers` service and adds policy; - it does not reach into the core instance API. -- Follow standard Go formatting (`gofmt`) and conventional commit messages. +The suite exercises the real Workers → Pool → consumer contract graph, routing +strategies, overflow, supervision, autoscaling, teardown, and concurrent stress. ## License -This project is distributed under the [Apache License 2.0](./LICENSE). Work on this -project is done out of passion — if you want to support it financially, you can donate -through [GitHub Sponsors](https://github.com/sponsors/JairusSW). - -## Contact - -Please file issues at [GitHub Issues](https://github.com/JairusSW/pool/issues). To chat, -join the [Wago Discord](https://wago.sh/discord). - -- **GitHub:** [https://github.com/wago-org/](https://github.com/wago-org/) -- **Website:** [https://wago.sh/](https://wago.sh/) -- **Discord:** [https://wago.sh/discord](https://wago.sh/discord) +Apache-2.0. See [LICENSE](./LICENSE). diff --git a/behaviors_test.go b/behaviors_test.go index 658d218..7a3a001 100644 --- a/behaviors_test.go +++ b/behaviors_test.go @@ -159,11 +159,15 @@ func TestIntegrationOnEvent(t *testing.T) { var mu sync.Mutex kinds := map[PoolEventKind]int{} - r.d.pools.OnEvent(func(ev *PoolEvent) { + events, err := r.d.pools.ObserveEvents(func(ev *PoolEvent) { mu.Lock() kinds[ev.Kind]++ mu.Unlock() }) + if err != nil { + t.Fatal(err) + } + defer func() { _ = r.d.pools.UnsubscribeEvents(events) }() r.invoke(t, "start") id := r.d.pool() diff --git a/go.mod b/go.mod index fcefafc..211ca79 100644 --- a/go.mod +++ b/go.mod @@ -2,6 +2,6 @@ module github.com/JairusSW/pool go 1.24.0 -require github.com/wago-org/wago v0.0.0-20260711053856-7d89b98c854d +require github.com/wago-org/wago v0.0.0-20260812144524-1c58c9862d25 -require github.com/wago-org/workers v0.0.0-20260712075105-c0bceb592dc0 +require github.com/wago-org/workers v0.0.0-20260812144603-4a90ff25d5e4 diff --git a/go.sum b/go.sum index 59883be..3720533 100644 --- a/go.sum +++ b/go.sum @@ -1,4 +1,4 @@ -github.com/wago-org/wago v0.0.0-20260711053856-7d89b98c854d h1:fzJn6UlsGwiqwyTN+Si6oFoETUhUoJ/Rphhx94M5Pqc= -github.com/wago-org/wago v0.0.0-20260711053856-7d89b98c854d/go.mod h1:d44S+59u6VHeqEaIZBKTeB8viyswTP97O9IKA6Gyzc4= -github.com/wago-org/workers v0.0.0-20260712075105-c0bceb592dc0 h1:CRsh9Z+Iilmr6odi9Xlg2asrpffciQrHIARIVlzQ9sg= -github.com/wago-org/workers v0.0.0-20260712075105-c0bceb592dc0/go.mod h1:lRgtcCNsXr30h+ZovoKMGhKMvk6slR1cwFDZ7HoBo2g= +github.com/wago-org/wago v0.0.0-20260812144524-1c58c9862d25 h1:gopNHvaUaDSzmW2Hfd32r8dOwwzYUHH3ph1L/IuhJUA= +github.com/wago-org/wago v0.0.0-20260812144524-1c58c9862d25/go.mod h1:6XmxI3S5qJ+YAzObyXjC6FmxHtbLhBBxrglzix42zl8= +github.com/wago-org/workers v0.0.0-20260812144603-4a90ff25d5e4 h1:Z7S9Hj+JZ8OdNWLgzRjUQOvPIEQpEFNtyRSoHnUYOlI= +github.com/wago-org/workers v0.0.0-20260812144603-4a90ff25d5e4/go.mod h1:FLuFlQ7n+DNP0VNAZuGc702UIxC/JfLuhyuacMQNFp8= diff --git a/integration_test.go b/integration_test.go index 64c460e..256d589 100644 --- a/integration_test.go +++ b/integration_test.go @@ -1,34 +1,79 @@ package pool import ( + "context" + "encoding/json" + "errors" + "sort" "sync" "testing" "time" "github.com/wago-org/wago" - "github.com/wago-org/wago/testutil/wasmtest" + wagoplugin "github.com/wago-org/wago/plugin" + "github.com/wago-org/wago/tests/wasmtest" "github.com/wago-org/workers" ) -// Guest ABI op codes for the single `env.run` host import used by the test guest. const ( - opNext = 0 // worker: drain one message; -> 0 continue, 1 stop, 2 trap - opCreate = 1 // main: create a pool on table a with b workers; -> pool id - opReconcile = 2 // main: reconcile the last-created pool - opScale = 3 // main: scale the last-created pool to a - opSubmitFrom = 4 // main: reconcile-then-route a task with tag a + opNext = 0 + opCreate = 1 + opReconcile = 2 + opScale = 3 + opSubmitFrom = 4 ) -// driver is a tiny host plugin that bridges the guest to the Pools and Workers -// services. It is the "embedder" layer: it owns the host imports and passes the -// live caller through to pool operations, exactly as a real application would. -type driver struct { - once sync.Once - pools *Pools - workers *workers.Workers - createOpts PoolOptions +type registerFunc func(*wago.Registrar) error + +func (f registerFunc) Register(reg *wago.Registrar) error { return f(reg) } + +func testPluginSet(t *testing.T, providers []wago.PluginProvider, configs map[string]json.RawMessage) wago.PluginSet { + t.Helper() + set := wago.PluginSet{Providers: providers} + for _, provider := range providers { + digest, err := wago.DefinitionDigest(provider.Definition) + if err != nil { + t.Fatal(err) + } + selection := wago.PluginSelection{ + ID: provider.Definition.ID, DefinitionDigest: digest, Direct: true, + Dependencies: map[string]string{}, Config: configs[provider.Definition.ID], + } + for _, requirement := range provider.Definition.Requires { + selection.Dependencies[requirement.ID] = requirement.Version + } + for _, authority := range provider.Definition.Authorities { + selection.Grants = append(selection.Grants, wago.AuthorityGrant{Name: authority.Name, Scope: authority.Scope}) + } + for _, requirement := range provider.Definition.Consumes { + var owners []string + for _, candidate := range providers { + for _, provided := range candidate.Definition.Provides { + if provided.ID == requirement.ID && provided.Major == requirement.Major { + owners = append(owners, candidate.Definition.ID) + } + } + } + sort.Strings(owners) + if requirement.Mode != wago.ContractMany && len(owners) > 1 { + owners = owners[:1] + } + selection.Contracts = append(selection.Contracts, wago.ContractBinding{ID: requirement.ID, Major: requirement.Major, Providers: owners}) + } + set.Selections = append(set.Selections, selection) + } + return set +} - gate sync.RWMutex // write-locked to pause workers before they dispatch +type driver struct { + pools *Pools + poolRef *wagoplugin.Ref[Service] + workers *wagoplugin.Ref[workers.Service] + opts PoolOptions + message workers.Subscription + gate sync.RWMutex + ctx context.Context + cancel context.CancelFunc mu sync.Mutex lastPool PoolID @@ -40,32 +85,60 @@ type driver struct { trapNext map[WorkerID]bool } -func newDriver(pools *Pools, wk *workers.Workers, opts PoolOptions) *driver { - return &driver{pools: pools, workers: wk, createOpts: opts, got: map[WorkerID]int{}, trapNext: map[WorkerID]bool{}} -} - -func (d *driver) Info() wago.ExtensionInfo { - return wago.ExtensionInfo{ - ID: "pool.driver", Name: "pool test driver", - RequiresCapabilities: []wago.PluginCapability{wago.PluginHostImports}, - } -} - -func (d *driver) Register(reg *wago.Registry) error { - imports, err := reg.HostImports() - if err != nil { - return err - } - imports.Module("env").Func("run", d.run). - Params(wago.ValI32, wago.ValI32, wago.ValI32).Results(wago.ValI32) - return nil +func driverProvider(d *driver) wago.PluginProvider { + definition := wago.PluginDefinition{ + ID: "example.com/pool-driver", Version: "1.0.0", + Provenance: wago.PluginProvenance{Repository: "https://example.com/pool-driver", License: "MIT"}, + Requires: []wago.PluginRequirement{ + {ID: PluginID, Version: "^0.1.0"}, + {ID: workers.PluginID, Version: "^0.1.0"}, + }, + Authorities: []wago.AuthorityRequest{{ + Name: wago.AuthorityHostImportDefine, Mode: wago.AuthorityRequired, + Reason: "bridge the integration guest to Pool and Workers", + Scope: wago.AuthorityScope{Modules: []string{"env"}}, + }}, + Consumes: []wago.ContractRequirement{ + {ID: Contract.ID(), Major: Contract.Major(), Mode: wago.ContractRequired}, + {ID: workers.Contract.ID(), Major: workers.Contract.Major(), Mode: wago.ContractRequired}, + }, + } + return wago.PluginProvider{Definition: definition, New: func() wago.Plugin { + return registerFunc(func(reg *wago.Registrar) error { + var err error + d.poolRef, err = wagoplugin.Require(reg, Contract) + if err != nil { + return err + } + d.workers, err = wagoplugin.Require(reg, workers.Contract) + if err != nil { + return err + } + imports, err := reg.HostImports() + if err != nil { + return err + } + module, err := imports.Module("env") + if err != nil { + return err + } + module.Func("run", d.run).Params(wago.ValI32, wago.ValI32, wago.ValI32).Results(wago.ValI32) + return reg.Lifecycle(wago.PluginLifecycle{ + Start: d.start, + Stop: func(context.Context) error { + d.cancel() + return d.workers.With(func(service workers.Service) error { return service.Unsubscribe(d.message) }) + }, + }) + }) + }} } -// wire registers the message-counting observer once, before any worker is -// spawned (the first run call is the guest's create). -func (d *driver) wire() { - d.once.Do(func() { - d.workers.OnMessage(func(ctx *workers.MessageContext) error { +func (d *driver) start(context.Context) error { + d.ctx, d.cancel = context.WithCancel(context.Background()) + return d.workers.With(func(service workers.Service) error { + var err error + d.message, err = service.ObserveMessages(func(ctx *workers.MessageContext) error { d.mu.Lock() d.got[ctx.WorkerID]++ d.msgs++ @@ -75,53 +148,60 @@ func (d *driver) wire() { d.mu.Unlock() return nil }) + return err }) } func (d *driver) run(caller wago.HostModule, params, results []uint64) { - d.wire() op, a, b := uint32(params[0]), uint32(params[1]), uint32(params[2]) switch op { case opNext: - // Gate before dispatching so a paused pool holds a real backlog. d.gate.RLock() d.gate.RUnlock() - if err := d.workers.DispatchNext(caller); err != nil { + if err := d.workers.With(func(service workers.Service) error { return service.DispatchNext(d.ctx, caller) }); err != nil { results[0] = 1 return } - if wid, err := d.workers.Current(caller); err == nil { + _ = d.workers.With(func(service workers.Service) error { + wid, err := service.Current(caller) + if err != nil { + results[0] = 1 + return nil + } d.mu.Lock() trap := d.trapNext[wid] delete(d.trapNext, wid) d.mu.Unlock() if trap { results[0] = 2 - return } - } - results[0] = 0 + return nil + }) case opCreate: - opts := d.createOpts + opts := d.opts opts.MinWorkers = b - id, err := d.pools.Create(caller, a, opts) - d.mu.Lock() - if err == nil { - d.lastPool = id - d.allPools = append(d.allPools, id) - } else { - d.createErrs = append(d.createErrs, err) - } - d.mu.Unlock() - if err == nil { - results[0] = uint64(id) - } + _ = d.poolRef.With(func(service Service) error { + id, err := service.Create(caller, a, opts) + d.mu.Lock() + if err != nil { + d.createErrs = append(d.createErrs, err) + } else { + d.lastPool = id + d.allPools = append(d.allPools, id) + results[0] = uint64(id) + } + d.mu.Unlock() + return err + }) case opReconcile: - _ = d.pools.Reconcile(caller, d.pool()) + _ = d.poolRef.With(func(service Service) error { return service.Reconcile(caller, d.pool()) }) case opScale: - _ = d.pools.Scale(caller, d.pool(), a) + _ = d.poolRef.With(func(service Service) error { return service.Scale(caller, d.pool(), a) }) case opSubmitFrom: - _, _ = d.pools.SubmitFrom(caller, d.pool(), uint64(a), []byte("pump")) + _ = d.poolRef.With(func(service Service) error { + _, err := service.SubmitFrom(caller, d.pool(), uint64(a), []byte("pump")) + return err + }) } } @@ -131,103 +211,59 @@ func (d *driver) pool() PoolID { return d.lastPool } -func (d *driver) pools_() []PoolID { +func (d *driver) messages() int { d.mu.Lock() defer d.mu.Unlock() - return append([]PoolID(nil), d.allPools...) + return d.msgs } -func (d *driver) messages() int { +func (d *driver) pools_() []PoolID { d.mu.Lock() defer d.mu.Unlock() - return d.msgs + return append([]PoolID(nil), d.allPools...) } func (d *driver) distribution() map[WorkerID]int { d.mu.Lock() defer d.mu.Unlock() - out := map[WorkerID]int{} - for k, v := range d.got { - out[k] = v + out := make(map[WorkerID]int, len(d.got)) + for id, count := range d.got { + out[id] = count } return out } -// buildGuest hand-assembles a wasm module exposing: -// - table[0] = $worker: loop { r=run(NEXT); if r==2 trap; if r!=0 return } -// - export "start": run(CREATE, 0, count) -// - export "reconcile": run(RECONCILE, 0, 0) -// - export "scale": run(SCALE, scaleTarget, 0) func buildGuest(count, scaleTarget uint32) []byte { - // Types: 0 = ()->(), 1 = (i32,i32,i32)->(i32). typeVoid := []byte{0x60, 0x00, 0x00} typeRun := []byte{0x60, 0x03, 0x7f, 0x7f, 0x7f, 0x01, 0x7f} - types := wasmtest.Vec(typeVoid, typeRun) - - // Import env.run : type 1. imp := append(wasmtest.Name("env"), wasmtest.Name("run")...) - imp = append(imp, 0x00) // func import - imp = append(imp, wasmtest.ULEB(1)...) // type index 1 - imports := wasmtest.Vec(imp) - - // Five local funcs, all type 0. (Imported run is func index 0.) + imp = append(imp, 0x00) + imp = append(imp, wasmtest.ULEB(1)...) funcs := wasmtest.Vec(wasmtest.ULEB(0), wasmtest.ULEB(0), wasmtest.ULEB(0), wasmtest.ULEB(0), wasmtest.ULEB(0)) - - // One funcref table, min size 1. - table := wasmtest.Vec([]byte{0x70, 0x00, 0x01}) - - // Exports: start=func2, reconcile=func3, scale=func4, pump=func5. exports := wasmtest.Vec( - wasmtest.ExportEntry("start", 0x00, 2), - wasmtest.ExportEntry("reconcile", 0x00, 3), - wasmtest.ExportEntry("scale", 0x00, 4), - wasmtest.ExportEntry("pump", 0x00, 5), + wasmtest.ExportEntry("start", 0x00, 2), wasmtest.ExportEntry("reconcile", 0x00, 3), + wasmtest.ExportEntry("scale", 0x00, 4), wasmtest.ExportEntry("pump", 0x00, 5), ) - - // Active element segment: table[0] = func1 ($worker). elem := []byte{0x00, 0x41, 0x00, 0x0b} - elem = append(elem, wasmtest.ULEB(1)...) // one entry - elem = append(elem, wasmtest.ULEB(1)...) // func index 1 - elems := wasmtest.Vec(elem) - - // $worker with one i32 local: loop { r=run(0,0,0); if r==2 unreachable; if r==0 continue }. + elem = append(elem, wasmtest.ULEB(1)...) + elem = append(elem, wasmtest.ULEB(1)...) workerBody := []byte{ - 0x03, 0x40, // loop void - 0x41, 0x00, 0x41, 0x00, 0x41, 0x00, // i32.const 0 x3 (op,a,b) - 0x10, 0x00, // call 0 (run) - 0x21, 0x00, // local.set 0 - 0x20, 0x00, // local.get 0 - 0x41, 0x02, // i32.const 2 - 0x46, // i32.eq - 0x04, 0x40, // if void - 0x00, // unreachable - 0x0b, // end if - 0x20, 0x00, // local.get 0 - 0x45, // i32.eqz - 0x0d, 0x00, // br_if 0 (continue loop while r==0) - 0x0b, // end loop - 0x0b, // end func - } - worker := codeWithI32Locals(1, workerBody) - - start := wasmtest.Code(cmd(opCreate, 0, count)) - reconcile := wasmtest.Code(cmd(opReconcile, 0, 0)) - scale := wasmtest.Code(cmd(opScale, scaleTarget, 0)) - pump := wasmtest.Code(cmd(opSubmitFrom, 7, 0)) - code := wasmtest.Vec(worker, start, reconcile, scale, pump) - + 0x03, 0x40, 0x41, 0x00, 0x41, 0x00, 0x41, 0x00, 0x10, 0x00, + 0x21, 0x00, 0x20, 0x00, 0x41, 0x02, 0x46, 0x04, 0x40, 0x00, 0x0b, + 0x20, 0x00, 0x45, 0x0d, 0x00, 0x0b, 0x0b, + } + code := wasmtest.Vec( + codeWithI32Locals(1, workerBody), wasmtest.Code(cmd(opCreate, 0, count)), + wasmtest.Code(cmd(opReconcile, 0, 0)), wasmtest.Code(cmd(opScale, scaleTarget, 0)), + wasmtest.Code(cmd(opSubmitFrom, 7, 0)), + ) return wasmtest.Module( - wasmtest.Section(1, types), - wasmtest.Section(2, imports), - wasmtest.Section(3, funcs), - wasmtest.Section(4, table), - wasmtest.Section(7, exports), - wasmtest.Section(9, elems), - wasmtest.Section(10, code), + wasmtest.Section(1, wasmtest.Vec(typeVoid, typeRun)), wasmtest.Section(2, wasmtest.Vec(imp)), + wasmtest.Section(3, funcs), wasmtest.Section(4, wasmtest.Vec([]byte{0x70, 0x00, 0x01})), + wasmtest.Section(7, exports), wasmtest.Section(9, wasmtest.Vec(elem)), wasmtest.Section(10, code), ) } -// cmd emits `run(op, a, b); drop; end` for a void command function. func cmd(op, a, b uint32) []byte { body := []byte{0x41} body = append(body, wasmtest.SLEB32(int32(op))...) @@ -235,88 +271,80 @@ func cmd(op, a, b uint32) []byte { body = append(body, wasmtest.SLEB32(int32(a))...) body = append(body, 0x41) body = append(body, wasmtest.SLEB32(int32(b))...) - body = append(body, 0x10, 0x00, 0x1a, 0x0b) // call 0; drop; end - return body + return append(body, 0x10, 0x00, 0x1a, 0x0b) } -// codeWithI32Locals wraps a function body that declares n i32 locals into a code -// section entry (wasmtest.Code only supports zero locals). func codeWithI32Locals(n uint32, body []byte) []byte { - var decls []byte - if n == 0 { - decls = []byte{0x00} - } else { - decls = append([]byte{0x01}, wasmtest.ULEB(n)...) - decls = append(decls, 0x7f) // i32 - } + decls := append([]byte{0x01}, wasmtest.ULEB(n)...) + decls = append(decls, 0x7f) fn := append(decls, body...) return append(wasmtest.ULEB(uint32(len(fn))), fn...) } -// rig wires a runtime with the workers plugin (generous limits), the pool -// plugin, and the test driver, then compiles and instantiates the guest. type rig struct { - d *driver - rt *wago.Runtime - in *wago.Instance - // mod is retained for the rig's whole lifetime on purpose: the runtime sets a - // finalizer on the compiled module that closes its executable code mapping when - // the module is garbage-collected, after which forking a worker fails with - // "compiled module is closed". Keeping the reference alive prevents that. + d *driver + rt *wago.Runtime + in *wago.Instance mod *wago.Module } func newRig(t *testing.T, opts PoolOptions, count, scaleTarget uint32) *rig { - // RunnableWorkers is set high so autoscale tests are deterministic regardless of - // the host's core count; the CPU-budget guard is exercised on its own. return newRigLim(t, opts, count, scaleTarget, Limits{MaxPools: 32, MaxWorkersPerPool: 64, MaxTotalWorkers: 256, RunnableWorkers: 256}) } func newRigLim(t *testing.T, opts PoolOptions, count, scaleTarget uint32, limits Limits) *rig { t.Helper() - rt := wago.NewRuntime() - wk := workers.New(workers.WithLimits(workers.WorkerLimits{MaxLiveWorkers: 256, MaxQueueBytes: 512 << 20})) - if err := rt.Use(wk); err != nil { - t.Fatalf("use workers: %v", err) - } - poolPlugin := New(WithWorkers(wk.Service()), WithLimits(limits)) - if err := rt.Use(poolPlugin); err != nil { - t.Fatalf("use pool: %v", err) + d := &driver{opts: opts, got: map[WorkerID]int{}, trapNext: map[WorkerID]bool{}} + poolProvider := Provider() + var poolPlugin *plugin + poolProvider.New = func() wago.Plugin { poolPlugin = new(plugin); return poolPlugin } + // Deliberately link the graph in reverse dependency order. LoadPlugins must + // derive Workers -> Pool -> consumer from Requires and contract bindings. + providers := []wago.PluginProvider{driverProvider(d), poolProvider, workers.Provider()} + limits = normalizeLimits(limits) + poolConfig, err := json.Marshal(map[string]uint32{ + "maxPools": limits.MaxPools, "maxWorkersPerPool": limits.MaxWorkersPerPool, + "maxTotalWorkers": limits.MaxTotalWorkers, "runnableWorkers": limits.RunnableWorkers, + }) + if err != nil { + t.Fatal(err) } - d := newDriver(poolPlugin.Service(), wk.Service(), opts) - if err := rt.Use(d); err != nil { - t.Fatalf("use driver: %v", err) + set := testPluginSet(t, providers, map[string]json.RawMessage{ + workers.PluginID: json.RawMessage(`{"maxLiveWorkers":256,"maxQueueBytes":536870912}`), + PluginID: poolConfig, + }) + rt := wago.NewRuntime() + if err := rt.LoadPlugins(context.Background(), set); err != nil { + t.Fatal(err) } + d.pools = poolPlugin.service mod, err := rt.Compile(buildGuest(count, scaleTarget)) if err != nil { - t.Fatalf("compile: %v", err) + t.Fatal(err) } in, err := rt.Instantiate(nil, mod) if err != nil { - t.Fatalf("instantiate: %v", err) + t.Fatal(err) } return &rig{d: d, rt: rt, in: in, mod: mod} } +func (r *rig) close() { _ = r.in.Close(); _ = r.rt.Close() } + func (r *rig) invoke(t *testing.T, name string) { t.Helper() if _, err := r.in.Invoke(name); err != nil { - t.Fatalf("invoke %s: %v", name, err) + t.Fatal(err) } } -func (r *rig) close() { - _ = r.in.Close() - _ = r.rt.Close() -} - func (r *rig) stats(t *testing.T) PoolStats { t.Helper() - st, err := r.d.pools.Stats(r.d.pool()) + stats, err := r.d.pools.Stats(r.d.pool()) if err != nil { - t.Fatalf("stats: %v", err) + t.Fatal(err) } - return st + return stats } func waitFor(t *testing.T, what string, cond func() bool) { @@ -331,185 +359,54 @@ func waitFor(t *testing.T, what string, cond func() bool) { t.Fatalf("timeout waiting for %s", what) } -func TestIntegrationRoundRobinLoadBalancing(t *testing.T) { - r := newRig(t, PoolOptions{Strategy: RoundRobin}, 4, 0) +func TestContractGraphRoutesAndScales(t *testing.T) { + r := newRig(t, PoolOptions{Strategy: RoundRobin, MaxWorkers: 4}, 2, 4) defer r.close() r.invoke(t, "start") - - st := r.stats(t) - if st.Live != 4 { - t.Fatalf("live = %d, want 4", st.Live) + if got := r.stats(t).Live; got != 2 { + t.Fatalf("live = %d, want 2", got) } - - const n = 200 - for i := 0; i < n; i++ { + for i := 0; i < 20; i++ { if _, err := r.d.pools.Submit(r.d.pool(), uint64(i), []byte("task")); err != nil { - t.Fatalf("submit %d: %v", i, err) + t.Fatal(err) } } - waitFor(t, "all tasks dispatched", func() bool { return r.d.messages() == n }) - - st = r.stats(t) - if st.Routed != n || st.Rejected != 0 { - t.Fatalf("routed=%d rejected=%d, want %d,0", st.Routed, st.Rejected, n) - } - // Round-robin over 4 workers => exactly n/4 each. - dist := r.d.distribution() - if len(dist) != 4 { - t.Fatalf("distribution touched %d workers, want 4: %v", len(dist), dist) - } - for wid, c := range dist { - if c != n/4 { - t.Fatalf("worker %d got %d, want %d (even round-robin)", wid, c, n/4) - } - } -} - -func TestIntegrationBroadcast(t *testing.T) { - r := newRig(t, PoolOptions{Strategy: Broadcast}, 5, 0) - defer r.close() - r.invoke(t, "start") - - delivered, err := r.d.pools.Broadcast(r.d.pool(), 7, []byte("ping")) - if err != nil { + waitFor(t, "tasks", func() bool { return r.d.messages() == 20 }) + r.invoke(t, "scale") + waitFor(t, "scale", func() bool { return r.stats(t).Live == 4 }) + if err := r.rt.Close(); err != nil { t.Fatal(err) } - if delivered != 5 { - t.Fatalf("broadcast delivered to %d, want 5", delivered) - } - waitFor(t, "every worker received the broadcast", func() bool { return r.d.messages() == 5 }) - dist := r.d.distribution() - if len(dist) != 5 { - t.Fatalf("broadcast reached %d workers, want 5", len(dist)) + if err := r.d.poolRef.With(func(Service) error { return nil }); !errors.Is(err, wago.ErrPermissionDenied) { + t.Fatalf("pool contract after close = %v", err) } -} - -func TestIntegrationRestartOnFailure(t *testing.T) { - r := newRig(t, PoolOptions{Strategy: RoundRobin, Restart: RestartOnFailure}, 4, 0) - defer r.close() - - r.d.mu.Lock() - r.d.trapTag = 0xDEAD - r.d.mu.Unlock() - r.invoke(t, "start") - - // Send one poisoned task; whichever worker gets it traps and fails. - if _, err := r.d.pools.Submit(r.d.pool(), 0xDEAD, []byte("boom")); err != nil { - t.Fatalf("submit trap: %v", err) - } - waitFor(t, "a worker to fail", func() bool { return r.stats(t).Failed >= 1 }) - if got := r.stats(t).Live; got != 3 { - t.Fatalf("after failure live = %d, want 3", got) - } - - // Reconcile from a live caller heals the pool back to full strength. - r.invoke(t, "reconcile") - waitFor(t, "pool to heal", func() bool { return r.stats(t).Live == 4 }) - st := r.stats(t) - if st.Failed != 1 || st.Restarted < 1 { - t.Fatalf("after heal failed=%d restarted=%d, want 1,>=1", st.Failed, st.Restarted) - } -} - -func TestIntegrationScaleGrows(t *testing.T) { - // Manual pool with headroom (band [2,6], autoscale off): Scale is authoritative - // and grows the pool to the requested size from a live caller. - r := newRig(t, PoolOptions{Strategy: RoundRobin, MaxWorkers: 6}, 2, 6) - defer r.close() - r.invoke(t, "start") - if got := r.stats(t).Live; got != 2 { - t.Fatalf("initial live = %d, want 2", got) + if err := r.d.workers.With(func(workers.Service) error { return nil }); !errors.Is(err, wago.ErrPermissionDenied) { + t.Fatalf("driver Workers contract after close = %v", err) } - r.invoke(t, "scale") - waitFor(t, "pool to scale up", func() bool { return r.stats(t).Live == 6 }) - - // A second reconcile must not disturb a manual pool (autoscale off). - r.invoke(t, "reconcile") - if got := r.stats(t).Live; got != 6 { - t.Fatalf("manual pool drifted after reconcile: live = %d, want 6", got) + if err := r.d.pools.ref.With(func(workers.Service) error { return nil }); !errors.Is(err, wago.ErrPermissionDenied) { + t.Fatalf("Pool's Workers contract after close = %v", err) } -} - -func TestIntegrationScaleClamped(t *testing.T) { - // Fixed pool (Max==Min==3): Scale beyond capacity is clamped, not grown. - r := newRig(t, PoolOptions{Strategy: RoundRobin}, 3, 10) - defer r.close() - r.invoke(t, "start") - r.invoke(t, "scale") - if got := r.stats(t).Live; got != 3 { - t.Fatalf("fixed-pool scale clamped live = %d, want 3", got) + if pools := r.d.pools.List(); len(pools) != 0 { + t.Fatalf("Pool retained resources after close: %v", pools) } } -func TestIntegrationAutoscaleFromBacklog(t *testing.T) { - // Elastic band [1,6], target 4 tasks/worker. Pause workers, build a backlog of - // 12 on the single worker, then reconcile: desired = ceil(12/4) = 3. - r := newRig(t, PoolOptions{Strategy: RoundRobin, MaxWorkers: 6, TargetPerWorker: 4}, 1, 0) - defer r.close() - - r.d.gate.Lock() // pause: workers block before dispatching - resumed := false - resume := func() { - if !resumed { - r.d.gate.Unlock() - resumed = true - } - } - defer resume() // never close the runtime while workers are gated - - r.invoke(t, "start") - if got := r.stats(t).Live; got != 1 { - t.Fatalf("initial live = %d, want 1", got) - } - - const backlog = 12 - for i := 0; i < backlog; i++ { - if _, err := r.d.pools.Submit(r.d.pool(), uint64(i), []byte("q")); err != nil { - t.Fatalf("submit %d: %v", i, err) +func TestProviderRejectsMissingWorkersAndStrictConfig(t *testing.T) { + if err := wago.ValidatePluginSet(testPluginSet(t, []wago.PluginProvider{Provider()}, nil)); err == nil { + t.Fatal("pool without workers succeeded") + } + for _, config := range []json.RawMessage{ + json.RawMessage(`null`), + json.RawMessage(`[]`), + json.RawMessage(`{"maxPools":null}`), + json.RawMessage(`{"maxPools":1,"maxPools":2}`), + json.RawMessage(`{"unknown":1}`), + json.RawMessage(`{"maxPools":0}`), + json.RawMessage(`{"maxWorkersPerPool":10,"maxTotalWorkers":2}`), + } { + set := testPluginSet(t, []wago.PluginProvider{workers.Provider(), Provider()}, map[string]json.RawMessage{PluginID: config}) + if err := wago.ValidatePluginSet(set); err == nil { + t.Fatalf("accepted invalid config %s", config) } } - waitFor(t, "backlog to register", func() bool { return r.stats(t).Outstanding == backlog }) - - r.invoke(t, "reconcile") - if got := r.stats(t).Live; got != 3 { - t.Fatalf("autoscaled live = %d, want 3 (ceil(12/4))", got) - } - - resume() - waitFor(t, "backlog to drain", func() bool { return r.stats(t).Outstanding == 0 }) -} - -func TestIntegrationDrainGraceful(t *testing.T) { - r := newRig(t, PoolOptions{Strategy: RoundRobin}, 4, 0) - defer r.close() - r.invoke(t, "start") - waitFor(t, "workers to idle", func() bool { return r.stats(t).Live == 4 }) - - if err := r.d.pools.Drain(r.d.pool()); err != nil { - t.Fatalf("drain: %v", err) - } - waitFor(t, "pool to drain to zero", func() bool { return r.stats(t).Live == 0 }) - if _, err := r.d.pools.Submit(r.d.pool(), 1, nil); err != ErrPoolDraining { - t.Fatalf("submit after drain = %v, want ErrPoolDraining", err) - } -} - -func TestIntegrationDestroy(t *testing.T) { - r := newRig(t, PoolOptions{Strategy: RoundRobin}, 4, 0) - defer r.close() - r.invoke(t, "start") - id := r.d.pool() - waitFor(t, "workers to idle", func() bool { - st, _ := r.d.pools.Stats(id) - return st.Live == 4 - }) - if err := r.d.pools.Destroy(id); err != nil { - t.Fatalf("destroy: %v", err) - } - if _, err := r.d.pools.Stats(id); err != ErrPoolNotFound { - t.Fatalf("stats after destroy = %v, want ErrPoolNotFound", err) - } - if len(r.d.pools.List()) != 0 { - t.Fatalf("pools remain after destroy: %v", r.d.pools.List()) - } } diff --git a/manifest_test.go b/manifest_test.go new file mode 100644 index 0000000..0c0ebca --- /dev/null +++ b/manifest_test.go @@ -0,0 +1,108 @@ +package pool_test + +import ( + "encoding/json" + "os" + "reflect" + "testing" + + "github.com/JairusSW/pool" + poolregister "github.com/JairusSW/pool/register" + "github.com/wago-org/wago" + "github.com/wago-org/workers" +) + +type manifestAuthor struct { + Name string `json:"name"` +} + +type manifestPackage struct { + Module string `json:"module"` + Version string `json:"version"` + Name string `json:"name"` + Description string `json:"description"` + Stability wago.Stability `json:"stability"` + License string `json:"license"` + Homepage string `json:"homepage"` + Repository string `json:"repository"` + Authors []manifestAuthor `json:"authors"` + Engines map[string]string `json:"engines"` + Platforms []string `json:"platforms"` +} + +func TestManifestMatchesCatalogMetadataAndWorkersDependency(t *testing.T) { + raw, err := os.ReadFile("wago.json") + if err != nil { + t.Fatal(err) + } + var manifest struct { + Schema string `json:"$schema"` + Package manifestPackage `json:"package"` + Plugins map[string]string `json:"plugins"` + } + if err := json.Unmarshal(raw, &manifest); err != nil { + t.Fatal(err) + } + if manifest.Schema != "https://wago.sh/v1/schema.json" { + t.Fatalf("manifest schema = %q", manifest.Schema) + } + providers := poolregister.Providers() + if len(providers) != 1 { + t.Fatalf("catalog providers = %d, want 1", len(providers)) + } + definition := pool.Definition() + if !reflect.DeepEqual(providers[0].Definition, definition) { + t.Fatalf("catalog definition drifted\ncatalog=%#v\ncanonical=%#v", providers[0].Definition, definition) + } + assertManifestMetadata(t, manifest.Package, definition) + if len(definition.Requires) != 1 || definition.Requires[0].ID != workers.PluginID { + t.Fatalf("Workers definition requirement = %#v", definition.Requires) + } + if got := manifest.Plugins[workers.PluginID]; len(manifest.Plugins) != 1 || got != definition.Requires[0].Version { + t.Fatalf("Workers manifest dependency = %v, definition = %#v", manifest.Plugins, definition.Requires) + } + if len(definition.Consumes) != 1 || definition.Consumes[0].ID != workers.Contract.ID() || + definition.Consumes[0].Major != workers.Contract.Major() || definition.Consumes[0].Mode != wago.ContractRequired { + t.Fatalf("Workers Contract requirement = %#v", definition.Consumes) + } + assertProviderCatalogCurrent(t, "github.com/JairusSW/pool/register", providers) +} + +func assertProviderCatalogCurrent(t *testing.T, importPath string, providers []wago.PluginProvider) { + t.Helper() + want, err := wago.EncodeProviderCatalog(importPath, providers) + if err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(wago.ProviderCatalogFile) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("%s is stale; run wago plugin catalog", wago.ProviderCatalogFile) + } + if _, err := wago.DecodeProviderCatalog(got); err != nil { + t.Fatalf("%s: %v", wago.ProviderCatalogFile, err) + } +} + +func assertManifestMetadata(t *testing.T, manifest manifestPackage, definition wago.PluginDefinition) { + t.Helper() + authors := make([]string, len(manifest.Authors)) + for i := range manifest.Authors { + authors[i] = manifest.Authors[i].Name + } + if manifest.Module != definition.ID || + manifest.Version != definition.Version || + manifest.Name != definition.Name || + manifest.Description != definition.Description || + manifest.Stability != definition.Stability || + manifest.License != definition.Provenance.License || + manifest.Homepage != definition.Provenance.Homepage || + manifest.Repository != definition.Provenance.Repository || + !reflect.DeepEqual(authors, definition.Provenance.Authors) || + !reflect.DeepEqual(manifest.Engines, definition.Compatibility.Engines) || + !reflect.DeepEqual(manifest.Platforms, definition.Compatibility.Platforms) { + t.Fatalf("manifest metadata drifted\nmanifest=%#v\ndefinition=%#v", manifest, definition) + } +} diff --git a/pool.go b/pool.go index 6471a7f..38cd200 100644 --- a/pool.go +++ b/pool.go @@ -6,25 +6,30 @@ // load-balancing strategy, keeps the pool at its target size by respawning or // scaling workers, and can grow and shrink with load. It builds entirely on the // primitives of the github.com/wago-org/workers plugin — Spawn, Send, -// DispatchNext, Link, Kill, plus the OnMessage/OnExit host observers — and adds +// DispatchNext, Link, Kill, plus owned message and exit subscriptions — and adds // the policy layer that workers deliberately leaves out: routing, supervision, // autoscaling, draining, and per-pool metrics. // -// The pool plugin uses no core capabilities of its own. It composes the workers -// service through workers.ServiceKey, so a host that registers pool must also -// register workers (grant workers its instance.manage / instance.lifecycle -// capabilities). See the README for the full model. +// Pool requests no Wago Authorities of its own. Its explicit package dependency +// selects Workers, and its typed Contract reference keeps every cross-plugin +// call and shutdown dependency inside a revocable callback lease. See the README +// for the full model. package pool import ( + "bytes" "context" + "encoding/json" + "errors" + "fmt" + "io" "runtime" "sort" "sync" "sync/atomic" "github.com/wago-org/wago" - "github.com/wago-org/wago/plugin" + wagoplugin "github.com/wago-org/wago/plugin" "github.com/wago-org/workers" ) @@ -43,8 +48,7 @@ func OptimalWorkers() uint32 { return 1 } -// PluginName is the registration name of the pool plugin. -const PluginName = "JairusSW/pool" +const PluginID = "github.com/JairusSW/pool" // WorkerID and WorkerOptions are re-exported from the workers plugin for the // convenience of callers that only import pool. @@ -317,98 +321,196 @@ type PoolEvent struct { Err error } -// ServiceKey exposes the Pools service to composing plugins. -var ServiceKey = plugin.NewServiceKey[*Pools]("wago.pool/v1") - -// Plugin is the wago.Extension you register with a runtime. -type Plugin struct { - service *Pools - limits Limits - workers *workers.Workers +// Service is Pool's major-versioned cross-plugin API. +type Service interface { + Create(wago.HostModule, uint32, PoolOptions) (PoolID, error) + Submit(PoolID, uint64, []byte) (WorkerID, error) + SubmitKeyed(PoolID, uint64, uint64, []byte) (WorkerID, error) + SubmitFrom(wago.HostModule, PoolID, uint64, []byte) (WorkerID, error) + Broadcast(PoolID, uint64, []byte) (int, error) + Reconcile(wago.HostModule, PoolID) error + Scale(wago.HostModule, PoolID, uint32) error + Drain(PoolID) error + Destroy(PoolID) error + Stats(PoolID) (PoolStats, error) + List() []PoolID + ObserveEvents(func(*PoolEvent)) (EventSubscription, error) + UnsubscribeEvents(EventSubscription) error } -// Option configures the Plugin at construction. -type Option func(*Plugin) +var Contract = wagoplugin.NewContract[Service](PluginID+"/service", 1) -// WithLimits sets the aggregate resource limits for the pool service. Zero -// fields fall back to the package defaults (see Limits). -func WithLimits(l Limits) Option { return func(p *Plugin) { p.limits = l } } +type pluginConfig struct { + MaxPools *uint32 `json:"maxPools,omitempty"` + MaxWorkersPerPool *uint32 `json:"maxWorkersPerPool,omitempty"` + MaxTotalWorkers *uint32 `json:"maxTotalWorkers,omitempty"` + RunnableWorkers *uint32 `json:"runnableWorkers,omitempty"` +} -// WithWorkers wires the pool to an existing workers service explicitly, for -// programmatic embedding via Runtime.Use (where cross-plugin service resolution -// does not run). Obtain the handle from workers.Plugin.Service(). When omitted, -// the pool resolves the workers service through workers.ServiceKey, which the -// runtime binds on the manifest/LoadPlugins plan path. -func WithWorkers(svc *workers.Workers) Option { return func(p *Plugin) { p.workers = svc } } +var configSchema = json.RawMessage(`{ + "type": "object", + "additionalProperties": false, + "properties": { + "maxPools": {"type": "integer", "minimum": 1, "maximum": 65536}, + "maxWorkersPerPool": {"type": "integer", "minimum": 1, "maximum": 65536}, + "maxTotalWorkers": {"type": "integer", "minimum": 1, "maximum": 1048576}, + "runnableWorkers": {"type": "integer", "minimum": 1, "maximum": 1048576} + } +}`) -// New creates the pool plugin. -func New(opts ...Option) *Plugin { - p := &Plugin{} - for _, opt := range opts { - opt(p) +func decodePluginConfig(raw json.RawMessage) (pluginConfig, Limits, error) { + if len(raw) == 0 { + raw = json.RawMessage(`{}`) } - return p -} - -func (*Plugin) Info() wago.ExtensionInfo { - return wago.ExtensionInfo{ - ID: "wago.pool", Name: "Pool", Version: "0.0.0", - Description: "Elastic, load-balanced, self-healing pools of Wago workers", - Stability: wago.Experimental, Repository: "https://github.com/JairusSW/pool", - License: "Apache-2.0", - Tags: []string{"pool", "workers", "load-balancing", "autoscaling", "supervision"}, - Compat: wago.Compatibility{Engines: map[string]string{"wago": ">=0.1.0"}}, - // The pool plugin uses no core capabilities directly; it composes the - // workers service, which holds the privileged instance capabilities. + if err := validateConfigObject(raw); err != nil { + return pluginConfig{}, Limits{}, fmt.Errorf("pool: config: %w", err) + } + var cfg pluginConfig + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.DisallowUnknownFields() + if err := dec.Decode(&cfg); err != nil { + return pluginConfig{}, Limits{}, fmt.Errorf("pool: config: %w", err) + } + if err := dec.Decode(new(any)); err != io.EOF { + return pluginConfig{}, Limits{}, fmt.Errorf("pool: config has a trailing JSON value") + } + limits := normalizeLimits(Limits{}) + if cfg.MaxPools != nil { + limits.MaxPools = *cfg.MaxPools + } + if cfg.MaxWorkersPerPool != nil { + limits.MaxWorkersPerPool = *cfg.MaxWorkersPerPool + } + if cfg.MaxTotalWorkers != nil { + limits.MaxTotalWorkers = *cfg.MaxTotalWorkers } + if cfg.RunnableWorkers != nil { + limits.RunnableWorkers = *cfg.RunnableWorkers + } + if limits.MaxPools == 0 || limits.MaxPools > 65536 || limits.MaxWorkersPerPool == 0 || limits.MaxWorkersPerPool > 65536 || + limits.MaxTotalWorkers == 0 || limits.MaxTotalWorkers > 1<<20 || limits.RunnableWorkers == 0 || limits.RunnableWorkers > 1<<20 || + limits.MaxWorkersPerPool > limits.MaxTotalWorkers { + return pluginConfig{}, Limits{}, fmt.Errorf("pool: config limits are invalid") + } + return cfg, limits, nil } -func (p *Plugin) Register(reg *wago.Registry) error { - p.service = newPools(p.workers, p.limits) - if p.workers == nil { - ref, err := plugin.Require(reg, workers.ServiceKey) +func validateConfigObject(raw json.RawMessage) error { + dec := json.NewDecoder(bytes.NewReader(raw)) + token, err := dec.Token() + if err != nil { + return err + } + if token != json.Delim('{') { + return fmt.Errorf("must be a JSON object") + } + seen := map[string]struct{}{} + for dec.More() { + keyToken, err := dec.Token() if err != nil { return err } - p.service.ref = ref + key, ok := keyToken.(string) + if !ok { + return fmt.Errorf("object key is not a string") + } + if _, duplicate := seen[key]; duplicate { + return fmt.Errorf("duplicate field %q", key) + } + seen[key] = struct{}{} + var value json.RawMessage + if err := dec.Decode(&value); err != nil { + return err + } + if bytes.Equal(bytes.TrimSpace(value), []byte("null")) { + return fmt.Errorf("field %q must not be null", key) + } } - return plugin.Provide(reg, ServiceKey, p.service) + if _, err := dec.Token(); err != nil { + return err + } + if err := dec.Decode(new(any)); err != io.EOF { + if err == nil { + return fmt.Errorf("has a trailing JSON value") + } + return err + } + return nil } -// Stop marks the service closed. Worker teardown is handled by the workers -// service on runtime shutdown (linked workers are drained), so Stop only stops -// the pool service from accepting new operations. -func (p *Plugin) Stop(context.Context) error { - if p == nil || p.service == nil { - return nil +func Definition() wago.PluginDefinition { + return wago.PluginDefinition{ + ID: PluginID, Name: "Pool", Version: "0.1.0", + Description: "Elastic, load-balanced, self-healing pools of Wago workers.", + Stability: wago.Experimental, + Compatibility: wago.Compatibility{Engines: map[string]string{"wago": ">=0.1.0"}}, + Provenance: wago.PluginProvenance{ + Homepage: "https://github.com/JairusSW/pool", Repository: "https://github.com/JairusSW/pool", License: "Apache-2.0", + Authors: []string{"Jairus Tanaka"}, + }, + Requires: []wago.PluginRequirement{{ID: workers.PluginID, Version: "^0.1.0"}}, + Consumes: []wago.ContractRequirement{{ID: workers.Contract.ID(), Major: workers.Contract.Major(), Mode: wago.ContractRequired}}, + Provides: []wago.ContractSpec{Contract.Spec()}, ConfigSchema: append(json.RawMessage(nil), configSchema...), } - p.service.close() - return nil } -// Service returns the Pools handle, or nil before Register. -func (p *Plugin) Service() *Pools { - if p == nil { - return nil +func Provider() wago.PluginProvider { + return wago.PluginProvider{ + Definition: Definition(), New: func() wago.Plugin { return new(plugin) }, + ValidateConfig: func(raw json.RawMessage) error { _, _, err := decodePluginConfig(raw); return err }, } - return p.service } -func init() { wago.RegisterExtension(PluginName, func() wago.Extension { return New() }) } +type plugin struct{ service *Pools } -// Pools is the host-side service: it creates and manages pools and routes tasks -// across their workers. Obtain it with Plugin.Service() or through ServiceKey. +func (p *plugin) Register(reg *wago.Registrar) error { + var cfg pluginConfig + if err := reg.Config(&cfg); err != nil { + return err + } + limits := normalizeLimits(Limits{}) + if cfg.MaxPools != nil { + limits.MaxPools = *cfg.MaxPools + } + if cfg.MaxWorkersPerPool != nil { + limits.MaxWorkersPerPool = *cfg.MaxWorkersPerPool + } + if cfg.MaxTotalWorkers != nil { + limits.MaxTotalWorkers = *cfg.MaxTotalWorkers + } + if cfg.RunnableWorkers != nil { + limits.RunnableWorkers = *cfg.RunnableWorkers + } + ref, err := wagoplugin.Require(reg, workers.Contract) + if err != nil { + return err + } + p.service = newPools(ref, limits) + if err := wagoplugin.Provide(reg, Contract, Service(p.service)); err != nil { + return err + } + return reg.Lifecycle(wago.PluginLifecycle{ + Start: func(context.Context) error { return p.service.start() }, + Stop: func(context.Context) error { return p.service.close() }, + }) +} + +// Pools implements Pool's typed Service Contract. Consumers access it only +// inside the callback passed to plugin.Ref.With. type Pools struct { - ref *plugin.Ref[*workers.Workers] + ref *wagoplugin.Ref[workers.Service] limits Limits wireMu sync.Mutex - workers *workers.Workers wired bool + message workers.Subscription + exit workers.Subscription - hasObs atomic.Bool - obsMu sync.RWMutex - obs []func(*PoolEvent) + hasObs atomic.Bool + obsMu sync.Mutex + nextObs uint64 + obs map[uint64]*eventObserver + observerPanics []error mu sync.Mutex pools map[PoolID]*pool @@ -418,10 +520,10 @@ type Pools struct { closed bool } -func newPools(explicit *workers.Workers, limits Limits) *Pools { +func newPools(ref *wagoplugin.Ref[workers.Service], limits Limits) *Pools { return &Pools{ - workers: explicit, limits: normalizeLimits(limits), next: 1, - pools: map[PoolID]*pool{}, byWorker: map[WorkerID]*pool{}, + ref: ref, limits: normalizeLimits(limits), next: 1, + pools: map[PoolID]*pool{}, byWorker: map[WorkerID]*pool{}, obs: map[uint64]*eventObserver{}, } } @@ -505,45 +607,154 @@ func (pl *pool) rand() uint64 { return x } -// ensureWired resolves the workers service (available after the plugin plan -// commits) and registers the pool's OnMessage/OnExit observers exactly once. It -// is safe to call on every public method and retries until the service resolves. -func (p *Pools) ensureWired() error { +// start resolves the exact reviewed Workers binding and owns both observer +// subscriptions before Pool becomes active. +func (p *Pools) start() error { p.wireMu.Lock() defer p.wireMu.Unlock() if p.wired { return nil } - svc := p.workers - if svc == nil { - if p.ref == nil { - return ErrPoolsInactive + if p.ref == nil { + return ErrPoolsInactive + } + if err := p.ref.With(func(svc workers.Service) error { + if svc == nil { + return ErrWorkersUnavailable } - got, err := p.ref.Get() + var err error + p.message, err = svc.ObserveMessages(func(ctx *workers.MessageContext) error { p.onMessage(ctx); return nil }) if err != nil { return err } - if got == nil { - return ErrWorkersUnavailable + p.exit, err = svc.ObserveExits(func(ctx *workers.WorkerExitContext) { p.onExit(ctx) }) + if err != nil { + _ = svc.Unsubscribe(p.message) + p.message = workers.Subscription{} + return err } - svc = got + return nil + }); err != nil { + return err } - svc.OnMessage(func(ctx *workers.MessageContext) error { p.onMessage(ctx); return nil }) - svc.OnExit(func(ctx *workers.WorkerExitContext) { p.onExit(ctx) }) - p.workers = svc p.wired = true return nil } -// OnEvent registers observers notified of pool lifecycle and routing events. -func (p *Pools) OnEvent(fns ...func(*PoolEvent)) { - if len(fns) == 0 { - return +func (p *Pools) ensureWired() error { + p.wireMu.Lock() + wired := p.wired + p.wireMu.Unlock() + if !wired { + return ErrPoolsInactive + } + return nil +} + +func (p *Pools) withWorkers(fn func(workers.Service) error) error { + if err := p.ensureWired(); err != nil { + return err + } + if p.ref == nil { + return ErrWorkersUnavailable + } + return p.ref.With(fn) +} + +// EventSubscription is an opaque observer token. Pass it back through +// Service.UnsubscribeEvents inside a leased contract call. +type EventSubscription struct{ id uint64 } + +type eventObserver struct { + id uint64 + fn func(*PoolEvent) + mu sync.Mutex + cond *sync.Cond + active bool + inFlight uint32 +} + +func newEventObserver(id uint64, fn func(*PoolEvent)) *eventObserver { + o := &eventObserver{id: id, fn: fn, active: true} + o.cond = sync.NewCond(&o.mu) + return o +} + +func (o *eventObserver) invoke(event *PoolEvent) (panicErr error) { + o.mu.Lock() + if !o.active { + o.mu.Unlock() + return nil + } + o.inFlight++ + o.mu.Unlock() + func() { + defer func() { + if recovered := recover(); recovered != nil { + panicErr = fmt.Errorf("pool: event observer %d panicked: %v", o.id, recovered) + } + }() + o.fn(event) + }() + o.mu.Lock() + o.inFlight-- + if o.inFlight == 0 { + o.cond.Broadcast() + } + o.mu.Unlock() + return panicErr +} + +func (o *eventObserver) stop() { + o.mu.Lock() + o.active = false + for o.inFlight != 0 { + o.cond.Wait() + } + o.mu.Unlock() +} + +// ObserveEvents registers an observer until its returned Subscription closes. +func (p *Pools) ObserveEvents(fn func(*PoolEvent)) (EventSubscription, error) { + if p == nil || fn == nil { + return EventSubscription{}, fmt.Errorf("pool: invalid event observer") + } + p.mu.Lock() + if p.closed { + p.mu.Unlock() + return EventSubscription{}, ErrPoolsClosed } p.obsMu.Lock() - p.obs = append(p.obs, fns...) - p.obsMu.Unlock() + p.nextObs++ + if p.nextObs == 0 { + p.obsMu.Unlock() + p.mu.Unlock() + return EventSubscription{}, fmt.Errorf("pool: observer ID space exhausted") + } + id := p.nextObs + observer := newEventObserver(id, fn) + p.obs[id] = observer p.hasObs.Store(true) + p.obsMu.Unlock() + p.mu.Unlock() + return EventSubscription{id: id}, nil +} + +// UnsubscribeEvents removes one observer and waits for callbacks already in +// flight. It is idempotent. Do not call it from inside that observer callback. +func (p *Pools) UnsubscribeEvents(subscription EventSubscription) error { + if p == nil || subscription.id == 0 { + return fmt.Errorf("pool: invalid event subscription") + } + p.obsMu.Lock() + observer := p.obs[subscription.id] + delete(p.obs, subscription.id) + p.hasObs.Store(len(p.obs) != 0) + p.obsMu.Unlock() + if observer != nil { + observer.stop() + } + return nil } // fire delivers ev to observers. It must never be called while holding p.mu, so @@ -553,11 +764,23 @@ func (p *Pools) fire(ev *PoolEvent) { if !p.hasObs.Load() { return } - p.obsMu.RLock() - fns := p.obs - p.obsMu.RUnlock() - for _, fn := range fns { - fn(ev) + p.obsMu.Lock() + ids := make([]uint64, 0, len(p.obs)) + for id := range p.obs { + ids = append(ids, id) + } + sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] }) + observers := make([]*eventObserver, 0, len(ids)) + for _, id := range ids { + observers = append(observers, p.obs[id]) + } + p.obsMu.Unlock() + for _, observer := range observers { + if panicErr := observer.invoke(ev); panicErr != nil { + p.obsMu.Lock() + p.observerPanics = append(p.observerPanics, panicErr) + p.obsMu.Unlock() + } } } @@ -652,15 +875,26 @@ func (p *Pools) grow(caller wago.HostModule, id PoolID, n uint32) (uint32, error // Reserve the global slot before forking so concurrent grows cannot // overshoot MaxTotalWorkers; released below on any failure to add. p.total++ - wopts, tableIndex, svc := pl.wopts, pl.tableIndex, p.workers + wopts, tableIndex := pl.wopts, pl.tableIndex p.mu.Unlock() - wid, err := svc.Spawn(caller, tableIndex, wopts) + var wid WorkerID + err := p.withWorkers(func(svc workers.Service) error { + var err error + wid, err = svc.Spawn(caller, tableIndex, wopts) + if err != nil { + return err + } + if err := svc.Link(caller, wid); err != nil { + _ = svc.Kill(wid) + return err + } + return nil + }) if err != nil { p.unreserve() return spawned, err } - _ = svc.Link(caller, wid) // tie the worker's lifetime to its creator p.mu.Lock() pl = p.pools[id] @@ -669,7 +903,7 @@ func (p *Pools) grow(caller wago.HostModule, id PoolID, n uint32) (uint32, error p.total-- // release: the member is never added } p.mu.Unlock() - _ = svc.Kill(wid) + _ = p.withWorkers(func(svc workers.Service) error { return svc.Kill(wid) }) return spawned, ErrPoolNotFound } p.addMemberLocked(pl, wid) @@ -772,9 +1006,17 @@ func (p *Pools) submit(id PoolID, key, tag uint64, payload []byte) (WorkerID, er } pl.submitted++ if pl.strategy == Broadcast { - ids, svc := pl.snapshotIDs(), p.workers + ids := pl.snapshotIDs() p.mu.Unlock() - p.broadcastTo(id, svc, ids, tag, payload) + var delivered int + err := p.withWorkers(func(svc workers.Service) error { + delivered = p.broadcastTo(id, svc, ids, tag, payload) + return nil + }) + if err != nil { + return 0, err + } + _ = delivered return 0, nil } ordered := pl.order(key) @@ -787,24 +1029,30 @@ func (p *Pools) submit(id PoolID, key, tag uint64, payload []byte) (WorkerID, er } else if len(ordered) > 0 { cands = []WorkerID{ordered[0].id} } - svc := p.workers + overflow := pl.overflow p.mu.Unlock() if len(cands) == 0 { p.recordRejected(id) return 0, ErrPoolEmpty } - for _, wid := range cands { - err := svc.Send(wid, tag, payload) - if err == nil { - p.recordRouted(id, wid) - return wid, nil + var accepted WorkerID + if err := p.withWorkers(func(svc workers.Service) error { + for _, wid := range cands { + if err := svc.Send(wid, tag, payload); err == nil { + accepted = wid + return nil + } } - // A full mailbox, or a worker that vanished mid-route, just means we try - // the next candidate; any other error is likewise non-fatal to the pool. - _ = err + return nil + }); err != nil { + return 0, err + } + if accepted != 0 { + p.recordRouted(id, accepted) + return accepted, nil } - if pl.overflow == OverflowShed { + if overflow == OverflowShed { p.recordShed(id) return 0, nil } @@ -892,12 +1140,17 @@ func (p *Pools) Broadcast(id PoolID, tag uint64, payload []byte) (int, error) { return 0, ErrPoolDraining } pl.submitted++ - ids, svc := pl.snapshotIDs(), p.workers + ids := pl.snapshotIDs() p.mu.Unlock() - return p.broadcastTo(id, svc, ids, tag, payload), nil + var delivered int + err := p.withWorkers(func(svc workers.Service) error { + delivered = p.broadcastTo(id, svc, ids, tag, payload) + return nil + }) + return delivered, err } -func (p *Pools) broadcastTo(id PoolID, svc *workers.Workers, ids []WorkerID, tag uint64, payload []byte) int { +func (p *Pools) broadcastTo(id PoolID, svc workers.Service, ids []WorkerID, tag uint64, payload []byte) int { delivered := 0 for _, wid := range ids { if err := svc.Send(wid, tag, payload); err == nil { @@ -975,11 +1228,17 @@ func (p *Pools) Reconcile(caller wago.HostModule, id PoolID) error { if !pl.draining && desired > live { deficit = desired - live } - svc := p.workers p.mu.Unlock() - for _, wid := range toKill { - _ = svc.Kill(wid) + if len(toKill) != 0 { + if err := p.withWorkers(func(svc workers.Service) error { + for _, wid := range toKill { + _ = svc.Kill(wid) + } + return nil + }); err != nil { + return err + } } if deficit > 0 { if _, err := p.grow(caller, id, deficit); err != nil { @@ -1089,10 +1348,14 @@ func (p *Pools) Drain(id PoolID) error { m.killWhenIdle = true } } - svc := p.workers p.mu.Unlock() - for _, wid := range killNow { - _ = svc.Kill(wid) + if err := p.withWorkers(func(svc workers.Service) error { + for _, wid := range killNow { + _ = svc.Kill(wid) + } + return nil + }); err != nil { + return err } p.fire(&PoolEvent{Kind: PoolDrained, Pool: id}) return nil @@ -1119,10 +1382,14 @@ func (p *Pools) Destroy(id PoolID) error { p.total-- } } - svc := p.workers p.mu.Unlock() - for _, wid := range ids { - _ = svc.Kill(wid) + if err := p.withWorkers(func(svc workers.Service) error { + for _, wid := range ids { + _ = svc.Kill(wid) + } + return nil + }); err != nil { + return err } p.fire(&PoolEvent{Kind: PoolDestroyed, Pool: id}) return nil @@ -1177,10 +1444,9 @@ func (p *Pools) onMessage(ctx *workers.MessageContext) { } } } - svc := p.workers p.mu.Unlock() if kill { - _ = svc.Kill(ctx.WorkerID) + _ = p.withWorkers(func(svc workers.Service) error { return svc.Kill(ctx.WorkerID) }) } } @@ -1243,10 +1509,41 @@ func (p *Pools) applySupervisionLocked(pl *pool, kind workers.WorkerExitKind) { } } -func (p *Pools) close() { +func (p *Pools) close() error { p.mu.Lock() + if p.closed { + p.mu.Unlock() + return nil + } p.closed = true + p.mu.Unlock() + p.wireMu.Lock() + message, exit := p.message, p.exit + p.message, p.exit = workers.Subscription{}, workers.Subscription{} + p.wired = false + p.wireMu.Unlock() + var unsubscribeErr error + if p.ref != nil { + unsubscribeErr = p.ref.With(func(service workers.Service) error { + return errors.Join(service.Unsubscribe(message), service.Unsubscribe(exit)) + }) + } + p.obsMu.Lock() + observers := make([]*eventObserver, 0, len(p.obs)) + for _, observer := range p.obs { + observers = append(observers, observer) + } + p.obs = nil + p.hasObs.Store(false) + panicErrs := append([]error(nil), p.observerPanics...) + p.observerPanics = nil + p.obsMu.Unlock() + for _, observer := range observers { + observer.stop() + } + p.mu.Lock() p.pools = map[PoolID]*pool{} p.byWorker = map[WorkerID]*pool{} p.mu.Unlock() + return errors.Join(append([]error{unsubscribeErr}, panicErrs...)...) } diff --git a/register/catalog.go b/register/catalog.go new file mode 100644 index 0000000..738376f --- /dev/null +++ b/register/catalog.go @@ -0,0 +1,12 @@ +// Package register exposes Pool's explicit provider catalog to generated Wago +// runtimes. Importing this package has no registration side effects. +package register + +import ( + "github.com/JairusSW/pool" + "github.com/wago-org/wago" +) + +func Providers() []wago.PluginProvider { + return []wago.PluginProvider{pool.Provider()} +} diff --git a/register/catalog_test.go b/register/catalog_test.go new file mode 100644 index 0000000..cae5760 --- /dev/null +++ b/register/catalog_test.go @@ -0,0 +1,18 @@ +package register + +import ( + "testing" + + "github.com/JairusSW/pool" +) + +func TestProvidersIsExplicitAndFresh(t *testing.T) { + first, second := Providers(), Providers() + if len(first) != 1 || first[0].Definition.ID != pool.PluginID { + t.Fatalf("Providers() = %#v", first) + } + first[0].Definition.ID = "example.com/mutated" + if second[0].Definition.ID != pool.PluginID { + t.Fatal("provider catalog shares mutable definition state") + } +} diff --git a/register/register.go b/register/register.go deleted file mode 100644 index fa17101..0000000 --- a/register/register.go +++ /dev/null @@ -1,15 +0,0 @@ -// Package register activates the pool plugin's init-time registration for -// Wago-generated hosts. Applications normally import github.com/JairusSW/pool -// directly; generated hosts blank-import this package. -// -// import _ "github.com/JairusSW/pool/register" -// -// The pool plugin builds on the workers plugin, so a host that registers pool -// must also register workers. Blank-importing workers/register here guarantees -// the dependency is present in the build; the runtime enforces load ordering. -package register - -import ( - _ "github.com/JairusSW/pool" - _ "github.com/wago-org/workers/register" -) diff --git a/register/register_test.go b/register/register_test.go deleted file mode 100644 index aaa7f89..0000000 --- a/register/register_test.go +++ /dev/null @@ -1,22 +0,0 @@ -package register - -import ( - "testing" - - wago "github.com/wago-org/wago" -) - -// Importing this package must register the pool plugin and its workers -// dependency into the engine's global plugin registry as a side effect. -func TestRegistersPoolAndWorkers(t *testing.T) { - ext, ok := wago.NewExtension("JairusSW/pool") - if !ok { - t.Fatal("pool extension was not registered") - } - if got := ext.Info().ID; got != "wago.pool" { - t.Fatalf("pool extension ID = %q, want wago.pool", got) - } - if _, ok := wago.NewExtension("wago-org/workers"); !ok { - t.Fatal("workers dependency was not registered") - } -} diff --git a/stress_test.go b/stress_test.go index 372cc21..bbdbdbb 100644 --- a/stress_test.go +++ b/stress_test.go @@ -1,10 +1,14 @@ package pool import ( + "errors" "sync" "sync/atomic" "testing" "time" + + "github.com/wago-org/wago" + "github.com/wago-org/workers" ) // bigMailbox gives workers deep mailboxes so a stress burst never overflows and @@ -290,15 +294,28 @@ func TestStressRuntimeCloseWhileBusy(t *testing.T) { } }() } - time.Sleep(5 * time.Millisecond) - stop.Store(true) - wg.Wait() - done := make(chan struct{}) go func() { r.close(); close(done) }() select { case <-done: case <-time.After(10 * time.Second): + stop.Store(true) + wg.Wait() t.Fatal("runtime close hung while busy") } + // Keep producers racing with shutdown itself: close must revoke admission, + // drain accepted work, and return without requiring callers to stop first. + stop.Store(true) + wg.Wait() + if err := r.d.poolRef.With(func(Service) error { return nil }); !errors.Is(err, wago.ErrPermissionDenied) { + t.Fatalf("pool contract after close = %v", err) + } + if err := r.d.workers.With(func(workers.Service) error { return nil }); !errors.Is(err, wago.ErrPermissionDenied) { + t.Fatalf("Workers contract after close = %v", err) + } + messages := r.d.messages() + time.Sleep(10 * time.Millisecond) + if got := r.d.messages(); got != messages { + t.Fatalf("late callback after close: messages changed from %d to %d", messages, got) + } } diff --git a/wago.json b/wago.json index fef2c22..b6c909b 100644 --- a/wago.json +++ b/wago.json @@ -1,17 +1,20 @@ { - "$schema": "https://wago.sh/v0/schema.json", - "module": "github.com/JairusSW/pool", - "name": "Pool", - "version": "0.0.0", - "description": "Elastic, load-balanced, self-healing pools of Wago workers: create, manage, and distribute work across fleets of WebAssembly workers.", - "license": "Apache-2.0", - "homepage": "https://github.com/JairusSW/pool", - "repository": "https://github.com/JairusSW/pool", - "authors": ["JairusSW"], - "keywords": ["pool", "worker-pool", "workers", "load-balancing", "load-balancer", "autoscaling", "elastic", "supervision", "fleet", "dispatch", "concurrency", "webassembly", "wasm"], - "category": "concurrency", - "engines": { "wago": ">=0.1.0" }, + "$schema": "https://wago.sh/v1/schema.json", + "package": { + "module": "github.com/JairusSW/pool", + "version": "0.1.0", + "name": "Pool", + "description": "Elastic, load-balanced, self-healing pools of Wago workers.", + "stability": "experimental", + "license": "Apache-2.0", + "homepage": "https://github.com/JairusSW/pool", + "repository": "https://github.com/JairusSW/pool", + "category": "concurrency", + "tags": ["pool", "workers", "load-balancing", "autoscaling", "supervision", "webassembly", "wasm"], + "authors": [{"name": "Jairus Tanaka", "github": "JairusSW"}], + "engines": {"wago": ">=0.1.0"} + }, "plugins": { - "wago-org/workers": "^0.0.0" + "github.com/wago-org/workers": "^0.1.0" } } diff --git a/wago.providers.json b/wago.providers.json new file mode 100644 index 0000000..8c59f94 --- /dev/null +++ b/wago.providers.json @@ -0,0 +1,74 @@ +{ + "$schema": "https://wago.sh/v1/providers.schema.json", + "providers": [ + { + "importPath": "github.com/JairusSW/pool/register", + "definition": { + "id": "github.com/JairusSW/pool", + "name": "Pool", + "version": "0.1.0", + "description": "Elastic, load-balanced, self-healing pools of Wago workers.", + "stability": "experimental", + "compatibility": { + "engines": { + "wago": "\u003e=0.1.0" + } + }, + "provenance": { + "homepage": "https://github.com/JairusSW/pool", + "repository": "https://github.com/JairusSW/pool", + "license": "Apache-2.0", + "authors": [ + "Jairus Tanaka" + ] + }, + "requires": [ + { + "id": "github.com/wago-org/workers", + "version": "^0.1.0" + } + ], + "configSchema": { + "additionalProperties": false, + "properties": { + "maxPools": { + "maximum": 65536, + "minimum": 1, + "type": "integer" + }, + "maxTotalWorkers": { + "maximum": 1048576, + "minimum": 1, + "type": "integer" + }, + "maxWorkersPerPool": { + "maximum": 65536, + "minimum": 1, + "type": "integer" + }, + "runnableWorkers": { + "maximum": 1048576, + "minimum": 1, + "type": "integer" + } + }, + "type": "object" + }, + "provides": [ + { + "id": "github.com/JairusSW/pool/service", + "major": 1 + } + ], + "consumes": [ + { + "id": "github.com/wago-org/workers/service", + "major": 1, + "mode": "required" + } + ] + }, + "definitionDigest": "sha256:7b35e7dbafbeb8bc6c89f9171b70798fb5fd096d9ee8cd7014d5a39fcee7d9e0" + } + ] +} diff --git a/wiring_test.go b/wiring_test.go deleted file mode 100644 index 8fd6e69..0000000 --- a/wiring_test.go +++ /dev/null @@ -1,76 +0,0 @@ -package pool - -import ( - "strings" - "testing" - - "github.com/wago-org/wago" - _ "github.com/wago-org/workers/register" // globally register the workers plugin -) - -// TestPlanPathBindsWorkersService exercises the production wiring path: under -// LoadPlugins the runtime resolves workers.ServiceKey and binds it into the pool -// plugin automatically, with no explicit WithWorkers. A successful load proves -// the service dependency and its type match end-to-end. -func TestPlanPathBindsWorkersService(t *testing.T) { - rt := wago.NewRuntime() - defer rt.Close() - err := rt.LoadPlugins([]wago.PluginConfig{ - {Name: "wago-org/workers", Capabilities: []wago.PluginCapability{ - wago.PluginManagedInstances, wago.PluginInstanceHooks, - }}, - {Name: "JairusSW/pool"}, - }) - if err != nil { - t.Fatalf("LoadPlugins(workers, pool): %v", err) - } -} - -// TestPlanPathMissingWorkersFails confirms pool declares a hard dependency: loading -// it without a workers provider is a resolve error, not a silent half-wiring. -func TestPlanPathMissingWorkersFails(t *testing.T) { - rt := wago.NewRuntime() - defer rt.Close() - err := rt.LoadPlugins([]wago.PluginConfig{{Name: "JairusSW/pool"}}) - if err == nil { - t.Fatal("LoadPlugins(pool alone) succeeded; want a missing-provider error") - } - if !strings.Contains(err.Error(), "wago.workers/v1") { - t.Fatalf("error %q does not mention the required workers service", err) - } -} - -func TestEnsureWiredErrors(t *testing.T) { - // No explicit workers and no service ref: the service is inactive. - p := newPools(nil, Limits{}) - if err := p.ensureWired(); err != ErrPoolsInactive { - t.Fatalf("ensureWired(no workers, no ref) = %v, want ErrPoolsInactive", err) - } - // Public operations surface the same error rather than panicking. - if _, err := p.Submit(1, 0, nil); err != ErrPoolsInactive { - t.Fatalf("Submit on inactive service = %v, want ErrPoolsInactive", err) - } -} - -func TestPoolErrorStrings(t *testing.T) { - for _, e := range []poolError{ - ErrPoolNotFound, ErrPoolBackpressure, ErrPoolDraining, ErrInvalidPoolOptions, - ErrPoolWorkerLimit, ErrPoolsClosed, - } { - if e.Error() == "" { - t.Fatalf("empty error string for sentinel %v", string(e)) - } - } -} - -func TestSeedForHonorsExplicitSeed(t *testing.T) { - if got := (PoolOptions{HashSeed: 42}).seedFor(7); got != 42 { - t.Fatalf("explicit seed = %d, want 42", got) - } - // Derived seed is stable per pool ID and differs between IDs. - var opts PoolOptions - a, b := opts.seedFor(1), opts.seedFor(2) - if a == b || a != opts.seedFor(1) { - t.Fatalf("derived seeds not stable/distinct: %d %d", a, b) - } -}