Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/rules/wire-ops.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ paths:
- `server/operations/base.py` defines `Operation`, `Role` and `grant_rank`; `server/transport/mcp_primer.py` renders the primer and answers `prompts/*` / `resources/*`; `server/observability/budgets.py::effective_budget_ms` resolves each op's latency budget (runtime and per-consumer overrides).
- Nothing below `server/transport/` imports it (`tests/unit/test_transport_layering.py`), and `package_contract_allowlist.txt` only shrinks: fix the import, never add an exception.
- Models in `server/<domain>/api/models_<group>.py` import no handler; the `Operation` and its `OPERATIONS` entry live in `ops_<group>.py`. `server/operations/registry.py` imports each list, and the handler in `server/<domain>/app/` imports its models at module top. Transports discover ops from the registry; a change that edits transport code to add an op is in the wrong place.
- After any op or field change: `python -m scripts.codegen.<gen>` for `sdk_py_gen`, `sdk_py_client_gen`, `sdk_ts_gen`, `docs_reference`, `op_reach_census`. Generated SDK files are never edited by hand.
- After any op or field change: `python -m scripts.codegen.<gen>` for `sdk_py_gen`, `sdk_py_client_gen`, `sdk_ts_gen`, `sdk_rb_gen`, `docs_reference`, `op_reach_census`. Generated SDK files are never edited by hand.
- Whether an agent calls `memory.set` or `memory.pin` is the model's judgement from the tool description (`server/transport/mcp_descriptions.py`), not a pattern match: change the wording, not a regex.
- A field description is what a model reads when it calls the tool: one sentence on what the value does, its unit and its default.
- The default MCP `tools/list` has a token ceiling (`test_mcp_listing_budget.py`); a new tool joins the default listing only when an agent needs it without asking.
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ Adding a new operation ([full walkthrough](./docs/development/adding-an-operatio
1. Define `<Op>Input` / `<Op>Output` Pydantic models in `server/<domain>/api/models_<group>.py`.
2. Write the handler in `server/<domain>/app/<group>.py`; it imports the models at module top.
3. Declare the `Operation` in `server/<domain>/api/ops_<group>.py` and append it to that module's `OPERATIONS`.
4. Regenerate the SDK op tables and the Python per-op client: `python scripts/codegen/sdk_py_gen.py`, `python scripts/codegen/sdk_py_client_gen.py` and `python scripts/codegen/sdk_ts_gen.py` (CI runs each with `--check`). Add a method to the hand-written client in `sdk/python/loom/<group>.py` when the op belongs in the high-level API.
4. Regenerate the SDK op tables and the Python per-op client: `python scripts/codegen/sdk_py_gen.py`, `python scripts/codegen/sdk_py_client_gen.py`, `python scripts/codegen/sdk_ts_gen.py` and `python scripts/codegen/sdk_rb_gen.py` (each has a `--check`). Add a method to the hand-written client in `sdk/python/loom/<group>.py` when the op belongs in the high-level API.
5. Add unit tests, and an integration test under `tests/integration/` for a new read or write path.

The HTTP and MCP transports discover registered operations; you don't touch transport code.
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,7 @@ lint:
$(PYTHON) scripts/codegen/sdk_py_client_gen.py --check
$(PYTHON) scripts/codegen/sdk_py_admin_gen.py --check
$(PYTHON) scripts/codegen/sdk_ts_gen.py --check
$(PYTHON) scripts/codegen/sdk_rb_gen.py --check
$(PYTHON) scripts/codegen/agent_plugin_sync.py --check
$(PYTHON) scripts/codegen/settings_reference.py --check
$(PYTHON) scripts/codegen/settings_schema.py --check
Expand Down
1 change: 1 addition & 0 deletions docs/development/adding-an-operation.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ If the leakage suite cannot synthesize a body for your op, it reports it; add a
python scripts/codegen/sdk_py_gen.py # sdk/python/loom/ops.py
python scripts/codegen/sdk_py_client_gen.py # sdk/python/loom/client.py
python scripts/codegen/sdk_ts_gen.py # sdk/typescript/src/generated/ops.ts
python scripts/codegen/sdk_rb_gen.py # sdk/ruby/lib/clickhouse/loom/generated/*.rb
python scripts/codegen/op_reach_census.py --write-manifest # server/operations/reach_manifest.py
python scripts/codegen/docs_reference.py # generated regions of docs/reference/*.md
make format
Expand Down
1 change: 1 addition & 0 deletions docs/development/code-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ ruff formatting, an atomic write, and the write-or-check protocol).
| `sdk_py_client_gen.py` | `sdk/python/loom/client.py`: `OpsClient`, one method per op (`agent.client.tracker.source_add`), typed with the `ops.py` names | `server.operations.registry.ALL_OPERATIONS` | yes |
| `sdk_py_admin_gen.py` | `sdk/python/loom/admin_routes.py`: `AdminRoutes`, one method per `/v1/admin/*` route (`agent.deployment.routes.settings_set`) | the `/v1/admin/*` routers: each route's method, path, gate and docstring | yes |
| `sdk_ts_gen.py` | `sdk/typescript/src/generated/ops.ts`: one interface per model, the `OPS` table, the `OpMap` type | `server.operations.registry.ALL_OPERATIONS` (the models' JSON Schemas) | yes |
| `sdk_rb_gen.py` | `sdk/ruby/lib/clickhouse/loom/generated/`: `ops.rb` (the `OPS` table, one class per op group, one method per op) and `errors.rb` (one `Error` subclass per server code) | `server.operations.registry.ALL_OPERATIONS`; the error codes as `docs_reference.py` collects them | yes |
| `agent_plugin_sync.py` | the Codex plugin's copies of the hook scripts (`hooks/*.mjs`), the shared library (`lib/*.mjs`), the skills and the listing icon (`assets/`), byte-identical with the executable bit | `integrations/claude-code-plugin/` | yes |
| `settings_reference.py` | the generated [configuration reference](../reference/configuration.md) | `server/settings/catalog.py`: each field's env var, type, default, and the comment above it | yes |
| `settings_schema.py` | `schemas/loom-settings.schema.json`, the JSON Schema of the YAML settings file | the settings fields' types, bounds, defaults and comments, secrets left out | yes |
Expand Down
292 changes: 292 additions & 0 deletions scripts/codegen/sdk_rb_gen.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,292 @@
"""Generate the Ruby SDK's operation table, its per-op methods and its error classes
from the server's registry (the twins are sdk_py_client_gen.py and sdk_ts_gen.py).

Two files under `sdk/ruby/lib/clickhouse/loom/generated/`:

- `ops.rb`: `OPS`, one `OpSpec` per served op (idempotency, read-only, role,
maturity, the input's field names and the required ones), and one class per op
group with one method per op. `client.<group>.<method>(**fields)` posts
`/v1/<group>.<rest>`: the group is the op name's first segment and the method is
the rest with `.` turned into `_`, as in the Python SDK (`tracker.source.add` →
`client.tracker.source_add`). A method that would shadow a method every Ruby
object has takes a trailing underscore (`memory.inspect` → `memory.inspect_`).
- `errors.rb`: one `Error` subclass per server error code (`PINNED_CONFLICT` →
`ClickHouse::Loom::PinnedConflict`), from the same sources as the errors
reference (`docs_reference.py`), and `ERROR_CLASSES`, the code → class map
`Error.from_response` reads.

So a new server op or error code reaches the Ruby SDK by regeneration alone.

python scripts/codegen/sdk_rb_gen.py # write the files
python scripts/codegen/sdk_rb_gen.py --check # exit 1 when a file is stale (CI)
"""

from __future__ import annotations

import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path

ROOT = Path(__file__).resolve().parents[2]
if str(ROOT) not in sys.path: # run as a file
sys.path.insert(0, str(ROOT))

from scripts.codegen.common import GeneratedFile, OpFacts, pascal # noqa: E402

COMMAND = "python scripts/codegen/sdk_rb_gen.py"
OUT_DIR = ROOT / "sdk" / "ruby" / "lib" / "clickhouse" / "loom" / "generated"
HEADER = (
"# frozen_string_literal: true\n"
"\n"
"# GENERATED by scripts/codegen/sdk_rb_gen.py from {source} — do not edit.\n"
f"# Regenerate: {COMMAND} (CI: --check)\n"
)

#: The public methods of every Ruby object (`Object.public_instance_methods` on
#: Ruby 3.1 to 3.3). A generated method of one of these names would shadow it, so
#: irb or a test failure printing a group would make an HTTP call.
RUBY_OBJECT_METHODS = frozenset(
"__id__ __send__ class clone define_singleton_method display dup enum_for eql? equal? "
"extend freeze frozen? hash inspect instance_eval instance_exec instance_of? "
"instance_variable_defined? instance_variable_get instance_variable_set "
"instance_variables is_a? itself kind_of? method methods nil? object_id private_methods "
"protected_methods public_method public_methods public_send remove_instance_variable "
"respond_to? send singleton_class singleton_method singleton_methods tap then to_enum "
"to_s yield_self".split()
)

#: Methods the client defines by hand. A group of one of these names would be
#: hidden behind it; `search` is the one the client resolves itself (`client.search`
#: with no query is the `search.*` group).
CLIENT_METHODS = frozenset(
{"base_url", "call", "delete", "get", "identity", "op_group", "retry_policy", "set"}
)

#: Constants `ClickHouse::Loom` defines by hand, and the top-level ones its code names
#: unqualified (`Timeout::Error`, `JSON.parse`, ...): an error class of one of these
#: names would hide it inside the module, so it takes an `Error` suffix instead.
RESERVED_CONSTANTS = frozenset(
{
"CATEGORIES",
"Client",
"Configuration",
"DeleteResult",
"Error",
"Generated",
"Memory",
"OpGroup",
"OpSpec",
"RetryAfter",
"RetryPolicy",
"SearchHit",
"SearchResult",
"SetResult",
# top-level constants the hand-written code uses unqualified
"DidYouMean",
"Errno",
"Float",
"IOError",
"JSON",
"Net",
"OpenSSL",
"SecureRandom",
"SocketError",
"Struct",
"Time",
"Timeout",
"URI",
}
)

#: The keyword a generated method takes its transport options under
#: (`idempotency_key:`, `retry_policy:`); no op may have a field of this name.
OPTIONS_KEYWORD = "request_options"


def split_op(name: str) -> tuple[str, str]:
"""`tracker.source.add` → (`tracker`, `source_add`); `memory.inspect` → (`memory`,
`inspect_`)."""
group, _, rest = name.partition(".")
method = rest.replace(".", "_")
if method in RUBY_OBJECT_METHODS:
method += "_"
if not (group.isidentifier() and method.isidentifier() and group.islower()):
raise SystemExit(f"op {name!r} has no valid Ruby method name")
if group in RUBY_OBJECT_METHODS or group in CLIENT_METHODS:
raise SystemExit(f"op group {group!r} would shadow a method of the Ruby client")
return group, method


def _words(values: list[str]) -> str:
return "%w[" + " ".join(values) + "].freeze" if values else "[].freeze"


def _summary(schema: dict[str, object]) -> str:
text = str(schema.get("description") or "").strip().split("\n")[0]
return text if len(text) <= 160 else text[:157].rstrip() + "..."


def op_rows() -> list[tuple[OpFacts, list[str], list[str], str]]:
"""(facts, field names, required field names, summary) per served op, by name."""
from server.operations.registry import ALL_OPERATIONS

rows = []
for op in sorted(ALL_OPERATIONS, key=lambda o: o.name):
schema = op.input_model.model_json_schema()
fields = list(schema.get("properties") or {})
if OPTIONS_KEYWORD in fields:
raise SystemExit(f"op {op.name!r} has a field named {OPTIONS_KEYWORD!r}")
bad = [f for f in fields if not f.isidentifier()]
if bad:
raise SystemExit(
f"op {op.name!r} has fields that are not Ruby keyword arguments: {bad}"
)
required = [f for f in fields if f in set(schema.get("required") or [])]
rows.append((OpFacts.of(op), fields, required, _summary(schema)))
return rows


def build_ops() -> str:
rows = op_rows()
groups: dict[str, list[tuple[str, OpFacts, list[str], str]]] = defaultdict(list)
for facts, _fields, required, summary in rows:
group, method = split_op(facts.name)
if any(m == method for m, *_ in groups[group]):
raise SystemExit(f"two ops map to {group}.{method}")
groups[group].append((method, facts, required, summary))
out = [
HEADER.format(source="server.operations.registry.ALL_OPERATIONS"),
"module ClickHouse",
" module Loom",
" # Every served operation: its spec in OPS, and one method per op on the class",
" # of its group (`client.memory.set(...)` is `memory.set`).",
" module Generated",
f" OP_COUNT = {len(rows)}",
"",
" # The declared spec of every served op, by name; the HTTP path is /v1/<name>.",
" OPS = {",
]
for facts, fields, required, _summary in rows:
group, method = split_op(facts.name)
out.append(
f" {json.dumps(facts.name)} => OpSpec.new({json.dumps(facts.name)}, "
f"{json.dumps(group)}, {json.dumps(method)}, {str(facts.idempotent).lower()}, "
f"{str(facts.read_only).lower()}, {json.dumps(facts.role)}, "
f"{json.dumps(facts.maturity)}, {_words(fields)}, {_words(required)}),"
)
out += [" }.freeze", ""]
for group in sorted(groups):
out += _group_class(group, groups[group])
out += [" # The class of each op group, by name.", " GROUPS = {"]
out += [f" {json.dumps(g)} => {pascal(g)}Ops," for g in sorted(groups)]
out += [
" }.freeze",
"",
" # One accessor per op group on the client: `client.memory`, `client.task`, ...",
" module GroupAccessors",
]
for i, group in enumerate(sorted(groups)):
out += ([""] if i else []) + [
f" def {group}",
f" op_group({json.dumps(group)})",
" end",
]
out += [" end", " end", " end", "end"]
return "\n".join(out) + "\n"


def _group_class(group: str, ops: list[tuple[str, OpFacts, list[str], str]]) -> list[str]:
out = [f" # The `{group}.*` operations.", f" class {pascal(group)}Ops < OpGroup"]
for i, (method, facts, required, summary) in enumerate(ops):
traits = [facts.role, "read-only" if facts.read_only else "writes"]
if facts.idempotent and not facts.read_only:
traits.append("Idempotency-Key")
if facts.maturity != "stable":
traits.append(facts.maturity)
out += [""] if i else []
out.append(f" # `{facts.name}` ({', '.join(traits)}).")
if summary:
out.append(f" # {summary}")
if required:
out.append(f" # Required: {', '.join(required)}.")
out += [
f" def {method}(body = nil, {OPTIONS_KEYWORD}: nil, **fields)",
f" call_op({json.dumps(facts.name)}, body, fields, {OPTIONS_KEYWORD})",
" end",
]
return [*out, " end", ""]


def error_codes() -> list[tuple[str, str, int, str]]:
"""(code, class name, HTTP status or 0, summary) for every code the errors
reference lists, by code."""
from scripts.codegen.docs_reference import (
CLASS_BY_CODE,
_code_literals,
_typed_errors,
)

typed = _typed_errors()
codes = sorted({*typed, *_code_literals(), *CLASS_BY_CODE})
rows, taken = [], dict.fromkeys(RESERVED_CONSTANTS, "(hand-written)")
for code in codes:
name = pascal(code.lower())
if name in RESERVED_CONSTANTS:
name += "Error"
if not name.isidentifier() or not name[:1].isupper():
raise SystemExit(f"error code {code!r} has no valid Ruby class name")
if name in taken:
raise SystemExit(f"error codes {taken[name]!r} and {code!r} both map to {name}")
taken[name] = code
status, doc, _qual = typed.get(code, (0, "", ""))
rows.append((code, name, status, doc.strip().split("\n")[0]))
return rows


def build_errors() -> str:
rows = error_codes()
out = [
HEADER.format(source="the server's error codes (as docs/reference/errors.md)"),
"module ClickHouse",
" module Loom",
]
for code, name, status, doc in rows:
note = f"HTTP {status}. {doc}".strip() if status else "Server code"
out += [
f" # `{code}`: {note}",
f" class {name} < Error",
f" CODE = {json.dumps(code)}",
" end",
"",
]
out += [
f" # The Error subclass for each server code ({len(rows)} codes); a code not",
" # listed raises Error itself.",
" ERROR_CLASSES = {",
*[f" {json.dumps(code)} => {name}," for code, name, _s, _d in rows],
" }.freeze",
" end",
"end",
]
return "\n".join(out) + "\n"


TARGETS = (
(GeneratedFile(OUT_DIR / "ops.rb", COMMAND), build_ops),
(GeneratedFile(OUT_DIR / "errors.rb", COMMAND), build_errors),
)


def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
ap.add_argument("--check", action="store_true", help="exit 1 when a file is stale")
args = ap.parse_args(argv)
return max(target.sync(build(), check=args.check) for target, build in TARGETS)


if __name__ == "__main__":
raise SystemExit(main())
Loading