feat(installer): public Python API for the bundled catalogue and skill folders

.github/workflows/ci.yml

@@ -47,6 +47,9 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+      # uv builds the installer wheel for the slow wheel tests in
+      # tests/installer/ (they skip without it).
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
       - name: Install dependencies
         # tests/installer/ imports robotframework_plugins; install it from the
         # in-repo package so the test collector can resolve it. The

installer/CHANGELOG.md

@@ -9,6 +9,8 @@ The installer has calendar versions (`YYYY.M.PATCH`), independent of the plugins
 - **The marketplace is `robotframework-plugins`.** `--mode plugin` writes the new marketplace entry and source, and re-installing replaces an old `robotframework-agentskills` entry, keeping its `--ref` pin.
 
 ### Added
+- **A public Python API, `robotframework_plugins.bundle`** ([#5](https://github.com/MarketSquare/robotframework-plugins/issues/5)), so other tools such as rf-mcp can depend on this package and serve the bundled skills: `catalog()` (the bundled plugins with their libraries and supported Robot Framework range), `skills_dirs()` (each active plugin's `skills/` folder, also when the package is imported from a zip) and `CATALOG_FORMAT` (`1`). `fetch(ref="stable")` returns the same for a branch, tag or commit of the repository, downloaded once into the user cache and checked before use; it is the only part of the package that uses the network, and only when called. See "Python API" in the README for the compatibility promise.
+- **`catalog.json` carries `"format": 1`**, and each entry its `robotframework` range.
 - **Several plugins.** The installer bundles every listed plugin (`rf-core`, `rf-web`, `rf-api`, `rf-mobile`, `rf-desktop` and the `rf-agentskills` bundle). By default it installs `rf-core` plus the plugins for the libraries the project declares in `pyproject.toml` or `requirements*.txt`, or every verified plugin when it declares none. `--plugins <csv>|all` chooses them; an unknown or removed plugin is an error, a deprecated one a warning.
 - **`robotframework-plugins catalog [--json]`** lists the bundled plugins with version, tier and status. `version` prints every bundled plugin's version.
 - **Upgrades keep what is installed.** On an upgrade, `install` covers the plugins already installed and adds the ones the project now declares. The summary names the skills and plugins an install removed, including a plugin the catalogue no longer has; files you changed are kept and reported.

installer/README.md

@@ -90,9 +90,52 @@ robotframework-plugins doctor               # what works and what needs your att
 robotframework-plugins version              # this version and the bundled plugin versions
 ```
 
+## Python API
+
+Other tools can depend on `robotframework-plugins` and read the plugins it bundles, for example to serve the official skills from an MCP server. The public module is `robotframework_plugins.bundle`:
+
+| Name | What it gives you |
+|---|---|
+| `CATALOG_FORMAT` | The catalogue format, `1`. Check it at runtime. |
+| `catalog()` | The bundled plugins (active and deprecated): `name`, `version`, `status`, `tier`, `summary`, `libraries` (`name`, `package`, `versions`), `robotframework`, `dependencies`, `bundle`, `bundled`. A new list on each call. |
+| `skills_dirs()` | `{plugin: Path}` of each active plugin's `skills/` folder, `rf-core` first; bundles such as `rf-agentskills` are left out. The folders exist for the life of the process. |
+| `fetch(ref="stable")` | The same for a branch, tag or commit of this repository: a `Snapshot` with `ref`, `commit`, `catalog()` and `skills_dirs()`. Raises `FetchError` on any failure. |
+
+```python
+from packaging.specifiers import SpecifierSet
+from robotframework_plugins import bundle
+
+assert bundle.CATALOG_FORMAT == 1
+
+# Which plugins match the libraries a project has installed?
+installed = {"robotframework-browser": "20.1.0"}
+for plugin in bundle.catalog():
+    for lib in plugin["libraries"]:
+        if lib["package"] in installed:
+            print(plugin["name"], lib["name"], installed[lib["package"]] in SpecifierSet(lib["versions"]))
+
+# Skill folders, each holding <skill>/SKILL.md and references/
+skills = bundle.skills_dirs()
+
+# Optional: newer content from GitHub, with the bundled copy as the fallback
+try:
+    skills = bundle.fetch("main").skills_dirs()
+except bundle.FetchError:
+    pass
+```
+
+**Compatibility.** The names above are public. Removing or renaming one, or removing or changing the meaning of a listed catalogue field, bumps `CATALOG_FORMAT` and is named in the [changelog](CHANGELOG.md). New fields and new plugins don't. The skill folder layout (`<skills dir>/<skill>/SKILL.md`, with `references/` and other files next to it) follows the [Agent Skills format](https://agentskills.io) and is part of the contract. The calendar version moves every month, so pin a range such as `robotframework-plugins>=2026.11,<2027` and rely on `CATALOG_FORMAT`. Everything else in the package, including `robotframework_plugins._assets`, is private and can change in any release.
+
+**`fetch()` and the network.** `fetch()` is the only part of the API that uses the network, and only when you call it: importing the module, `catalog()` and `skills_dirs()` read the installed package. It resolves the ref to a commit on GitHub, downloads that commit's archive once, checks it (only plain files and folders, no paths outside the snapshot, a valid `catalog.yaml` and `plugin.json` for every plugin), and keeps it in `~/.cache/robotframework-plugins/snapshots/<commit>/` (`%LOCALAPPDATA%` on Windows; `cache_dir=` changes it). A commit SHA that is already cached needs no network. Nothing in a snapshot is run.
+
+Choose the ref deliberately:
+
+- `stable` holds released plugins. It only exists after the first plugin release; until then, `fetch()` raises `FetchError: ref 'stable' not found`.
+- `main` holds everything merged: reviewed and checked by CI, but not released. Plugin versions there can be ahead of any release, the full evaluations run later, and a change can still be reverted. Skills ship scripts that agents run, so serve `main` only where that is acceptable, and report `Snapshot.commit` so you know what you served.
+
 ## No telemetry
 
-The installer sends nothing anywhere. It only touches the network for the marketplace and package operations you ask for.
+The installer sends nothing anywhere. It only touches the network for the marketplace and package operations you ask for. The `bundle.fetch()` function of the Python API downloads from GitHub only when a program calls it; the installer's commands never do.
 
 ## More
 

installer/hatch_build.py

@@ -20,15 +20,14 @@
 
 from __future__ import annotations
 
-import json
+import importlib.util
 import shutil
+import sys
 from pathlib import Path
 from typing import Any
 
-import yaml
 from hatchling.builders.hooks.plugin.interface import BuildHookInterface
 
-
 # The path layout assumed:
 #   <repo-root>/plugins/<name>/                ← canonical sources
 #   <repo-root>/installer/                     ← this package; hatch_build.py lives here
@@ -39,14 +38,28 @@
 # level up.
 PLUGINS_REL = Path("..") / "plugins"
 ASSETS_REL = Path("src") / "robotframework_plugins" / "_assets"
-#: Statuses the installer bundles; ``removed`` plugins stay out.
-BUNDLED_STATUSES = ("active", "deprecated")
 _IGNORE_ANYWHERE = shutil.ignore_patterns(
     "__pycache__", "*.pyc", ".pytest_cache", ".mypy_cache", ".ruff_cache", "*.egg-info",
 )
-#: Plugin tests and evals are for this repository's CI, not for users. Only a
-#: plugin's own top-level folders: skills ship example ``tests/`` folders too.
-_PLUGIN_ONLY = {"tests", "evals"}
+CATALOG_SRC_REL = Path("src") / "robotframework_plugins" / "_catalog.py"
+
+
+def _load_catalog_module():
+    """The package's ``_catalog`` module, loaded by path: the package isn't
+    importable while its own wheel is being built. ``bundle.fetch()`` builds
+    snapshot catalogues with the same code."""
+    path = Path(__file__).resolve().parent / CATALOG_SRC_REL
+    spec = importlib.util.spec_from_file_location("_robotframework_plugins_catalog", path)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+_catalog = _load_catalog_module()
+#: Re-exported for scripts and tests that read them from the hook.
+BUNDLED_STATUSES = _catalog.BUNDLED_STATUSES
+catalog_entries = _catalog.catalog_entries
 
 
 def plugin_ignore(plugin_root: Path):
@@ -56,44 +69,12 @@ def plugin_ignore(plugin_root: Path):
     def ignore(directory: str, names: list[str]) -> set[str]:
         ignored = set(_IGNORE_ANYWHERE(directory, names))
         if Path(directory).resolve() == root:
-            ignored |= _PLUGIN_ONLY & set(names)
+            ignored |= _catalog.PLUGIN_ONLY_DIRS & set(names)
         return ignored
 
     return ignore
 
 
-def catalog_entries(plugins_dir: Path) -> list[dict[str, Any]]:
-    """One entry per bundled plugin, from its catalog.yaml and plugin.json."""
-    entries = []
-    for d in sorted(p for p in plugins_dir.iterdir() if p.is_dir()):
-        manifest_path = d / "catalog.yaml"
-        if not manifest_path.is_file():
-            raise FileNotFoundError(f"{manifest_path} missing; run scripts/build-catalog.py --check")
-        m = yaml.safe_load(manifest_path.read_text(encoding="utf-8"))
-        if m.get("status") not in BUNDLED_STATUSES:
-            continue
-        pj = json.loads((d / ".claude-plugin" / "plugin.json").read_text(encoding="utf-8"))
-        deps = [x if isinstance(x, str) else x["name"] for x in pj.get("dependencies", [])]
-        entry = {
-            "name": d.name,
-            "version": pj.get("version"),
-            "tier": m["tier"],
-            "status": m["status"],
-            "summary": m["summary"],
-            "libraries": m.get("libraries", []),
-            "dependencies": deps,
-            # A bundle ships a copy of other plugins (catalog.yaml ``bundle``) and
-            # no content of its own; file mode installs those plugins instead.
-            "bundle": bool(m.get("bundle")) and not _has_content(d),
-            "bundled": list(m.get("bundle", [])),
-        }
-        for key in ("replacement", "deprecated_since"):
-            if key in m:
-                entry[key] = str(m[key])
-        entries.append(entry)
-    return entries
-
-
 class CustomBuildHook(BuildHookInterface):  # type: ignore[misc]
     PLUGIN_NAME = "custom"
 
@@ -136,21 +117,12 @@ def initialize(self, version: str, build_data: dict[str, Any]) -> None:
         for entry in entries:
             src = plugins_src / entry["name"]
             shutil.copytree(src, assets_dst / "plugins" / entry["name"], ignore=plugin_ignore(src))
-        (assets_dst / "catalog.json").write_text(
-            json.dumps({"plugins": entries}, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
-        )
+        (assets_dst / "catalog.json").write_text(_catalog.catalog_document(entries), encoding="utf-8")
         self.app.display_info(
             f"robotframework-plugins: bundled {', '.join(e['name'] for e in entries)} → {assets_dst} "
             f"({_count_files(assets_dst)} files)"
         )
 
 
-def _has_content(plugin_dir: Path) -> bool:
-    """Ships a skill, a subagent or a hook."""
-    return (any((s / "SKILL.md").is_file() for s in (plugin_dir / "skills").glob("*"))
-            or any((plugin_dir / "agents").glob("*.md"))
-            or (plugin_dir / "hooks" / "hooks.json").is_file())
-
-
 def _count_files(root: Path) -> int:
     return sum(1 for p in root.rglob("*") if p.is_file())

installer/src/robotframework_plugins/_catalog.py

@@ -0,0 +1,123 @@
+"""Build catalogue entries from plugin folders.
+
+Shared by the wheel's build hook (``hatch_build.py`` loads this file by path,
+because the package isn't importable at build time) and by ``bundle.fetch()``,
+which builds the catalogue of a downloaded repository snapshot. Both must
+produce the same entries, so the rules live here once.
+
+Only the standard library and PyYAML (a package dependency) are used.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+#: Version of the catalogue's shape (``catalog.json`` ``format``). Bumped only
+#: on a breaking change to the documented entry fields; see the README's
+#: "Python API" section.
+CATALOG_FORMAT = 1
+#: Statuses that are bundled; ``removed`` plugins stay out.
+BUNDLED_STATUSES = ("active", "deprecated")
+#: A plugin's own top-level folders that are for this repository's CI, not for
+#: users. Skills ship example ``tests/`` folders too, so only the plugin root.
+PLUGIN_ONLY_DIRS = frozenset({"tests", "evals"})
+_REQUIRED_MANIFEST = ("tier", "status", "summary")
+
+
+class CatalogError(ValueError):
+    """A plugin folder whose manifest can't be turned into a catalogue entry."""
+
+
+def has_content(plugin_dir: Path) -> bool:
+    """Ships a skill, a subagent or a hook."""
+    return (any((s / "SKILL.md").is_file() for s in (plugin_dir / "skills").glob("*"))
+            or any((plugin_dir / "agents").glob("*.md"))
+            or (plugin_dir / "hooks" / "hooks.json").is_file())
+
+
+def _manifest(plugin_dir: Path) -> dict[str, Any]:
+    path = plugin_dir / "catalog.yaml"
+    if not path.is_file():
+        raise CatalogError(f"{plugin_dir.name}: catalog.yaml missing; run scripts/build-catalog.py --check")
+    try:
+        m = yaml.safe_load(path.read_text(encoding="utf-8"))
+    except yaml.YAMLError as exc:
+        raise CatalogError(f"{plugin_dir.name}: catalog.yaml is not valid YAML ({exc})") from exc
+    if not isinstance(m, dict):
+        raise CatalogError(f"{plugin_dir.name}: catalog.yaml is not a mapping")
+    missing = [k for k in _REQUIRED_MANIFEST if not m.get(k)]
+    if missing:
+        raise CatalogError(f"{plugin_dir.name}: catalog.yaml lacks {', '.join(missing)}")
+    return m
+
+
+def _plugin_json(plugin_dir: Path) -> dict[str, Any]:
+    path = plugin_dir / ".claude-plugin" / "plugin.json"
+    if not path.is_file():
+        raise CatalogError(f"{plugin_dir.name}: .claude-plugin/plugin.json missing")
+    try:
+        pj = json.loads(path.read_text(encoding="utf-8"))
+    except json.JSONDecodeError as exc:
+        raise CatalogError(f"{plugin_dir.name}: plugin.json is not valid JSON ({exc})") from exc
+    if not isinstance(pj, dict) or not pj.get("version"):
+        raise CatalogError(f"{plugin_dir.name}: plugin.json lacks version")
+    return pj
+
+
+def catalog_entries(plugins_dir: Path) -> list[dict[str, Any]]:
+    """One entry per bundled plugin under ``plugins_dir``, from its catalog.yaml and plugin.json."""
+    entries = []
+    for d in sorted(p for p in plugins_dir.iterdir() if p.is_dir()):
+        m = _manifest(d)
+        if m["status"] not in BUNDLED_STATUSES:
+            continue
+        pj = _plugin_json(d)
+        deps = [x if isinstance(x, str) else x["name"] for x in pj.get("dependencies", [])]
+        entry = {
+            "name": d.name,
+            "version": pj.get("version"),
+            "tier": m["tier"],
+            "status": m["status"],
+            "summary": m["summary"],
+            "libraries": m.get("libraries", []),
+            "robotframework": str(m.get("robotframework", "")),
+            "dependencies": deps,
+            # A bundle ships a copy of other plugins (catalog.yaml ``bundle``) and
+            # no content of its own; file mode installs those plugins instead.
+            "bundle": bool(m.get("bundle")) and not has_content(d),
+            "bundled": list(m.get("bundle", [])),
+        }
+        for key in ("replacement", "deprecated_since"):
+            if key in m:
+                entry[key] = str(m[key])
+        entries.append(entry)
+    return entries
+
+
+def catalog_document(entries: list[dict[str, Any]]) -> str:
+    """The ``catalog.json`` text for ``entries``."""
+    return json.dumps({"format": CATALOG_FORMAT, "plugins": entries}, indent=2, ensure_ascii=False) + "\n"
+
+
+#: The plugin listed first by ``skills_dirs()``: the others depend on it.
+CORE_PLUGIN = "rf-core"
+
+
+def skill_dirs(entries: list[dict[str, Any]], root: Path) -> dict[str, Path]:
+    """``{name: <root>/plugins/<name>/skills}`` for every active plugin in ``entries``
+    that isn't a bundle and has at least one ``*/SKILL.md``; ``rf-core`` first,
+    then catalogue order."""
+    names = [e["name"] for e in entries if e.get("status") == "active" and not e.get("bundle")]
+    if CORE_PLUGIN in names:
+        names.remove(CORE_PLUGIN)
+        names.insert(0, CORE_PLUGIN)
+    out: dict[str, Path] = {}
+    for name in names:
+        skills = root / "plugins" / name / "skills"
+        if any(skills.glob("*/SKILL.md")):
+            out[name] = skills.resolve()
+    return out