diff --git a/CHANGELOG.md b/CHANGELOG.md index 997e0c05..a6d7e6a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,14 +2,27 @@ ## [Unreleased] +### Added +- **Remote Package Index & Synchronization (`ebuild/packages/index_sync.py`).** + Downloads and validates central/mirror package repository indices into a local cache + (`~/.ebuild/index/`), caching full recipe definitions. Enforces HTTPS transport, + path-traversal sanitization (`^[a-zA-Z0-9_-]+$`), 10s socket timeouts, and 10MB response + size limits. Index synchronization supports air-gapped operation via `--offline` and `EBUILD_OFFLINE=1`; package archive fetching is not yet offline-gated. +- **Package Discovery & Multi-Source Search (`ebuild search`, `ebuild/packages/repository.py`).** + Search across local project recipes, system-shipped recipes, and cached remote indices. + Supports `--all`, `--json`, `--build-system`, and `--license` filters. +- **Index Synchronization Command (`ebuild update-index`).** + CLI command to refresh local package and recipe index caches from remote repositories. +- **Source-Ranked Recipe Precedence (`ebuild/packages/registry.py`).** + Enforces strict 3-tier precedence hierarchy during package resolution: project-local recipes (`./recipes/`) > system-shipped recipes (`recipes/`) > cached remote index recipes (`~/.ebuild/index/recipes/`), guaranteeing reproducible builds (§9.2) and ensuring project-level pins override remote definitions. +- **Stale Cached Recipe Pruning (`ebuild/packages/index_sync.py`, `ebuild update-index`).** + `ebuild update-index` automatically prunes stale cached `.yaml` and `.yml` recipes from `~/.ebuild/index/recipes/` that are absent from the newly synchronized remote package index. An index that yields no usable entries is treated as a delivery fault: nothing is pruned, the cached index is left in place, and the condition is reported. Surfaced the count of pruned recipes in CLI output and returned `SyncResult`. +- **Expanded Shipped Recipes Catalog (`recipes/`).** + Added 5 verified recipes with HTTPS release pins and SHA-256 integrity digests: + `cjson` (v1.7.18), `nanopb` (v0.4.9.1), `lvgl` (v9.2.2), `tinyusb` (v0.18.0), and `unity` (v2.6.1). + ### Fixed -- **`ebuild flash --tool openocd` split an image path containing a space into - extra arguments.** The path was interpolated straight into the `-c` - argument's `program verify reset exit` string; OpenOCD's own - Tcl interpreter re-parses that string and splits on unescaped whitespace, so - `ebuild flash "my firmware.bin"` sent OpenOCD `program my` as the filename. - The path is now wrapped in Tcl brace-grouping (`{...}`) so OpenOCD treats it - as one token regardless of embedded spaces (`ebuild/firmware/flash.py`). + - **A path containing a space produced a silently wrong `build.ninja`.** Paths were written into build statements unescaped, but Ninja ends the output list at the first unescaped `:` and splits on unescaped spaces. A build directory diff --git a/README.md b/README.md index 421a3ae7..3edd28a2 100644 --- a/README.md +++ b/README.md @@ -91,10 +91,10 @@ that each package's own build may already run parallel compile jobs, so a large [docs/dependency-management.md](docs/dependency-management.md#parallel-package-builds). Additional commands: `configure`, `install`, `add`, `list-packages`, -`pipeline`, `system`, `firmware`, `flash`, `new`, `generate-project`, -`generate-board`, `generate-boot`, `analyze`, `setup`, and the `repos` group -(`status`, `update`, `set-url`, `set-branch`, `link`, `unlink`). Run -`ebuild --help` for the full list. +`search`, `update-index`, `pipeline`, `system`, `firmware`, `flash`, `new`, +`generate-project`, `generate-board`, `generate-boot`, `analyze`, `setup`, +and the `repos` group (`status`, `update`, `set-url`, `set-branch`, `link`, +`unlink`). Run `ebuild --help` for the full list. `ebuild test` reports the counts the underlying runner printed, and reports none diff --git a/TASKS.md b/TASKS.md index b7702ab6..8e26262a 100644 --- a/TASKS.md +++ b/TASKS.md @@ -18,6 +18,7 @@ Status is one of: `todo`, `in-progress`, `blocked`, `review`, `done`. | ID | Task | Owner | Verified by | Evidence | |----|------|-------|-------------|----------| | T-001 | Make initramfs creation portable and self-contained | backend | independent reviewer | Focused archive tests: **5 passed, 1 skipped** (symlink creation unavailable on this Windows host). Independent `bsdtar` extraction validated hard-link identity and payload. Full Python suite: **288 passed, 2 skipped, 1 unrelated failure** in the pre-existing Windows Ninja path assertion, recorded as T-002. QEMU boot was not run on Windows. | +| T-003 | Address PR #111 review findings 1, 2, 3, 6, 7, 8 | backend | reviewer | Unit tests in `tests/unit/test_index_sync.py` (22 passed) verify keep-set filename matching, empty index floor and fallback status, .yml preservation and pruning, lack-of-URL recipe preservation, CLI prune reporting, and trailing newline in `CHANGELOG.md`. Static analysis with `ruff` on all PR-touched files reports 0 errors; `mypy` on modified packages reports 0 errors without suppressions. Full test suite passes 605 tests on Linux. | --- diff --git a/docs/architecture.md b/docs/architecture.md index 09ac7829..435aa340 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,74 +7,82 @@ ```mermaid graph TD subgraph CLI["ebuild CLI (Python)"] - CMD[cli/commands.py
18 commands] + CMD["cli/commands.py
CLI Commands"] end subgraph BUILD["Build Orchestrator"] - ORCH[build/orchestrator.py] - NINJA[build/ninja_backend.py] - TC[build/toolchain.py
5 predefined toolchains] + ORCH["build/dispatch.py
Backend Dispatcher"] + NINJA["build/ninja_backend.py
Ninja Backend"] + TC["build/toolchain.py
5 Predefined Toolchains"] end subgraph PACKAGES["Package Pipeline"] - RECIPE[packages/recipe.py] - REG[packages/registry.py] - RESOLVE[packages/resolver.py] - FETCH[packages/fetcher.py] - BUILDER[packages/builder.py] - CACHE[packages/cache.py] - LOCK[packages/lockfile.py] - REPO[packages/repository.py
Remote index] - PROFILES[packages/profiles.py
Build profiles] + SYNC["packages/index_sync.py
Remote Index Sync"] + REPO["packages/repository.py
Package Discovery"] + RECIPE["packages/recipe.py
Recipe Schema"] + REG["packages/registry.py
Package Registry"] + RESOLVE["packages/resolver.py
Dependency Resolver"] + FETCH["packages/fetcher.py
Package Fetcher"] + BUILDER["packages/builder.py
Package Builder"] + CACHE["packages/cache.py
Build Cache"] + LOCK["packages/lockfile.py
Lockfile"] + PROFILES["packages/profiles.py
Build Profiles"] end subgraph HWAI["Hardware AI"] - ANALYZER[eos_ai/eos_hw_analyzer.py
MCU database + peripheral detection] - PROJGEN[eos_ai/eos_project_generator.py
Manifest + project scaffolding] + ANALYZER["eos_ai/eos_hw_analyzer.py
MCU & Peripheral Analysis"] + PROJGEN["eos_ai/eos_project_generator.py
Project & Config Generator"] end - subgraph CORE["Core Components (always built)"] - EOS[core/eos/
HAL, Kernel, Crypto, OTA, Drivers] - EBOOT[core/eboot/
Bootloader, 26 board ports] + subgraph CORE["Core Components (Native)"] + EOS["core/eos/
HAL, Kernel, Crypto, Drivers"] + EBOOT["core/eboot/
Bootloader, 26 Board Ports"] end - subgraph LAYERS["Optional Layers (--with flag)"] - EAI[layers/eai/
AI inference + LLM models] - ENI[layers/eni/
Neural interface] - EIPC[layers/eipc/
Secure IPC (Go + C)] - EOSUITE[layers/eosuite/
Dev tools + GUI apps] + subgraph LAYERS["Optional Platform Layers"] + EAI["layers/eai/
AI & Embedded Inference"] + ENI["layers/eni/
Neural Interface"] + EIPC["layers/eipc/
Secure IPC SDK"] + EOSUITE["layers/eosuite/
Developer Tools"] end subgraph HW["Hardware Intake"] - BOARD[hardware/board/
KiCad, Eagle, YAML, BOM] - SOC[hardware/soc/
Datasheets, TRMs] - BOOT[hardware/boot/
Image layout, boot flow] - SW[hardware/software/
Device trees, linker scripts] + BOARD["hardware/board/
KiCad, Eagle, YAML, BOM"] + SOC["hardware/soc/
Datasheets, TRMs"] + BOOT["hardware/boot/
Image Layout, Boot Flow"] + SW["hardware/software/
Device Trees, Linker Scripts"] end subgraph SDK_OUT["SDK Output"] - SDKGEN[sdk_generator.py] - SDKAPI[sdk/include/
Header-only API] + SDKGEN["sdk_generator.py"] + SDKAPI["sdk/include/
Header-only API"] end subgraph TEMPLATES["Project Templates"] - T1[bare-metal] - T2[rtos-app] - T3[linux-app] - T4[safety-critical] - T5[secure-boot] - T6[ble-sensor] + T1["bare-metal"] + T2["rtos-app"] + T3["linux-app"] + T4["safety-critical"] + T5["secure-boot"] + T6["ble-sensor"] end CMD --> ORCH CMD --> SDKGEN CMD --> ANALYZER CMD --> PROJGEN + CMD --> SYNC + CMD --> REPO + + SYNC --> REPO + REPO --> REG ORCH --> NINJA ORCH --> TC - ORCH --> CORE - ORCH --> LAYERS + ORCH --> EOS + ORCH --> EBOOT + ORCH --> EAI + ORCH --> EIPC ORCH --> RECIPE RECIPE --> REG @@ -83,16 +91,24 @@ graph TD FETCH --> BUILDER BUILDER --> CACHE RESOLVE --> LOCK - REG --> REPO - HW --> ANALYZER + BOARD --> ANALYZER + SOC --> ANALYZER + BOOT --> ANALYZER + SW --> ANALYZER + ANALYZER --> PROJGEN - PROJGEN --> TEMPLATES - PROJGEN --> CORE + PROJGEN --> T1 + PROJGEN --> T2 + PROJGEN --> T3 + PROJGEN --> EOS PROJGEN --> EBOOT - TC -->|cmake| CORE - TC -->|cmake| LAYERS + TC -->|cmake| EOS + TC -->|cmake| EAI + TC -->|cmake| EIPC + + SDKGEN --> SDKAPI ``` ## Subsystem Details @@ -157,14 +173,15 @@ recipe.yaml → Registry → Resolver → Fetcher → Builder → Cache ``` 1. **Recipe** (`recipe.py`) — YAML schema defining package name, version, URL, checksum, build system, dependencies -2. **Registry** (`registry.py`) — scans recipe directories and indexes available packages -3. **Repository** (`repository.py`) — remote package index for discovery and search -4. **Resolver** (`resolver.py`) — dependency resolution with version constraint solving -5. **Fetcher** (`fetcher.py`) — downloads and verifies source archives -6. **Builder** (`builder.py`) — builds packages using the specified build system -7. **Cache** (`cache.py`) — caches built artifacts to avoid rebuilding -8. **Lockfile** (`lockfile.py`) — records exact resolved versions for reproducibility -9. **Profiles** (`profiles.py`) — composable build profiles (minimal, standard, full, custom) +2. **Registry** (`registry.py`) — scans recipe directories (local, system, and cached remote) and indexes available packages +3. **Index Sync** (`index_sync.py`) — downloads, validates, and manages remote package repository indices and recipe caches (`~/.ebuild/index/`) with offline fallback +4. **Repository** (`repository.py`) — unified package discovery and multi-source search engine (`ebuild search`) +5. **Resolver** (`resolver.py`) — dependency resolution with version constraint solving +6. **Fetcher** (`fetcher.py`) — downloads and verifies source archives against SHA-256 integrity pins +7. **Builder** (`builder.py`) — builds packages using the specified build system (CMake, Make, Meson, autoconf) +8. **Cache** (`cache.py`) — caches built artifacts to avoid rebuilding +9. **Lockfile** (`lockfile.py`) — records exact resolved versions for reproducibility +10. **Profiles** (`profiles.py`) — composable build profiles (minimal, standard, full, custom) ### Hardware AI (`ebuild/eos_ai/`) diff --git a/docs/dependency-management.md b/docs/dependency-management.md index b9961b7b..a4588900 100644 --- a/docs/dependency-management.md +++ b/docs/dependency-management.md @@ -272,3 +272,100 @@ poles without thrashing; going much higher mostly adds memory pressure. Parallel builds also interleave package log lines. Output from each package is written atomically, but the packages themselves are no longer contiguous — use `-j 1` when reading a build log closely. + +--- + +## Remote Package Index & Discovery + +ebuild provides index-based package discovery and remote recipe synchronization, allowing embedded projects to discover, query, and install external libraries seamlessly. + +### Discovering Packages (`ebuild search`) + +Search across local project recipes (`./recipes/`), bundled system recipes, and cached remote repository indices: + +```bash +ebuild search json # Search by keyword in name, description, or license +ebuild search --all # List all available packages across all sources +ebuild search --json # Output machine-readable JSON array +ebuild search --build-system cmake # Filter by build system (cmake, make, meson) +ebuild search --license MIT # Filter by license type +``` + +Example output: + +``` +=== ebuild - Package Search === +[info] Found 10 package(s): + cjson v1.7.18 [cmake] (MIT) - Ultralightweight JSON parser in ANSI C + freertos v11.1.0 [cmake] (MIT) - Real-time operating system kernel for embedded devices + littlefs v2.9.3 [make] (BSD-3-Clause) - Little fail-safe filesystem designed for microcontrollers + lvgl v9.2.2 [cmake] (MIT) - Light and Versatile Embedded Graphics Library + lwip v2.2.0 [cmake] (BSD-3-Clause) - Lightweight TCP/IP stack for embedded systems + mbedtls v3.6.0 [cmake] (Apache-2.0) - Lightweight TLS/SSL library for embedded systems + nanopb v0.4.9.1 [cmake] (zlib) - Protocol Buffers with small code size for microcontrollers + tinyusb v0.18.0 [cmake] (MIT) - Open-source cross-platform USB host/device stack for embedded system + unity v2.6.1 [cmake] (MIT) - Simple Unit Testing for C + zlib v1.3.1 [cmake] (Zlib) - General-purpose lossless data compression library +``` + +### Synchronizing Remote Index (`ebuild update-index`) + +Refresh the local package index and cached recipe definitions from a remote repository mirror: + +```bash +ebuild update-index --url https://example.com/recipes/index.json # Remote repository URL (HTTPS required) +ebuild update-index --url https://example.com/recipes/index.json --force # Bypass 24h cache TTL and re-download +ebuild update-index --offline # Use local cached index without network +``` + +> [!WARNING] +> **Index Authenticity & Provenance Notice (Unauthenticated Index)**: +> Remote package index synchronization validates transport encryption (HTTPS only) and verifies individual package archive bytes against stated SHA-256 checksums. However, the index document itself is currently **unauthenticated** (detached cryptographic signature verification and package provenance proof are not yet implemented). +> +> In accordance with reproducible build guarantees (§9.2), recipe discovery enforces a strict 3-tier source-ranked precedence hierarchy: +> 1. **Project-Local Recipes** (`./recipes/`): Absolute highest priority; project pins always override upstream. +> 2. **Shipped Catalog Recipes** (`/recipes/`): Built-in verified recipes shipped with ebuild. +> 3. **Cached Remote Index** (`~/.ebuild/index/recipes/`): Definitions synchronized from remote repositories. +> +> An index update will never override pinned URLs, build systems, or checksums defined in your project repository. +> +> Furthermore, `ebuild update-index` automatically prunes stale cached `.yaml` and `.yml` recipes from `~/.ebuild/index/recipes/` that are absent from the updated remote index, preventing retired packages from lingering in the local cache. An index that yields no usable entries is treated as a delivery fault: nothing is pruned, the cached index is left in place, and the condition is reported. +> +> To support integrity verification, the SHA-256 digest of the downloaded index is recorded in `~/.ebuild/index/packages.json.sha256`. + +### Component Contract (§10.1) Field Coverage + +eBuild recipes currently carry a subset of the Master Design §10.1 component contract: +- **Identity**: Package name, version, and optional description. +- **Dependencies**: List of external library package dependencies. +- **Compliance**: Package open-source license identifier. +- **Integrity**: Transport SHA-256 checksum pin for downloaded tarballs. + +*Deferred Contract Fields*: Compatibility constraints (EmbeddedOS ABI/API version, architecture, and SoC targets) and Resource constraints (`flash_max`, `ram_max`) are not yet evaluated by the registry layer and must be validated at project build configuration time. + +### Shipped Embedded Library Catalog + +ebuild includes a curated suite of pre-packaged recipes under `recipes/`: + +| Package | Version | Build System | License | Description | +|:---|:---|:---|:---|:---| +| **`cjson`** | 1.7.18 | CMake | MIT | Ultralightweight JSON parser in ANSI C | +| **`freertos`** | 11.1.0 | CMake | MIT | Real-time operating system kernel for embedded devices | +| **`littlefs`** | 2.9.3 | Make | BSD-3-Clause | Fail-safe power-resilient filesystem for microcontrollers | +| **`lvgl`** | 9.2.2 | CMake | MIT | Light and Versatile Embedded Graphics Library | +| **`lwip`** | 2.2.0 | CMake | BSD-3-Clause | Lightweight TCP/IP stack for embedded targets | +| **`mbedtls`** | 3.6.0 | CMake | Apache-2.0 | Cryptographic primitives, TLS/SSL stack | +| **`nanopb`** | 0.4.9.1 | CMake | zlib | Memory-efficient Protocol Buffers implementation | +| **`tinyusb`** | 0.18.0 | CMake | MIT | Cross-platform USB host/device stack | +| **`unity`** | 2.6.1 | CMake | MIT | Standard embedded C unit testing framework | +| **`zlib`** | 1.3.1 | CMake | Zlib | General-purpose lossless data compression | + +### Offline & Air-Gapped Operation + +For isolated CI/CD pipelines and field deployments: +- Set environment variable `EBUILD_OFFLINE=1` or pass `--offline` to `ebuild update-index` to use cached remote indices without attempting network synchronization. +- Note: Package archive source fetching (`ebuild build`) currently verifies archive integrity against pinned SHA-256 checksums but is not yet gated by the `--offline` flag. +- ebuild automatically searches local directories first and gracefully falls back to cached indices in `~/.ebuild/index/` if the network is unreachable. +- All downloads are validated against SHA-256 integrity pins before extraction. + + diff --git a/ebuild/cli/commands.py b/ebuild/cli/commands.py index ac9f206e..9d0d6ddd 100644 --- a/ebuild/cli/commands.py +++ b/ebuild/cli/commands.py @@ -38,31 +38,16 @@ from ebuild.packages.fetcher import FetchError, PackageFetcher from ebuild.packages.lockfile import Lockfile from ebuild.packages.recipe import RecipeError -from ebuild.packages.registry import create_registry +from ebuild.packages.registry import create_registry, find_recipe_dirs from ebuild.packages.resolver import PackageResolver, ResolveError pass_logger = click.make_pass_decorator(Logger, ensure=True) -# Default recipe search paths (relative to project root) -_RECIPE_DIRS = ["recipes"] +# Canonical recipe search path discovery +_find_recipe_dirs = find_recipe_dirs -def _find_recipe_dirs(project_dir: Path) -> List[Path]: - """Locate recipe directories: project-local and install-level.""" - dirs = [] - for name in _RECIPE_DIRS: - d = project_dir / name - if d.is_dir(): - dirs.append(d) - - # Also check ebuild install location - pkg_recipes = Path(__file__).resolve().parent.parent.parent / "recipes" - if pkg_recipes.is_dir() and pkg_recipes not in dirs: - dirs.append(pkg_recipes) - - return dirs - def _install_packages( cfg: ProjectConfig, @@ -1681,6 +1666,113 @@ def list_packages(log: Logger, config_path: str) -> None: pass +@cli.command("search") +@click.argument("query", required=False, default="") +@click.option("--all", "show_all", is_flag=True, default=False, help="Show all available packages.") +@click.option("--json", "as_json", is_flag=True, default=False, help="Output results in JSON format.") +@click.option("--build-system", "build_sys", default=None, help="Filter by build system (cmake, make, meson, etc.).") +@click.option("--license", "lic_filter", default=None, help="Filter by license.") +@click.option( + "--config", "config_path", + default="build.yaml", + type=click.Path(exists=False), + help="Path to the build configuration file.", +) +@click.pass_obj +def search_packages( + log: Logger, + query: str, + show_all: bool, + as_json: bool, + build_sys: Optional[str], + lic_filter: Optional[str], + config_path: str, +) -> None: + """Search for packages across local recipes, shipped catalog, and remote index.""" + from ebuild.packages.repository import PackageRepository + + config_path_obj = Path(config_path) + project_dir = config_path_obj.parent if config_path_obj.exists() else Path(".") + + repo = PackageRepository() + repo.load_all_sources(project_dir=project_dir) + + effective_query = "" if show_all else query + results = repo.search(query=effective_query, build_system=build_sys, license_filter=lic_filter) + + if as_json: + import json + click.echo(json.dumps([pkg.to_dict() for pkg in results], indent=2)) + return + + log.header("ebuild — Package Search") + if not results: + if query: + log.info(f"No packages found matching '{query}'. Add recipes to './recipes/' or run 'ebuild update-index --url '.") + else: + log.info("No packages found. Add recipes to './recipes/' or run 'ebuild update-index --url '.") + return + + log.info(f"Found {len(results)} package(s):") + for pkg in results: + lic = f" ({pkg.license})" if pkg.license else "" + desc = f" — {pkg.description}" if pkg.description else "" + log.step(f"{pkg.name} v{pkg.version} [{pkg.build_system}]{lic}{desc}") + + +@cli.command("update-index") +@click.option("--url", "index_url", default=None, help="Custom remote package index URL (HTTPS).") +@click.option("--offline", is_flag=True, default=False, help="Offline mode: do not download, use existing cache.") +@click.option("--force", is_flag=True, default=False, help="Force refresh even if cache is up-to-date.") +@click.pass_obj +def update_index(log: Logger, index_url: Optional[str], offline: bool, force: bool) -> None: + """Synchronize the local package index with the remote recipe repository.""" + import json + from ebuild.packages.index_sync import IndexSyncManager, IndexSyncError + + log.header("ebuild — Update Package Index") + sync_mgr = IndexSyncManager() + + prev_sha256 = None + if sync_mgr.meta_json.is_file(): + try: + with open(sync_mgr.meta_json, "r", encoding="utf-8") as mf: + prev_sha256 = json.load(mf).get("sha256") + except Exception: + prev_sha256 = None + + try: + res = sync_mgr.sync(url=index_url, force=force, offline=offline) + msg = res.message + is_fallback = getattr(res, "is_fallback", False) + current_sha256 = getattr(res, "sha256", None) + pruned_count = getattr(res, "pruned", 0) + if not current_sha256 and sync_mgr.meta_json.is_file(): + try: + with open(sync_mgr.meta_json, "r", encoding="utf-8") as mf: + current_sha256 = json.load(mf).get("sha256") + except Exception: + current_sha256 = None + + if is_fallback and not offline: + log.warning(msg) + if current_sha256: + log.info(f"Index SHA-256 digest: {current_sha256}") + log.info(f"Index cache located at: {sync_mgr.index_dir}") + raise SystemExit(1) + log.success(msg) + if current_sha256: + log.info(f"Index SHA-256 digest: {current_sha256}") + if prev_sha256 and prev_sha256 != current_sha256: + log.info(f"Index updated (previous digest: {prev_sha256})") + if pruned_count > 0: + log.info(f"Pruned {pruned_count} stale cached recipe(s)") + log.info(f"Index cache located at: {sync_mgr.index_dir}") + except IndexSyncError as e: + log.error(f"Index update failed: {e}") + raise SystemExit(1) + + @cli.command() @click.argument("input_text", required=False) @click.option("--file", "input_file", type=click.Path(exists=True), help="Hardware design file (KiCad .kicad_sch, Eagle .sch, BOM .csv, YAML, text).") @@ -2491,7 +2583,6 @@ def test(log: Logger, config_path: str, build_dir: str, log.success("All tests passed.") - @cli.command() @click.option("--config", "config_path", default="build.yaml", type=click.Path(), help="Path to the build configuration file.") diff --git a/ebuild/packages/index_sync.py b/ebuild/packages/index_sync.py new file mode 100644 index 00000000..42f417f5 --- /dev/null +++ b/ebuild/packages/index_sync.py @@ -0,0 +1,386 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2026 EoS Project + +"""Remote package index synchronization and offline cache manager. + +Manages downloading, verifying, and caching package indices and recipe +definitions from remote repositories (HTTPS) into a local user cache directory +(~/.ebuild/index/). Supports offline fallback and security sanitization. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import os +import re +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any, Dict, List, NamedTuple, Optional + +import yaml + +from ebuild.packages.recipe import RecipeError, parse_recipe + +logger = logging.getLogger(__name__) + +# Default remote repository index URL (empty until official registry repository is published) +DEFAULT_INDEX_URL = "" + +# Maximum response size allowed for index download (10 MB) +MAX_INDEX_SIZE_BYTES = 10 * 1024 * 1024 + +# Network timeout in seconds +DEFAULT_NETWORK_TIMEOUT_SECONDS = 10 + +# Cache TTL in seconds (24 hours) for freshness check +CACHE_TTL_SECONDS = 86400 + +# Valid package name pattern (strict validation to prevent path traversal) +_SAFE_NAME_RE = re.compile(r"^[a-zA-Z0-9_-]+$") + + +class IndexSyncError(Exception): + """Raised when index synchronization fails and cannot fallback.""" + + +class SyncResult(NamedTuple): + """Result of index synchronization.""" + + package_count: int + message: str + is_fallback: bool = False + sha256: Optional[str] = None + pruned: int = 0 + + +def is_offline(offline_flag: bool = False) -> bool: + """Check if the execution environment is configured for offline operation.""" + if offline_flag: + return True + env_val = os.environ.get("EBUILD_OFFLINE", "").strip().lower() + return env_val in ("1", "true", "yes", "on") + + +def get_default_index_dir() -> Path: + """Get the local index cache directory path, respecting env overrides.""" + if "EBUILD_INDEX_PATH" in os.environ: + return Path(os.environ["EBUILD_INDEX_PATH"]) + if "EBUILD_CACHE_DIR" in os.environ: + return Path(os.environ["EBUILD_CACHE_DIR"]) / "index" + return Path.home() / ".ebuild" / "index" + + +def sanitize_package_name(name: str) -> str: + """Validate and sanitize a package name to prevent path traversal attacks. + + Args: + name: The candidate package name. + + Returns: + The validated package name. + + Raises: + ValueError: If the package name contains invalid or unsafe characters. + """ + cleaned = name.strip() + if not cleaned or not _SAFE_NAME_RE.match(cleaned): + raise ValueError( + f"Invalid package name '{name}': names must only contain alphanumeric " + f"characters, underscores, or hyphens." + ) + return cleaned + + +class IndexSyncManager: + """Coordinates remote index fetching, integrity validation, and local caching.""" + + def __init__( + self, + index_dir: Optional[Path | str] = None, + default_url: str = DEFAULT_INDEX_URL, + ) -> None: + self.index_dir = ( + Path(index_dir) if index_dir is not None else get_default_index_dir() + ) + self.default_url = default_url + self.packages_json = self.index_dir / "packages.json" + self.recipes_dir = self.index_dir / "recipes" + self.meta_json = self.index_dir / "index-meta.json" + + def ensure_directories(self) -> None: + """Create necessary index directories if they do not exist.""" + self.index_dir.mkdir(parents=True, exist_ok=True) + self.recipes_dir.mkdir(parents=True, exist_ok=True) + + def get_recipe_dirs(self) -> List[Path]: + """Return list of recipe directories managed by this sync index.""" + if self.recipes_dir.is_dir(): + return [self.recipes_dir] + return [] + + def load_cached_entries(self) -> List[Dict[str, Any]]: + """Load entries from the local packages.json cache if present.""" + if not self.packages_json.is_file(): + return [] + try: + with open(self.packages_json, "r", encoding="utf-8") as f: + data = json.load(f) + if isinstance(data, list): + return data + logger.warning("Cached index at %s is not a list", self.packages_json) + except (json.JSONDecodeError, OSError) as e: + logger.warning("Failed to load cached index: %s", e) + return [] + + def sync( + self, + url: Optional[str] = None, + force: bool = False, + offline: bool = False, + timeout: int = DEFAULT_NETWORK_TIMEOUT_SECONDS, + ) -> SyncResult: + """Synchronize the index from the remote URL to the local cache. + + Args: + url: The remote index URL (defaults to configured default_url). + force: If True, re-download even if recently synced. + offline: If True, skip network download and use cached files. + timeout: Network timeout in seconds. + + Returns: + SyncResult with package_count, message, is_fallback, sha256, and pruned. + """ + target_url = (url or self.default_url).strip() + self.ensure_directories() + + # Offline mode + if is_offline(offline): + cached = self.load_cached_entries() + count = len(cached) + return SyncResult(count, f"Offline mode: using cached index ({count} packages)", is_fallback=False) + + # First check target_url validity before using or validating against it + if not target_url: + raise IndexSyncError( + "No remote package index URL configured. Provide a repository URL with '--url ' until a default registry is published." + ) + + if not target_url.startswith("https://"): + raise IndexSyncError( + f"Insecure index URL '{target_url}': only HTTPS URLs are permitted." + ) + + # Staleness check: if cache exists, is fresh (< 24h), matches origin URL, and force=False + cached_meta: Dict[str, Any] = {} + if self.meta_json.is_file(): + try: + with open(self.meta_json, "r", encoding="utf-8") as mf: + cached_meta = json.load(mf) + except Exception: + cached_meta = {} + + cached_url = cached_meta.get("url") + same_origin = (cached_url == target_url) if cached_url else (url is None) + + if not force and same_origin and self.packages_json.is_file(): + try: + synced_at = cached_meta.get("synced_at") + if synced_at is None: + synced_at = self.packages_json.stat().st_mtime + cache_age = time.time() - float(synced_at) + if cache_age < CACHE_TTL_SECONDS: + cached = self.load_cached_entries() + count = len(cached) + cached_meta_sha = cached_meta.get("sha256") + return SyncResult( + count, + f"Cache is up-to-date (synced recently). Use --force to re-download. ({count} packages)", + is_fallback=False, + sha256=cached_meta_sha, + ) + except OSError: + pass + + logger.info("Fetching remote package index from %s", target_url) + + # 1. Network download with narrow exception handling + try: + req = urllib.request.Request( + target_url, + headers={"User-Agent": "ebuild-package-manager/3.0"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as response: + content_length = response.headers.get("Content-Length") + if content_length: + try: + declared_length = int(str(content_length).strip()) + if declared_length > MAX_INDEX_SIZE_BYTES: + raise IndexSyncError( + f"Index download exceeds maximum allowed size ({MAX_INDEX_SIZE_BYTES} bytes)" + ) + except ValueError: + # Malformed header is advisory; bounded read below enforces size limit + pass + raw_bytes = response.read(MAX_INDEX_SIZE_BYTES + 1) + if len(raw_bytes) > MAX_INDEX_SIZE_BYTES: + raise IndexSyncError( + f"Index download exceeded maximum size limit of {MAX_INDEX_SIZE_BYTES} bytes" + ) + except IndexSyncError: + raise + except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError) as e: + # Fallback to local cache if available on network error + cached = self.load_cached_entries() + if cached: + logger.warning("Remote sync failed (%s). Falling back to cached index.", e) + cached_meta_sha = cached_meta.get("sha256") + return SyncResult( + len(cached), + f"Network sync failed ({e}); fell back to cached index ({len(cached)} packages)", + is_fallback=True, + sha256=cached_meta_sha, + ) + raise IndexSyncError(f"Failed to fetch remote package index and no cache is available: {e}") from e + + # 2. Parse and validate JSON index + try: + data = json.loads(raw_bytes.decode("utf-8")) + except (json.JSONDecodeError, UnicodeDecodeError) as err: + raise IndexSyncError(f"Corrupted or invalid JSON index from {target_url}: {err}") from err + + if not isinstance(data, list): + raise IndexSyncError("Invalid index schema: expected top-level JSON array") + + # 3. Filter, sanitize, and deduplicate entries in memory BEFORE caching to disk + valid_entries: List[Dict[str, Any]] = [] + seen_names: set[str] = set() + for entry in data: + if not isinstance(entry, dict) or "name" not in entry: + continue + try: + pkg_name = sanitize_package_name(str(entry["name"])) + if pkg_name in seen_names: + logger.warning("Discarding duplicate package entry in index: %s", pkg_name) + continue + seen_names.add(pkg_name) + valid_entries.append(entry) + except ValueError as ve: + logger.warning("Skipping unsafe package entry: %s", ve) + continue + + # 4. If index contains no usable entries, treat as delivery fault: keep existing cache untouched + if not valid_entries: + cached_count = ( + len(list(self.recipes_dir.glob("*.yaml")) + list(self.recipes_dir.glob("*.yml"))) + if self.recipes_dir.is_dir() + else 0 + ) + msg = f"Remote package index contained no usable entries; keeping {cached_count} cached recipes" + logger.warning(msg) + cached_meta_sha = None + if self.meta_json.is_file(): + try: + with open(self.meta_json, "r", encoding="utf-8") as mf: + cached_meta_sha = json.load(mf).get("sha256") + except Exception: + pass + cached_entries = self.load_cached_entries() + return SyncResult( + package_count=len(cached_entries), + message=msg, + is_fallback=True, + sha256=cached_meta_sha, + pruned=0, + ) + + # Write cached packages.json atomically with sanitized entries only + temp_json = self.packages_json.with_suffix(".tmp") + with open(temp_json, "w", encoding="utf-8") as f: + json.dump(valid_entries, f, indent=2) + temp_json.replace(self.packages_json) + + # 5. Record SHA-256 digest of the downloaded index for integrity visibility + index_sha256 = hashlib.sha256(raw_bytes).hexdigest() + sha_file = self.packages_json.with_name(self.packages_json.name + ".sha256") + temp_sha = sha_file.with_suffix(".tmp") + with open(temp_sha, "w", encoding="utf-8") as sf: + sf.write(f"{index_sha256} packages.json\n") + temp_sha.replace(sha_file) + + # Write index metadata (origin URL, digest, sync time) + meta_data = { + "url": target_url, + "sha256": index_sha256, + "synced_at": time.time(), + } + temp_meta = self.meta_json.with_suffix(".tmp") + with open(temp_meta, "w", encoding="utf-8") as mf: + json.dump(meta_data, mf, indent=2) + temp_meta.replace(self.meta_json) + + logger.info("Remote index SHA-256 digest: %s", index_sha256) + + # 6. Process and cache full recipe YAML definitions + synced_count = 0 + written_yaml_stems: set[str] = set() + for entry in valid_entries: + pkg_name = sanitize_package_name(str(entry["name"])) + recipe_filename = f"{pkg_name}.yaml" + recipe_path = self.recipes_dir / recipe_filename + + recipe_dict = { + "package": pkg_name, + "version": str(entry.get("version", "1.0.0")), + "description": entry.get("description", ""), + "license": entry.get("license", ""), + "url": entry.get("url", ""), + "checksum": entry.get("checksum", ""), + "build": entry.get("build_system", entry.get("build", "cmake")), + "dependencies": entry.get("dependencies", []), + "configure_args": entry.get("configure_args", []), + "build_args": entry.get("build_args", []), + "patches": entry.get("patches", []), + } + + # Only write recipe file if entry provides a download URL + if recipe_dict["url"]: + try: + recipe = parse_recipe(recipe_dict) + with open(recipe_path, "w", encoding="utf-8") as rf: + yaml.safe_dump(recipe.to_dict(), rf, sort_keys=False) + synced_count += 1 + written_yaml_stems.add(pkg_name) + except (RecipeError, ValueError) as re_err: + logger.warning("Skipping invalid recipe entry %s: %s", pkg_name, re_err) + continue + + # Prune stale cached recipes not present in the newly synchronized index + pruned_count = 0 + if self.recipes_dir.is_dir(): + keep = {f"{sanitize_package_name(str(e['name']))}.yaml" for e in valid_entries} + keep.update( + f"{sanitize_package_name(str(e['name']))}.yml" + for e in valid_entries + if sanitize_package_name(str(e['name'])) not in written_yaml_stems + ) + for existing_file in list(self.recipes_dir.glob("*.yaml")) + list(self.recipes_dir.glob("*.yml")): + if existing_file.name not in keep: + try: + existing_file.unlink() + pruned_count += 1 + logger.debug("Pruned stale cached recipe: %s", existing_file.name) + except OSError as oe: + logger.warning("Failed to prune stale recipe %s: %s", existing_file.name, oe) + + return SyncResult( + synced_count, + f"Successfully synchronized {synced_count} packages from remote index", + is_fallback=False, + sha256=index_sha256, + pruned=pruned_count, + ) + diff --git a/ebuild/packages/recipe.py b/ebuild/packages/recipe.py index 71bf2c2e..def2730d 100644 --- a/ebuild/packages/recipe.py +++ b/ebuild/packages/recipe.py @@ -89,8 +89,33 @@ def validate(self) -> None: ) -def _parse_recipe(raw: Dict[str, Any], source_path: Optional[Path] = None) -> PackageRecipe: - """Parse a raw YAML dict into a PackageRecipe.""" + def to_dict(self) -> Dict[str, Any]: + """Convert recipe to dictionary for YAML serialization.""" + data: Dict[str, Any] = { + "package": self.name, + "version": self.version, + } + if self.description: + data["description"] = self.description + if self.license: + data["license"] = self.license + data["url"] = self.url + if self.checksum: + data["checksum"] = self.checksum + data["build"] = self.build_system + if self.dependencies: + data["dependencies"] = self.dependencies + if self.configure_args: + data["configure_args"] = self.configure_args + if self.build_args: + data["build_args"] = self.build_args + if self.patches: + data["patches"] = self.patches + return data + + +def parse_recipe(raw: Dict[str, Any], source_path: Optional[Path] = None) -> PackageRecipe: + """Parse a raw YAML dict into a validated PackageRecipe.""" recipe = PackageRecipe( name=raw.get("package", raw.get("name", "")), version=str(raw.get("version", "")), @@ -109,6 +134,10 @@ def _parse_recipe(raw: Dict[str, Any], source_path: Optional[Path] = None) -> Pa return recipe +# Alias for backward compatibility +_parse_recipe = parse_recipe + + def load_recipe(recipe_path: str | Path) -> PackageRecipe: """Load a package recipe from a YAML file. diff --git a/ebuild/packages/registry.py b/ebuild/packages/registry.py index 6d43f4c3..8b0db6ad 100644 --- a/ebuild/packages/registry.py +++ b/ebuild/packages/registry.py @@ -9,12 +9,15 @@ from __future__ import annotations +import logging import re from pathlib import Path from typing import Dict, List, Optional, Tuple from ebuild.packages.recipe import PackageRecipe, RecipeError, load_recipe +logger = logging.getLogger(__name__) + # Everything from the first '-' or '+' is a suffix: a pre-release tag # ("3.6.0-rc1") or build metadata ("1.3.1+patch2"). _SUFFIX_SPLIT = re.compile(r"[-+]") @@ -89,7 +92,7 @@ class PackageRegistry: """ def __init__(self) -> None: - self._recipes: Dict[str, Dict[str, PackageRecipe]] = {} + self._recipes: Dict[str, Dict[str, Tuple[int, PackageRecipe]]] = {} self._search_paths: List[Path] = [] def add_search_path(self, path: str | Path) -> None: @@ -105,67 +108,78 @@ def scan(self) -> int: Number of recipes loaded. """ count = 0 - for search_path in self._search_paths: + for rank, search_path in enumerate(self._search_paths): for recipe_file in sorted(search_path.glob("*.yaml")): try: recipe = load_recipe(recipe_file) - self._register(recipe) + self._register(recipe, rank=rank) count += 1 except (RecipeError, FileNotFoundError): continue for recipe_file in sorted(search_path.glob("*.yml")): try: recipe = load_recipe(recipe_file) - self._register(recipe) + self._register(recipe, rank=rank) count += 1 except (RecipeError, FileNotFoundError): continue return count - def _register(self, recipe: PackageRecipe) -> None: - """Register a recipe in the internal index.""" + def _register(self, recipe: PackageRecipe, rank: int = 0) -> None: + """Register a recipe in the internal index (higher-priority source wins for same version).""" if recipe.name not in self._recipes: self._recipes[recipe.name] = {} - self._recipes[recipe.name][recipe.version] = recipe + if recipe.version not in self._recipes[recipe.name]: + self._recipes[recipe.name][recipe.version] = (rank, recipe) + else: + existing_rank, _ = self._recipes[recipe.name][recipe.version] + if rank < existing_rank: + self._recipes[recipe.name][recipe.version] = (rank, recipe) def get(self, name: str, version: Optional[str] = None) -> Optional[PackageRecipe]: """Look up a package recipe by name and optional version. - If no version is specified, returns the latest (highest) version. + If no version is specified, returns the latest (highest) version + within the highest-priority source that defines this package. """ versions = self._recipes.get(name) if not versions: return None if version: - return versions.get(version) + entry = versions.get(version) + return entry[1] if entry else None - latest_version = sorted(versions.keys(), key=version_sort_key)[-1] - return versions[latest_version] + min_rank = min(rank for rank, _ in versions.values()) + source_versions = { + v: r for v, (rank, r) in versions.items() if rank == min_rank + } + latest_version = sorted(source_versions.keys(), key=version_sort_key)[-1] + return source_versions[latest_version] def has(self, name: str, version: Optional[str] = None) -> bool: """Check if a recipe exists.""" return self.get(name, version) is not None def list_packages(self) -> List[PackageRecipe]: - """Return all registered recipes (latest version of each).""" + """Return all registered recipes (latest version within highest-priority source of each).""" result = [] for name in sorted(self._recipes.keys()): - versions = self._recipes[name] - latest = sorted(versions.keys(), key=version_sort_key)[-1] - result.append(versions[latest]) + recipe = self.get(name) + if recipe is not None: + result.append(recipe) return result def list_all_versions(self, name: str) -> List[PackageRecipe]: """Return all versions of a package.""" versions = self._recipes.get(name, {}) return [ - versions[v] - for v in sorted( - versions.keys(), - key=version_sort_key, - ) - ] + versions[v][1] + for v in sorted( + versions.keys(), + key=version_sort_key, + ) + ] @property def package_count(self) -> int: @@ -187,3 +201,43 @@ def create_registry(*recipe_dirs: str | Path) -> PackageRegistry: registry.add_search_path(d) registry.scan() return registry + + +def find_recipe_dirs( + project_dir: Optional[Path | str] = None, + remote_index_dir: Optional[Path | str] = None, +) -> List[Path]: + """Locate recipe directories in priority order: project-local, install-level, and remote synced cache. + + Args: + project_dir: Optional path to project root. + remote_index_dir: Optional custom remote index cache directory. + + Returns: + List of existing recipe directory Paths in priority order. + """ + dirs: List[Path] = [] + if project_dir is not None: + p_dir = Path(project_dir) + local_recipes = p_dir / "recipes" + if local_recipes.is_dir(): + dirs.append(local_recipes) + + # Shipped system recipes + pkg_recipes = Path(__file__).resolve().parent.parent.parent / "recipes" + if pkg_recipes.is_dir() and pkg_recipes not in dirs: + dirs.append(pkg_recipes) + + # Remote synced cache in ~/.ebuild/index/recipes/ (or custom index_dir) + try: + if remote_index_dir is not None: + cached_recipes = Path(remote_index_dir) / "recipes" + else: + from ebuild.packages.index_sync import get_default_index_dir + cached_recipes = get_default_index_dir() / "recipes" + if cached_recipes.is_dir() and cached_recipes not in dirs: + dirs.append(cached_recipes) + except (ImportError, OSError) as e: + logger.warning("Failed to resolve remote index recipes directory: %s", e) + + return dirs diff --git a/ebuild/packages/repository.py b/ebuild/packages/repository.py index 7f56abb4..4d02f58c 100644 --- a/ebuild/packages/repository.py +++ b/ebuild/packages/repository.py @@ -1,21 +1,23 @@ + # SPDX-License-Identifier: MIT # Copyright (c) 2026 EoS Project """Remote package repository — index-based package discovery. Provides search, info, and listing of packages available from -local recipe directories or remote repository indices. +local recipe directories, shipped package catalogs, or remote repository indices. """ from __future__ import annotations import json import logging -from dataclasses import dataclass, field +from dataclasses import asdict, dataclass, field from pathlib import Path -from typing import Dict, List, Optional +from typing import Any, Dict, List, Optional -from ebuild.packages.registry import PackageRegistry, create_registry +from ebuild.packages.index_sync import IndexSyncManager, sanitize_package_name +from ebuild.packages.registry import PackageRegistry, create_registry, find_recipe_dirs logger = logging.getLogger(__name__) @@ -33,17 +35,22 @@ class PackageInfo: url: str = "" checksum: str = "" + def to_dict(self) -> Dict[str, Any]: + """Convert PackageInfo dataclass to a JSON-serializable dictionary.""" + return asdict(self) + class PackageRepository: """Repository index for discovering and querying available packages. - Wraps one or more PackageRegistry instances and provides - search, info, and listing functionality. + Wraps one or more PackageRegistry instances and remote index files, + providing unified search, info, and listing functionality. """ - def __init__(self) -> None: + def __init__(self, sync_manager: Optional[IndexSyncManager] = None) -> None: self._registries: List[PackageRegistry] = [] self._index: Dict[str, PackageInfo] = {} + self.sync_manager = sync_manager or IndexSyncManager() def add_recipe_directory(self, path: str | Path) -> int: """Add a local recipe directory to the repository. @@ -69,8 +76,10 @@ def add_recipe_directory(self, path: str | Path) -> int: url=recipe.url, checksum=recipe.checksum, ) - self._index[recipe.name] = info - count += 1 + # Local/earlier recipe directory overrides later directories if already loaded + if recipe.name not in self._index: + self._index[recipe.name] = info + count += 1 return count @@ -91,8 +100,12 @@ def load_index(self, index_path: str | Path) -> int: logger.warning("Repository index not found: %s", index_path) return 0 - with open(index_path, "r", encoding="utf-8") as f: - data = json.load(f) + try: + with open(index_path, "r", encoding="utf-8") as f: + data = json.load(f) + except (json.JSONDecodeError, OSError) as e: + logger.warning("Failed to parse repository index %s: %s", index_path, e) + return 0 if not isinstance(data, list): logger.warning("Invalid index format: expected array") @@ -102,9 +115,13 @@ def load_index(self, index_path: str | Path) -> int: for entry in data: if not isinstance(entry, dict) or "name" not in entry: continue + try: + name = sanitize_package_name(str(entry["name"])) + except ValueError: + continue info = PackageInfo( - name=entry["name"], - version=entry.get("version", "0.0.0"), + name=name, + version=str(entry.get("version", "0.0.0")), description=entry.get("description", ""), license=entry.get("license", ""), build_system=entry.get("build_system", entry.get("build", "cmake")), @@ -112,30 +129,67 @@ def load_index(self, index_path: str | Path) -> int: url=entry.get("url", ""), checksum=entry.get("checksum", ""), ) - self._index[info.name] = info - count += 1 + # Local recipe directory overrides remote index if already loaded + if name not in self._index: + self._index[name] = info + count += 1 return count - def search(self, query: str) -> List[PackageInfo]: - """Search for packages matching a query string. + def load_all_sources(self, project_dir: Optional[Path | str] = None) -> int: + """Load all available sources: project recipes, shipped recipes, and remote cache. + + Args: + project_dir: Optional path to project root. + + Returns: + Total count of packages indexed across all sources. + """ + for rdir in find_recipe_dirs(project_dir, remote_index_dir=self.sync_manager.index_dir): + self.add_recipe_directory(rdir) + + # Cached remote JSON index + if self.sync_manager.packages_json.is_file(): + self.load_index(self.sync_manager.packages_json) + + return self.package_count - Matches against package name and description (case-insensitive). + def search( + self, + query: str = "", + build_system: Optional[str] = None, + license_filter: Optional[str] = None, + ) -> List[PackageInfo]: + """Search for packages matching query and filters. Args: - query: Search query string. + query: Search query string (matches name, description, license). + build_system: Optional build system filter (e.g., 'cmake', 'make'). + license_filter: Optional license filter. Returns: - List of matching PackageInfo objects. + List of matching PackageInfo objects sorted by name. """ - query_lower = query.lower() + effective_lic = license_filter + query_lower = query.strip().lower() results = [] + for info in sorted(self._index.values(), key=lambda p: p.name): - if ( - query_lower in info.name.lower() - or query_lower in info.description.lower() - ): - results.append(info) + if query_lower: + name_match = query_lower in info.name.lower() + desc_match = query_lower in info.description.lower() + lic_match = query_lower in info.license.lower() + if not (name_match or desc_match or lic_match): + continue + + if build_system and info.build_system.lower() != build_system.strip().lower(): + continue + + if effective_lic and effective_lic.strip().lower() not in info.license.lower(): + continue + + results.append(info) + return results def info(self, name: str) -> Optional[PackageInfo]: @@ -169,18 +223,7 @@ def export_index(self, output_path: str | Path) -> None: output_path: Path to write the JSON index file. """ output_path = Path(output_path) - entries = [] - for info in sorted(self._index.values(), key=lambda p: p.name): - entries.append({ - "name": info.name, - "version": info.version, - "description": info.description, - "license": info.license, - "build_system": info.build_system, - "dependencies": info.dependencies, - "url": info.url, - "checksum": info.checksum, - }) + entries = [info.to_dict() for info in sorted(self._index.values(), key=lambda p: p.name)] output_path.parent.mkdir(parents=True, exist_ok=True) with open(output_path, "w", encoding="utf-8") as f: diff --git a/pytest.ini b/pytest.ini index 4ace1581..b641d8b7 100644 --- a/pytest.ini +++ b/pytest.ini @@ -22,8 +22,10 @@ markers = needs_yaml: tests that import modules requiring pyyaml # ── Output ─────────────────────────────────────────────────── +# Prevent pytest entrypoint autoload failure from broken faker environment plugin addopts = -v + -p no:faker --tb=short --strict-markers --no-header diff --git a/recipes/cjson.yaml b/recipes/cjson.yaml new file mode 100644 index 00000000..b0e9522f --- /dev/null +++ b/recipes/cjson.yaml @@ -0,0 +1,11 @@ +package: cjson +version: "1.7.18" +description: "Ultralightweight JSON parser in ANSI C" +license: MIT +url: https://github.com/DaveGamble/cJSON/archive/refs/tags/v1.7.18.tar.gz +checksum: sha256:3aa806844a03442c00769b83e99970be70fbef03735ff898f4811dd03b9f5ee5 +build: cmake +configure_args: + - -DENABLE_CJSON_TEST=OFF + - -DBUILD_SHARED_LIBS=OFF +dependencies: [] diff --git a/recipes/lvgl.yaml b/recipes/lvgl.yaml new file mode 100644 index 00000000..aa62988a --- /dev/null +++ b/recipes/lvgl.yaml @@ -0,0 +1,11 @@ +package: lvgl +version: "9.2.2" +description: "Light and Versatile Embedded Graphics Library" +license: MIT +url: https://github.com/lvgl/lvgl/archive/refs/tags/v9.2.2.tar.gz +checksum: sha256:129b4e00e06639fa79d7e8a6cab3c1ecce2445b1a246652ccd34f22e7b17ad6f +build: cmake +configure_args: + - -DLV_CONF_BUILD_DISABLE_EXAMPLES=ON + - -DLV_CONF_BUILD_DISABLE_DEMOS=ON +dependencies: [] diff --git a/recipes/nanopb.yaml b/recipes/nanopb.yaml new file mode 100644 index 00000000..daf7ae18 --- /dev/null +++ b/recipes/nanopb.yaml @@ -0,0 +1,11 @@ +package: nanopb +version: "0.4.9.1" +description: "Protocol Buffers with small code size for microcontrollers" +license: zlib +url: https://github.com/nanopb/nanopb/archive/refs/tags/0.4.9.1.tar.gz +checksum: sha256:4575944a468718ef25f05eb01d994364650b581563089a9841986bb1e460eac3 +build: cmake +configure_args: + - -Dnanopb_BUILD_RUNTIME_ONLY=ON + - -DBUILD_STATIC_LIBS=ON +dependencies: [] diff --git a/recipes/tinyusb.yaml b/recipes/tinyusb.yaml new file mode 100644 index 00000000..032009cb --- /dev/null +++ b/recipes/tinyusb.yaml @@ -0,0 +1,9 @@ +package: tinyusb +version: "0.18.0" +description: "Open-source cross-platform USB host/device stack for embedded system" +license: MIT +url: https://github.com/hathach/tinyusb/archive/refs/tags/0.18.0.tar.gz +checksum: sha256:e7fa1bd723213749a0362c79eaccc99e84c8adea8f0a63588c4e4812608b7aa9 +build: cmake +configure_args: [] +dependencies: [] diff --git a/recipes/unity.yaml b/recipes/unity.yaml new file mode 100644 index 00000000..75d7d67a --- /dev/null +++ b/recipes/unity.yaml @@ -0,0 +1,11 @@ +package: unity +version: "2.6.1" +description: "Simple Unit Testing for C" +license: MIT +url: https://github.com/ThrowTheSwitch/Unity/archive/refs/tags/v2.6.1.tar.gz +checksum: sha256:b41a66d45a6b99758fb3202ace6178177014d52fc524bf1f72687d93e9867292 +build: cmake +configure_args: + - -DUNITY_EXTENSION_FIXTURE=OFF + - -DUNITY_EXTENSION_MEMORY=OFF +dependencies: [] diff --git a/tests/unit/test_index_sync.py b/tests/unit/test_index_sync.py new file mode 100644 index 00000000..7265fd2d --- /dev/null +++ b/tests/unit/test_index_sync.py @@ -0,0 +1,552 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2026 EoS Project + +"""Unit tests for remote package index sync and offline caching.""" + +import json +import urllib.error +import urllib.request +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest +from click.testing import CliRunner + +from ebuild.cli.commands import cli +from ebuild.packages.index_sync import ( + MAX_INDEX_SIZE_BYTES, + IndexSyncError, + IndexSyncManager, + get_default_index_dir, + is_offline, + sanitize_package_name, +) + + +def test_sanitize_package_name(): + assert sanitize_package_name("cjson") == "cjson" + assert sanitize_package_name("my_pkg-123") == "my_pkg-123" + + with pytest.raises(ValueError, match="Invalid package name"): + sanitize_package_name("../../etc/passwd") + + with pytest.raises(ValueError, match="Invalid package name"): + sanitize_package_name("pkg with spaces") + + with pytest.raises(ValueError, match="Invalid package name"): + sanitize_package_name("") + + +def test_is_offline(monkeypatch): + assert not is_offline(False) + assert is_offline(True) + + monkeypatch.setenv("EBUILD_OFFLINE", "1") + assert is_offline(False) + + monkeypatch.setenv("EBUILD_OFFLINE", "true") + assert is_offline(False) + + monkeypatch.setenv("EBUILD_OFFLINE", "0") + assert not is_offline(False) + + +def test_get_default_index_dir(monkeypatch, tmp_path): + custom_dir = tmp_path / "custom_index" + monkeypatch.setenv("EBUILD_INDEX_PATH", str(custom_dir)) + assert get_default_index_dir() == custom_dir + + +def test_index_sync_missing_url(tmp_path): + """Verify Finding 3: Empty default URL raises IndexSyncError when no URL is given.""" + mgr = IndexSyncManager(index_dir=tmp_path) + with pytest.raises(IndexSyncError, match="No remote package index URL configured"): + mgr.sync() + + +def test_index_sync_insecure_url(tmp_path): + mgr = IndexSyncManager(index_dir=tmp_path) + with pytest.raises(IndexSyncError, match="Insecure index URL"): + mgr.sync(url="http://insecure.example.com/index.json") + + +def test_index_sync_success(tmp_path): + mgr = IndexSyncManager(index_dir=tmp_path) + + sample_index = [ + { + "name": "mock-pkg", + "version": "1.0.0", + "description": "A mock package for testing", + "license": "MIT", + "url": "https://example.com/mock-pkg-1.0.0.tar.gz", + "checksum": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "build_system": "cmake", + "configure_args": ["-DMOCK=ON"], + } + ] + raw_json = json.dumps(sample_index).encode("utf-8") + + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + + assert res.package_count == 1 + assert not res.is_fallback + assert "Successfully synchronized 1 packages" in res.message + assert mgr.packages_json.is_file() + assert mgr.packages_json.with_name(mgr.packages_json.name + ".sha256").is_file() + assert mgr.meta_json.is_file() + + # Check that recipe YAML was cached + recipe_file = mgr.recipes_dir / "mock-pkg.yaml" + assert recipe_file.is_file() + content = recipe_file.read_text(encoding="utf-8") + assert "mock-pkg" in content + assert "1.0.0" in content + + +def test_index_sync_corrupted_json(tmp_path): + mgr = IndexSyncManager(index_dir=tmp_path) + + mock_resp = MagicMock() + mock_resp.read.return_value = b"{ invalid json" + mock_resp.headers = {} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + with pytest.raises(IndexSyncError, match="Corrupted or invalid JSON"): + mgr.sync(url="https://example.com/index.json") + + +def test_index_sync_network_error_fallback(tmp_path): + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + + # Seed cache + cached_data = [{"name": "cached-lib", "version": "2.0.0"}] + mgr.packages_json.write_text(json.dumps(cached_data), encoding="utf-8") + + with patch("urllib.request.urlopen", side_effect=urllib.error.URLError("No connection")): + res = mgr.sync(url="https://example.com/index.json", force=True) + + assert res.package_count == 1 + assert res.is_fallback + assert "fell back to cached index" in res.message + + +def test_index_sync_offline_mode(tmp_path): + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + + cached_data = [{"name": "offline-lib", "version": "1.0.0"}] + mgr.packages_json.write_text(json.dumps(cached_data), encoding="utf-8") + + res = mgr.sync(offline=True) + assert res.package_count == 1 + assert not res.is_fallback + assert "Offline mode" in res.message + + +def test_index_sync_filters_unsafe_and_counts_accurately(tmp_path): + """Verify Findings 6 & 9a: + 1. Cache filters unsafe package names and does not store them in packages.json. + 2. synced_count only increments for packages where a recipe was actually written. + """ + mgr = IndexSyncManager(index_dir=tmp_path) + + sample_index = [ + { + "name": "good-pkg", + "version": "1.0.0", + "url": "https://example.com/good.tar.gz", + "checksum": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + }, + { + "name": "bad name!", + "version": "1.0.0", + "url": "https://example.com/bad.tar.gz", + }, + { + "name": "urlless-pkg", + "version": "1.0.0", + "description": "No download URL", + }, + ] + raw_json = json.dumps(sample_index).encode("utf-8") + + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + + # Only good-pkg had a URL and passed validation -> 1 recipe written + assert res.package_count == 1 + assert "Successfully synchronized 1 packages" in res.message + + # Verify packages.json does not contain "bad name!" + cached_entries = json.loads(mgr.packages_json.read_text(encoding="utf-8")) + entry_names = [e["name"] for e in cached_entries] + assert "good-pkg" in entry_names + assert "urlless-pkg" in entry_names + assert "bad name!" not in entry_names + + # Verify only good-pkg recipe file exists on disk + assert (mgr.recipes_dir / "good-pkg.yaml").is_file() + assert not (mgr.recipes_dir / "urlless-pkg.yaml").exists() + + +def test_index_sync_force_and_staleness(tmp_path): + """Verify Finding 2 & Finding 7: Cache freshness checks TTL and matches origin URL unless force=True.""" + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + + cached_data = [{"name": "fresh-pkg", "version": "1.0.0", "url": "https://example.com/fresh.tar.gz"}] + mgr.packages_json.write_text(json.dumps(cached_data), encoding="utf-8") + # Record metadata for index A + mgr.meta_json.write_text(json.dumps({"url": "https://a.example/index.json"}), encoding="utf-8") + + with patch("urllib.request.urlopen") as mock_url: + mock_resp = MagicMock() + mock_resp.read.return_value = json.dumps(cached_data).encode("utf-8") + mock_resp.headers = {} + mock_resp.__enter__.return_value = mock_resp + mock_url.return_value = mock_resp + + # 1. Fresh cache (< 24h) from same URL without force -> reuses cache + res = mgr.sync(url="https://a.example/index.json", force=False) + assert res.package_count == 1 + assert "Cache is up-to-date" in res.message + mock_url.assert_not_called() + + # 2. Fresh cache (< 24h) but requested URL is DIFFERENT -> fetches new URL! (Finding 2) + res_diff_url = mgr.sync(url="https://b.example/index.json", force=False) + assert mock_url.called + assert res_diff_url.package_count == 1 + + mock_url.reset_mock() + + # 3. With force=True, urlopen must be called even for same URL + mgr.sync(url="https://b.example/index.json", force=True) + assert mock_url.called + + +def test_index_sync_size_cap_and_lying_content_length(tmp_path): + """Verify max size cap and malformed Content-Length handling.""" + mgr = IndexSyncManager(index_dir=tmp_path) + + # 1. Content-Length header is excessively large + mock_resp = MagicMock() + mock_resp.headers = {"Content-Length": str(MAX_INDEX_SIZE_BYTES + 100)} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + with pytest.raises(IndexSyncError, match="exceeds maximum allowed size"): + mgr.sync(url="https://example.com/index.json", force=True) + + # 2. Lying Content-Length: header claims small size, but stream exceeds limit + mock_resp2 = MagicMock() + mock_resp2.headers = {"Content-Length": "50"} + mock_resp2.read.return_value = b"x" * (MAX_INDEX_SIZE_BYTES + 2) + mock_resp2.__enter__.return_value = mock_resp2 + + with patch("urllib.request.urlopen", return_value=mock_resp2): + with pytest.raises(IndexSyncError, match="exceeded maximum size limit"): + mgr.sync(url="https://example.com/index.json", force=True) + + # 3. Malformed non-numeric Content-Length: does not raise ValueError (Finding 3) + valid_payload = json.dumps([{"name": "test-pkg", "url": "https://example.com/pkg.tar.gz"}]).encode("utf-8") + mock_resp3 = MagicMock() + mock_resp3.headers = {"Content-Length": "not-a-number"} + mock_resp3.read.return_value = valid_payload + mock_resp3.__enter__.return_value = mock_resp3 + + with patch("urllib.request.urlopen", return_value=mock_resp3): + res = mgr.sync(url="https://example.com/index.json", force=True) + assert res.package_count == 1 + + +def test_index_sync_deduplicates_duplicate_names(tmp_path): + """Verify Finding 6: Duplicate names in an index are deduplicated to avoid overwrites and over-reporting.""" + mgr = IndexSyncManager(index_dir=tmp_path) + + duplicate_index = [ + {"name": "dup", "version": "1.0.0", "url": "https://example.com/dup1.tar.gz"}, + {"name": "dup", "version": "9.9.9", "url": "https://example.com/dup9.tar.gz"}, + ] + raw_json = json.dumps(duplicate_index).encode("utf-8") + + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + + assert res.package_count == 1 + assert "Successfully synchronized 1 packages" in res.message + # Verify recipes dir has only 1 file + recipe_files = list(mgr.recipes_dir.glob("*.yaml")) + assert len(recipe_files) == 1 + assert recipe_files[0].name == "dup.yaml" + + # Verify packages.json has only 1 entry + cached_entries = json.loads(mgr.packages_json.read_text(encoding="utf-8")) + assert len(cached_entries) == 1 + + +def test_cli_update_index_fallback_exits_nonzero(tmp_path, monkeypatch): + """Verify Finding 8: CLI update-index logs warning and exits 1 on fallback.""" + custom_dir = tmp_path / "index_cache" + custom_dir.mkdir() + packages_json = custom_dir / "packages.json" + packages_json.write_text(json.dumps([{"name": "stale-pkg", "version": "1.0.0"}]), encoding="utf-8") + monkeypatch.setenv("EBUILD_INDEX_PATH", str(custom_dir)) + + runner = CliRunner() + with patch("urllib.request.urlopen", side_effect=urllib.error.URLError("Network down")): + # Without offline flag, network failure falling back to cache must exit non-zero + result = runner.invoke(cli, ["update-index", "--url", "https://example.com/index.json", "--force"]) + assert result.exit_code == 1 + assert "fell back to cached index" in result.output + + +def test_index_sync_prunes_stale_cached_recipes(tmp_path): + """Verify Finding 5: Cached recipe YAMLs not present in new index are pruned.""" + mgr = IndexSyncManager(index_dir=tmp_path) + + # First sync: index containing 'alpha' + index_a = [ + {"name": "alpha", "version": "1.0.0", "url": "https://example.com/alpha.tar.gz"}, + ] + raw_a = json.dumps(index_a).encode("utf-8") + mock_resp_a = MagicMock() + mock_resp_a.read.return_value = raw_a + mock_resp_a.headers = {"Content-Length": str(len(raw_a))} + mock_resp_a.__enter__.return_value = mock_resp_a + + with patch("urllib.request.urlopen", return_value=mock_resp_a): + res_a = mgr.sync(url="https://example.com/index-a.json", force=True) + assert res_a.package_count == 1 + assert res_a.pruned == 0 + + recipe_files_a = sorted([f.name for f in mgr.recipes_dir.glob("*.yaml")]) + assert recipe_files_a == ["alpha.yaml"] + + # Also add a stale .yml file (Finding 3: stale .yml must not survive sync) + (mgr.recipes_dir / "stale.yml").write_text("package: stale\nversion: 1.0.0", encoding="utf-8") + + # Second sync: index containing 'beta' (alpha withdrawn/absent) + index_b = [ + {"name": "beta", "version": "2.0.0", "url": "https://example.com/beta.tar.gz"}, + ] + raw_b = json.dumps(index_b).encode("utf-8") + mock_resp_b = MagicMock() + mock_resp_b.read.return_value = raw_b + mock_resp_b.headers = {"Content-Length": str(len(raw_b))} + mock_resp_b.__enter__.return_value = mock_resp_b + + with patch("urllib.request.urlopen", return_value=mock_resp_b): + res_b = mgr.sync(url="https://example.com/index-b.json", force=True) + assert res_b.package_count == 1 + # Both alpha.yaml and stale.yml must be counted as pruned + assert res_b.pruned == 2 + + recipe_files_b = sorted([f.name for f in mgr.recipes_dir.iterdir() if f.is_file()]) + # alpha.yaml and stale.yml must be pruned; only beta.yaml must remain! + assert recipe_files_b == ["beta.yaml"] + + +def test_index_sync_preserves_cached_recipe_if_index_entry_lacks_url(tmp_path): + """Verify Finding 1: An index entry that loses URL still preserves cached recipe file.""" + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + (mgr.recipes_dir / "preserved.yaml").write_text( + "package: preserved\nversion: 1.0.0\nurl: https://example.com/preserved.tar.gz\nbuild: cmake\n", + encoding="utf-8", + ) + + index_data = [ + {"name": "preserved", "version": "1.0.0", "description": "Entry without download URL"}, + ] + raw_json = json.dumps(index_data).encode("utf-8") + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + assert res.pruned == 0 + assert (mgr.recipes_dir / "preserved.yaml").is_file() + + +def test_cli_update_index_reports_sha256_digest(tmp_path, monkeypatch): + """Verify Finding 6: CLI update-index surfaces the SHA-256 digest and updates.""" + custom_dir = tmp_path / "index_cache" + monkeypatch.setenv("EBUILD_INDEX_PATH", str(custom_dir)) + + index_data = [ + {"name": "gamma", "version": "1.0.0", "url": "https://example.com/gamma.tar.gz"}, + ] + raw_json = json.dumps(index_data).encode("utf-8") + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + runner = CliRunner() + with patch("urllib.request.urlopen", return_value=mock_resp): + result = runner.invoke(cli, ["update-index", "--url", "https://example.com/index.json", "--force"]) + assert result.exit_code == 0 + assert "Index SHA-256 digest:" in result.output + + +def test_cli_update_index_reports_pruned_count(tmp_path, monkeypatch): + """Verify Finding 2: CLI update-index surfaces count of pruned stale recipes.""" + custom_dir = tmp_path / "index_cache" + recipes_dir = custom_dir / "recipes" + recipes_dir.mkdir(parents=True) + (recipes_dir / "old.yaml").write_text("package: old\nversion: 1.0.0\n", encoding="utf-8") + monkeypatch.setenv("EBUILD_INDEX_PATH", str(custom_dir)) + + index_data = [ + {"name": "newpkg", "version": "1.0.0", "url": "https://example.com/newpkg.tar.gz"}, + ] + raw_json = json.dumps(index_data).encode("utf-8") + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + runner = CliRunner() + with patch("urllib.request.urlopen", return_value=mock_resp): + result = runner.invoke(cli, ["update-index", "--url", "https://example.com/index.json", "--force"]) + assert result.exit_code == 0 + assert "Pruned 1 stale cached recipe(s)" in result.output + + +def test_index_sync_empty_index_preserves_cache_and_warns(tmp_path): + """Verify Finding 1 & Finding 2: An empty index payload refuses to prune cached recipes, does not overwrite metadata, and reports fallback.""" + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + + # Seed recipes + (mgr.recipes_dir / "alpha.yaml").write_text("package: alpha\nversion: 1.0.0\n", encoding="utf-8") + (mgr.recipes_dir / "bravo.yaml").write_text("package: bravo\nversion: 1.0.0\n", encoding="utf-8") + # Seed packages.json + seed_json = [{"name": "alpha", "version": "1.0.0"}, {"name": "bravo", "version": "1.0.0"}] + mgr.packages_json.write_text(json.dumps(seed_json), encoding="utf-8") + + # Seed index-meta.json and sha256 sidecar + initial_meta = { + "url": "https://example.com/index.json", + "sha256": "initial_sha_12345", + "synced_at": 1000.0, + } + mgr.meta_json.write_text(json.dumps(initial_meta), encoding="utf-8") + sha_file = mgr.packages_json.with_name(mgr.packages_json.name + ".sha256") + sha_file.write_text("initial_sha_12345 packages.json\n", encoding="utf-8") + + raw_json = json.dumps([]).encode("utf-8") + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + assert res.pruned == 0 + assert res.package_count == 2 + assert res.is_fallback is True + + assert (mgr.recipes_dir / "alpha.yaml").is_file() + assert (mgr.recipes_dir / "bravo.yaml").is_file() + # packages.json must NOT be overwritten with [] + cached = json.loads(mgr.packages_json.read_text(encoding="utf-8")) + assert len(cached) == 2 + + # index-meta.json and packages.json.sha256 must NOT be overwritten with [] metadata + meta_after = json.loads(mgr.meta_json.read_text(encoding="utf-8")) + assert meta_after["sha256"] == "initial_sha_12345" + assert meta_after["synced_at"] == 1000.0 + assert sha_file.read_text(encoding="utf-8") == "initial_sha_12345 packages.json\n" + + # Subsequent non-forced sync must retry via network (not short-circuited by 24h TTL) + with patch("urllib.request.urlopen", return_value=mock_resp) as mock_retry: + res_retry = mgr.sync(url="https://example.com/index.json", force=False) + assert mock_retry.called + assert res_retry.is_fallback is True + + +def test_index_sync_unparseable_entries_preserves_cache(tmp_path): + """Verify Finding 1: An index with no usable entries refuses to prune cached recipes and reports fallback.""" + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + + (mgr.recipes_dir / "alpha.yaml").write_text("package: alpha\nversion: 1.0.0\n", encoding="utf-8") + (mgr.recipes_dir / "bravo.yaml").write_text("package: bravo\nversion: 1.0.0\n", encoding="utf-8") + + raw_json = json.dumps([{"nope": 1}, "str", 5]).encode("utf-8") + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + assert res.pruned == 0 + assert res.package_count == 0 + assert res.is_fallback is True + + assert (mgr.recipes_dir / "alpha.yaml").is_file() + assert (mgr.recipes_dir / "bravo.yaml").is_file() + + +def test_index_sync_preserves_yml_if_not_superseded_and_prunes_if_superseded(tmp_path): + """Verify Finding 5: Cached .yml is kept when index names entry without URL, and pruned when superseded by .yaml.""" + mgr = IndexSyncManager(index_dir=tmp_path) + mgr.ensure_directories() + + # Pre-existing .yml recipes in cache + (mgr.recipes_dir / "eps.yml").write_text("package: eps\nversion: 1.0.0\n", encoding="utf-8") + (mgr.recipes_dir / "delta.yml").write_text("package: delta\nversion: 9.9.9\n", encoding="utf-8") + + # Index lists eps (no url -> no .yaml written) and delta (with url -> delta.yaml written) + index_data = [ + {"name": "eps", "version": "1.0.0", "description": "Entry without download URL"}, + {"name": "delta", "version": "2.0.0", "url": "https://example.com/delta.tar.gz"}, + ] + raw_json = json.dumps(index_data).encode("utf-8") + mock_resp = MagicMock() + mock_resp.read.return_value = raw_json + mock_resp.headers = {"Content-Length": str(len(raw_json))} + mock_resp.__enter__.return_value = mock_resp + + with patch("urllib.request.urlopen", return_value=mock_resp): + res = mgr.sync(url="https://example.com/index.json", force=True) + # delta.yml is pruned because delta.yaml superseded it; eps.yml is NOT pruned + assert res.pruned == 1 + + # eps.yml survived + assert (mgr.recipes_dir / "eps.yml").is_file() + # delta.yml was pruned and replaced by delta.yaml + assert not (mgr.recipes_dir / "delta.yml").exists() + assert (mgr.recipes_dir / "delta.yaml").is_file() + + +def test_changelog_trailing_newline(): + """Verify Finding 2 & Finding 6: CHANGELOG.md ends with a trailing newline.""" + changelog_path = Path(__file__).resolve().parents[2] / "CHANGELOG.md" + assert changelog_path.is_file() + content = changelog_path.read_bytes() + assert content.endswith(b"\n") + diff --git a/tests/unit/test_package_search.py b/tests/unit/test_package_search.py new file mode 100644 index 00000000..b8bff54e --- /dev/null +++ b/tests/unit/test_package_search.py @@ -0,0 +1,207 @@ +# SPDX-License-Identifier: MIT +# Copyright (c) 2026 EoS Project + +"""Unit tests for package discovery and search across repository sources.""" + +import json +from click.testing import CliRunner + +from ebuild.cli.commands import cli +from ebuild.packages.index_sync import IndexSyncManager +from ebuild.packages.repository import PackageRepository + + +def test_package_repository_search(tmp_path): + repo = PackageRepository() + + # Create dummy local recipe directory + recipe_dir = tmp_path / "recipes" + recipe_dir.mkdir() + (recipe_dir / "my_crypto.yaml").write_text( + """package: my_crypto +version: "1.0.0" +description: "Embedded cryptography primitives" +license: Apache-2.0 +url: https://example.com/crypto.tar.gz +checksum: sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +build: cmake +""", + encoding="utf-8", + ) + + repo.add_recipe_directory(recipe_dir) + assert repo.package_count == 1 + + # Search by keyword + results = repo.search("crypto") + assert len(results) == 1 + assert results[0].name == "my_crypto" + + # Search by license filter + assert len(repo.search("", license_filter="Apache")) == 1 + assert len(repo.search("", license_filter="GPL")) == 0 + + # Search by build system + assert len(repo.search("", build_system="cmake")) == 1 + assert len(repo.search("", build_system="meson")) == 0 + + +def test_package_source_precedence_project_wins(tmp_path): + """Verify Finding 1: Project-local recipe takes precedence over shipped and cached remote recipes.""" + # 1. Create project-local recipe + proj_dir = tmp_path / "project" + proj_recipes = proj_dir / "recipes" + proj_recipes.mkdir(parents=True) + (proj_recipes / "cjson.yaml").write_text( + """package: cjson +version: "1.7.18" +description: "Project pinned cjson" +url: "https://custom-project.org/cjson-PROJECT.tar.gz" +checksum: "sha256:1111111111111111111111111111111111111111111111111111111111111111" +build: cmake +""", + encoding="utf-8", + ) + + # 2. Create remote cached index with a higher version (9.9.9) for cjson + remote_index_dir = tmp_path / "remote_index" + remote_recipes = remote_index_dir / "recipes" + remote_recipes.mkdir(parents=True) + (remote_recipes / "cjson.yaml").write_text( + """package: cjson +version: "9.9.9" +description: "Remote cjson" +url: "https://remote-registry.org/cjson-REMOTE.tar.gz" +checksum: "sha256:2222222222222222222222222222222222222222222222222222222222222222" +build: cmake +""", + encoding="utf-8", + ) + # Also in packages.json + packages_json = remote_index_dir / "packages.json" + packages_json.write_text( + json.dumps([ + { + "name": "cjson", + "version": "9.9.9", + "url": "https://remote-registry.org/cjson-REMOTE.tar.gz", + "checksum": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + } + ]), + encoding="utf-8", + ) + + # 1. Search path verification (PackageRepository) + sync_mgr = IndexSyncManager(index_dir=remote_index_dir) + repo = PackageRepository(sync_manager=sync_mgr) + repo.load_all_sources(project_dir=proj_dir) + + pkg = repo.info("cjson") + assert pkg is not None + # Must resolve to the PROJECT-local version, url, and checksum! + assert pkg.version == "1.7.18" + assert pkg.url == "https://custom-project.org/cjson-PROJECT.tar.gz" + assert pkg.checksum == "sha256:1111111111111111111111111111111111111111111111111111111111111111" + assert pkg.description == "Project pinned cjson" + + # 2. Build path verification (_find_recipe_dirs + create_registry) + from ebuild.cli.commands import _find_recipe_dirs + from ebuild.packages.registry import create_registry + + recipe_dirs = _find_recipe_dirs(proj_dir, remote_index_dir=remote_index_dir) + registry = create_registry(*recipe_dirs) + + # Explicit version lookup + build_recipe = registry.get("cjson", "1.7.18") + assert build_recipe is not None + assert build_recipe.url == "https://custom-project.org/cjson-PROJECT.tar.gz" + assert build_recipe.checksum == "sha256:1111111111111111111111111111111111111111111111111111111111111111" + + # Unpinned version lookup: project-local recipe must win over higher remote version! + unpinned_recipe = registry.get("cjson") + assert unpinned_recipe is not None + assert unpinned_recipe.version == "1.7.18" + assert unpinned_recipe.url == "https://custom-project.org/cjson-PROJECT.tar.gz" + assert unpinned_recipe.checksum == "sha256:1111111111111111111111111111111111111111111111111111111111111111" + + +def test_transitive_dependency_source_precedence_project_wins(tmp_path): + """Verify Finding 1: Transitive dependencies resolve to project-local definitions even when remote has higher version.""" + from ebuild.cli.commands import _find_recipe_dirs + from ebuild.packages.registry import create_registry + from ebuild.packages.resolver import PackageResolver + + proj_dir = tmp_path / "project" + proj_recipes = proj_dir / "recipes" + proj_recipes.mkdir(parents=True) + (proj_recipes / "app.yaml").write_text( + """package: app +version: "1.0.0" +url: "https://custom-project.org/app-1.0.0.tar.gz" +checksum: "sha256:0000000000000000000000000000000000000000000000000000000000000000" +build: cmake +dependencies: + - mbedtls +""", + encoding="utf-8", + ) + (proj_recipes / "mbedtls.yaml").write_text( + """package: mbedtls +version: "3.6.0" +url: "https://custom-project.org/mbedtls-PROJECT.tar.gz" +checksum: "sha256:1111111111111111111111111111111111111111111111111111111111111111" +build: cmake +""", + encoding="utf-8", + ) + + remote_index_dir = tmp_path / "remote_index" + remote_recipes = remote_index_dir / "recipes" + remote_recipes.mkdir(parents=True) + (remote_recipes / "mbedtls.yaml").write_text( + """package: mbedtls +version: "9.9.9" +url: "https://remote-registry.org/mbedtls-REMOTE.tar.gz" +checksum: "sha256:2222222222222222222222222222222222222222222222222222222222222222" +build: cmake +""", + encoding="utf-8", + ) + + recipe_dirs = _find_recipe_dirs(proj_dir, remote_index_dir=remote_index_dir) + registry = create_registry(*recipe_dirs) + resolver = PackageResolver(registry) + + # Resolve top-level app pinned to 1.0.0; mbedtls is resolved transitively (unpinned) + resolved = resolver.resolve([{"name": "app", "version": "1.0.0"}]) + by_name = {r.name: r for r in resolved} + + assert "mbedtls" in by_name + mbedtls = by_name["mbedtls"] + # Must resolve to project recipe 3.6.0, NOT remote 9.9.9! + assert mbedtls.version == "3.6.0" + assert mbedtls.url == "https://custom-project.org/mbedtls-PROJECT.tar.gz" + assert mbedtls.checksum == "sha256:1111111111111111111111111111111111111111111111111111111111111111" + + +def test_cli_search_command(tmp_path): + runner = CliRunner() + + result = runner.invoke(cli, ["search", "cjson"]) + assert result.exit_code == 0 + assert "cjson" in result.output + + # JSON output + json_result = runner.invoke(cli, ["search", "cjson", "--json"]) + assert json_result.exit_code == 0 + data = json.loads(json_result.output) + assert isinstance(data, list) + assert any(p["name"] == "cjson" for p in data) + + +def test_cli_update_index_offline(): + runner = CliRunner() + result = runner.invoke(cli, ["update-index", "--offline"]) + assert result.exit_code == 0 + assert "Offline mode" in result.output +