feat: parse command

README.md

@@ -2,7 +2,7 @@
 
 Python parser for [`***plain`](https://www.plainlang.org/docs/) specification files.
 
-Reads a `.plain`  module and its `import` / `requires` chain, resolves Liquid templates,
+Reads a `.plain` module and its `import` / `requires` chain, resolves Liquid templates,
 validates concepts and linked resources, and returns the marshalled specification tree
 plus the ordered list of functionalities (FRIDs).
 
@@ -12,21 +12,70 @@ plus the ordered list of functionalities (FRIDs).
 pip install plain-parser
 ```
 
-## Usage
+## Command line
 
-Validate a module and its `requires` chain from the command line (`<file>: OK` and exit `0` when valid; `Error: …` on stderr and exit `1` otherwise):
+```
+plain-parser check <file.plain> [--template-dir DIR] [--config-name NAME]
+plain-parser parse <file.plain> [--template-dir DIR] [--config-name NAME]
+```
 
-```bash
-plain-parser check my_spec.plain [--template-dir DIR] [--config-name NAME]
+Both commands parse the module and its `requires` chain, check that every linked resource is an existing
+file (not a directory), and walk the module's functionalities. They stop at the first error.
+
+### `check`
+
+Validates the module. In addition to the steps above it reads every linked file and rejects binary files and
+files holding a large base64 blob.
+
+```
+$ plain-parser check my_spec.plain
+my_spec.plain: OK
 ```
 
+Exit `0` when the module is valid. On an error, `Error: …` is written to stderr and the exit code is `1`.
+
+### `parse`
+
+Prints the parse result as one UTF-8 JSON document on stdout. Linked files are located but not read.
+
+```
+$ plain-parser parse my_spec.plain
+{
+  "format_version": 1,
+  "modules": [{"module": "...", "tree": {...}}, ...],
+  "resources": {...}
+}
+```
+
+- `format_version`: `1`.
+- `modules`: every module in the `requires` chain, in build order: deepest ancestor first, the module itself last.
+  Each entry has the module name and its specification tree, in the shape described under *Python* below.
+- `resources`: every linked-resource target in the chain, as written in the spec, mapped to the path it resolves
+  to, relative to the entry module's directory, with `/` as separator: `notes.md` when found next to the module,
+  `../tpl/notes.md` when found in a template directory. The directory part is lexical (symlinks not resolved) and
+  the target is appended as written (`..` kept), so joining the value onto the entry module's directory names the
+  file the parser matched. When no relative path exists (another drive on Windows) the value is the absolute path,
+  still with `/`. Two spellings of one file (`notes.md`, `./notes.md`) are two keys.
+
+Exit `0` on success. On a parser error the JSON is `{"format_version": 1, "type": "<error class>",
+"message": "…"}` and the exit code is `1`.
+
+### Options and exit codes
+
 Modules and `{% include %}` templates are looked up in the `.plain` file's directory, then in `--template-dir`.
 When `--template-dir` is not given, it is read from the `template-dir` key of `config.yaml` (or the file named by
-`--config-name`), found next to the `.plain` file or in the working directory. Finding it in both is a usage error
-(exit `2`). The config file is read even when `--template-dir` is given, so a malformed one fails the run either way.
+`--config-name`), found next to the `.plain` file or in the working directory. Finding it in both is a usage
+error. The config file is read even when `--template-dir` is given, so a malformed one fails the run either way.
 Other config keys are ignored.
 
-Or from Python:
+| Exit code | Meaning |
+|---|---|
+| `0` | Success |
+| `1` | The module, its chain, or a linked resource is invalid |
+| `2` | Usage error, including an ambiguous or malformed config file |
+| `130` | Interrupted |
+
+## Python
 
 ```python
 from plain_parser import plain_file_parser

src/plain_parser/cli.py

@@ -1,15 +1,18 @@
-"""Command-line entry point: ``plain-parser check <file.plain> [--template-dir DIR] [--config-name NAME]``."""
+"""Command-line entry point: ``plain-parser {check,parse} <file.plain> [--template-dir DIR] [--config-name NAME]``."""
 
 import argparse
+import json
 import os
 import sys
+from dataclasses import dataclass
 
 import yaml
 
 from plain_parser import loaders, plain_file, plain_spec
 
 DEFAULT_CONFIG_NAME = "config.yaml"
 TEMPLATE_DIR_CONFIG_KEYS = ("template_dir", "template-dir")
+PARSE_FORMAT_VERSION = 1
 
 
 class AmbiguousConfigFileError(Exception):
@@ -20,19 +23,29 @@ def build_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(prog="plain-parser", description="Tools for ***plain specification files.")
     subparsers = parser.add_subparsers(dest="command", required=True)
 
-    check_parser = subparsers.add_parser("check", help="Validate a .plain module and its requires chain.")
-    check_parser.add_argument("plain_file", help="Path to the .plain module.")
-    check_parser.add_argument(
-        "--template-dir",
-        help="Directory searched for modules and templates after the .plain file's own directory. "
-        "Overrides the template-dir value of the config file.",
+    check_parser = subparsers.add_parser(
+        "check",
+        help="Validate a .plain module and its requires chain, and read every linked file: "
+        "binary files and large base64 blobs are rejected.",
     )
-    check_parser.add_argument(
-        "--config-name",
-        default=DEFAULT_CONFIG_NAME,
-        help="Name of the config file to read template-dir from. Looked up in the .plain file's directory "
-        "and the current working directory. Defaults to %(default)s.",
+    parse_parser = subparsers.add_parser(
+        "parse",
+        help="Print the parse result of a .plain module and its requires chain as JSON. Linked files are "
+        "located but never read.",
     )
+    for subparser in (check_parser, parse_parser):
+        subparser.add_argument("plain_file", help="Path to the .plain module.")
+        subparser.add_argument(
+            "--template-dir",
+            help="Directory searched for modules and templates after the .plain file's own directory. "
+            "Overrides the template-dir value of the config file.",
+        )
+        subparser.add_argument(
+            "--config-name",
+            default=DEFAULT_CONFIG_NAME,
+            help="Name of the config file to read template-dir from. Looked up in the .plain file's directory "
+            "and the current working directory. Defaults to %(default)s.",
+        )
     return parser
 
 
@@ -71,23 +84,124 @@ def template_dir_from_config(config_file: str) -> str | None:
     return None
 
 
-def check(plain_file_path: str, template_dir: str | None) -> None:
-    """Raise if the module, its requires chain, or its linked resources are invalid."""
+@dataclass
+class ParseResult:
+    modules: list[
+        tuple[str, dict]
+    ]  # (module name, plain source tree) for the chain, deepest ancestor first, entry last.
+    # Each link target in the chain, as written, to the path it resolves to, relative to the entry module's dir
+    # with `/` as separator (loaders.relative_resource_path).
+    resources: dict[str, str]
+
+
+def parse_spec(plain_file_path: str, template_dir: str | None, load_resources: bool) -> ParseResult:
+    """Parse the module and its requires chain, resolve its links, and walk the entry's FRIDs; raise on the first error.
+
+    With ``load_resources``, each module's linked files are also read and checked (text only, no large base64 blob) right
+    after that module's links are collected, so a resource error is reported before a FRID-walk error.
+    """
+    # Absolute, so the chain parse and the resource paths below search the same directories.
     template_dirs = [os.path.dirname(os.path.abspath(plain_file_path))]
     if template_dir:
-        template_dirs.append(template_dir)
+        template_dirs.append(os.path.abspath(template_dir))
 
     chain = plain_file.parse_module_chain(os.path.basename(plain_file_path), template_dirs)
 
+    # The target joined as written onto its template dir (`..` kept, symlinks not resolved), made relative to the
+    # entry module's dir so the result does not depend on where the project sits on the machine.
+    resources: dict[str, str] = {}
     for module_name, plain_source_tree in chain:
         resources_list: list[dict] = []
         plain_spec.collect_linked_resources(plain_source_tree, resources_list, None, True)
-        loaders.load_linked_resources(template_dirs, resources_list, module_name)
+        if load_resources:
+            loaders.load_linked_resources(template_dirs, resources_list, module_name)
+        for resource in resources_list:
+            target = resource["target"]
+            resource_dir = loaders.find_resource_dir(template_dirs, target)
+            resources[target] = loaders.relative_resource_path(template_dirs[0], resource_dir, target)
 
     _, top_plain_source_tree = chain[-1]
     for frid in plain_spec.get_frids(top_plain_source_tree):
         plain_spec.get_specifications_for_frid(top_plain_source_tree, frid)
 
+    return ParseResult(chain, resources)
+
+
+def check(plain_file_path: str, template_dir: str | None) -> None:
+    """Raise if the module, its requires chain, or its linked resources are invalid."""
+    parse_spec(plain_file_path, template_dir, load_resources=True)
+
+
+def _error_output(type_name: str, message: str) -> dict:
+    return {"format_version": PARSE_FORMAT_VERSION, "type": type_name, "message": message}
+
+
+def _escape(text: str) -> str:
+    """The text with every non-ASCII character as a backslash escape, so it always encodes."""
+    return text.encode("ascii", "backslashreplace").decode("ascii")
+
+
+def _encode_output(output: dict) -> bytes:
+    """The output as UTF-8 JSON; raise if it is not JSON-serialisable or holds text that is not valid UTF-8."""
+    return (json.dumps(output, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
+
+
+def parse(plain_file_path: str, template_dir: str | None) -> int:
+    """Print the parse result, or the parser error, as one UTF-8 JSON document on stdout; return the exit code."""
+    output: dict
+    try:
+        result = parse_spec(plain_file_path, template_dir, load_resources=False)
+        output = {
+            "format_version": PARSE_FORMAT_VERSION,
+            "modules": [{"module": module_name, "tree": tree} for module_name, tree in result.modules],
+            "resources": result.resources,
+        }
+        exit_code = 0
+    except Exception as e:
+        output = _error_output(type(e).__name__, str(e))
+        exit_code = 1
+
+    try:
+        data = _encode_output(output)
+    except UnicodeEncodeError as e:
+        # A file or directory name that is not valid UTF-8, which is allowed on Linux.
+        if exit_code == 1:
+            # A parser error whose message names such a file. Keep its type and escape the name.
+            output = _error_output(output["type"], _escape(output["message"]))
+        else:
+            # A valid result that contains such a name. Report the output line that contains it.
+            line_start = e.object.rfind("\n", 0, e.start) + 1
+            line_end = e.object.find("\n", e.end)
+            line = e.object[line_start : line_end if line_end != -1 else None].strip().rstrip(",")
+            output = _error_output(
+                type(e).__name__, f"The parse result holds text that is not valid UTF-8: {_escape(line)}"
+            )
+        data = _encode_output(output)
+        exit_code = 1
+    except Exception as e:
+        # A value in the result that JSON cannot represent. Report it as a JSON error.
+        data = _encode_output(_error_output(type(e).__name__, _escape(str(e))))
+        exit_code = 1
+
+    # Write the UTF-8 bytes to stdout's byte stream, because sys.stdout encodes text with the system encoding, which
+    # may not be UTF-8.
+    buffer = getattr(sys.stdout, "buffer", None)
+    if buffer is None:
+        # A Python caller replaced stdout with a stream that takes only text.
+        sys.stdout.write(data.decode("utf-8"))
+        return exit_code
+
+    try:
+        sys.stdout.flush()  # Send out anything already written to sys.stdout first.
+        buffer.write(data)
+        buffer.flush()
+    except BrokenPipeError:
+        # The reader closed the pipe early. Send stdout to /dev/null, so Python's flush at exit does not fail again,
+        # and keep the exit code.
+        devnull = os.open(os.devnull, os.O_WRONLY)
+        os.dup2(devnull, sys.stdout.fileno())
+    return exit_code
+
 
 def main(argv: list[str] | None = None) -> int:
     parser = build_parser()
@@ -96,6 +210,8 @@ def main(argv: list[str] | None = None) -> int:
     try:
         # The config file is resolved even when --template-dir is given, so an ambiguous config is always an error.
         config_template_dir = _template_dir_from_config(parser, args)
+        if args.command == "parse":
+            return parse(args.plain_file, args.template_dir or config_template_dir)
         check(args.plain_file, args.template_dir or config_template_dir)
     except KeyboardInterrupt:
         return 130

src/plain_parser/loaders.py

@@ -36,20 +36,41 @@ def open_from(dirs, file_name):
     return None
 
 
-def resolve_linked_resource(dirs, file_name):
-    """Return the first existing path for file_name across dirs, or None if not found.
+def find_resource_dir(dirs, file_name):
+    """Return the first dir in which file_name exists, or None if not found.
 
     Mirrors the search order used by open_from so that validating a linked resource and
     loading it can never disagree.
     """
     for dir in dirs:
-        full_file_name = os.path.join(dir, file_name)
-        if os.path.exists(full_file_name):
-            return full_file_name
+        if os.path.exists(os.path.join(dir, file_name)):
+            return dir
 
     return None
 
 
+def resolve_linked_resource(dirs, file_name):
+    """Return the first existing path for file_name across dirs, or None if not found."""
+    resource_dir = find_resource_dir(dirs, file_name)
+    return None if resource_dir is None else os.path.join(resource_dir, file_name)
+
+
+def relative_resource_path(base_dir: str, resource_dir: str, target: str) -> str:
+    """The path of target in resource_dir, relative to base_dir, with `/` as separator.
+
+    The dir part is computed lexically (symlinks not resolved) and the target is appended as written (`..` kept),
+    so joining the result onto base_dir names the same file as os.path.join(resource_dir, target). When no relative
+    path exists (another drive on Windows), the absolute path is returned instead, still with `/`.
+    """
+    try:
+        rel_dir = os.path.relpath(resource_dir, base_dir)
+    except ValueError:
+        return os.path.join(resource_dir, target).replace(os.sep, "/")
+    if rel_dir == os.curdir:
+        return target
+    return rel_dir.replace(os.sep, "/") + "/" + target
+
+
 def load_linked_resources(template_dirs: list[str], resources_list, module_name: str):
     linked_resources = {}
 

tests/data/cli/parent_link/a/spec/top.plain

@@ -0,0 +1,7 @@
+***implementation reqs***
+
+- A req.
+
+***functional specs***
+
+- A functionality. See [the notes](../shared/notes.md).

tests/data/cli/parent_link/b/shared/notes.md

@@ -0,0 +1 @@
+notes

tests/data/cli/resource_before_frid_walk/base.plain

@@ -0,0 +1,8 @@
+***implementation reqs***
+
+- :Implementation: is written in Python.
+- :Implementation: follows [the data](../binary.bin).
+
+***functional specs***
+
+- :Implementation: has an entry point.

tests/data/cli/resource_before_frid_walk/snippet.plain

@@ -0,0 +1 @@
+the variable {{ name | code_variable }}

tests/data/cli/resource_before_frid_walk/top.plain

@@ -0,0 +1,14 @@
+---
+requires:
+  - base
+---
+
+***implementation reqs***
+
+- :Implementation: is written in Python.
+
+***functional specs***
+
+- :Implementation: names {% include "snippet.plain", name: "first" %}.
+
+- :Implementation: names {% include "snippet.plain", name: "second" %}.

tests/data/cli/shared_link_helper.plain

@@ -0,0 +1,7 @@
+***implementation reqs***
+
+- A req.
+
+***functional specs***
+
+- A functionality. See [the notes](notes.md).

tests/data/cli/shared_link_top.plain

@@ -0,0 +1,12 @@
+---
+requires:
+  - shared_link_helper
+---
+
+***implementation reqs***
+
+- A req.
+
+***functional specs***
+
+- A functionality. Also [the notes](notes.md).

tests/data/cli/template_link/spec/top.plain

@@ -0,0 +1,7 @@
+***implementation reqs***
+
+- A req.
+
+***functional specs***
+
+- A functionality. See [the notes](notes.md).

tests/data/cli/template_link/tpl/notes.md

@@ -0,0 +1 @@
+notes