feat: explicit test scenarios

README.md

@@ -2,9 +2,9 @@
 
 Python parser for [`***plain`](https://www.plainlang.org/docs/) specification files.
 
-Reads a `.plain` module and its `import` / `requires` chain, resolves Liquid templates,
-validates concepts and linked resources, and returns the marshalled specification tree
-plus the ordered list of functionalities (FRIDs).
+Reads a `.plain` module with its `import` / `requires` chain and its test plain files, resolves Liquid templates,
+validates concepts and linked resources, and returns the marshalled specification tree of every module in the
+chain, with the test scenarios of each functionality merged in.
 
 ## Install
 
@@ -15,24 +15,29 @@ pip install plain-parser
 ## Command line
 
 ```
-plain-parser check <file.plain> [--template-dir DIR] [--config-name NAME]
-plain-parser parse <file.plain> [--template-dir DIR] [--config-name NAME]
+plain-parser check <file.plain> [--template-dir DIR] [--test-dir DIR]... [--config-name NAME]
+plain-parser parse <file.plain> [--template-dir DIR] [--test-dir DIR]... [--config-name NAME]
 ```
 
-Both commands parse the module and its `requires` chain, check that every linked resource is an existing
-file (not a directory), and walk the module's functionalities. They stop at the first error.
+Both commands parse the module and its `requires` chain, bind the test plain files under the test directories to the
+functionalities, check that every linked resource is an existing file (not a directory), and walk the module's
+functionalities. They stop at the first error.
 
 ### `check`
 
 Validates the module. In addition to the steps above it reads every linked file and rejects binary files and
-files holding a large base64 blob.
+files holding a large base64 blob. When the config file names a `conformance-tests-script`, every functionality
+of every module in the chain needs at least one conformance scenario.
 
 ```
 $ plain-parser check my_spec.plain
 my_spec.plain: OK
 ```
 
-Exit `0` when the module is valid. On an error, `Error: …` is written to stderr and the exit code is `1`.
+Exit `0` when the module is valid. On an error, `Error: …` is written to stderr and the exit code is `1`. The
+message names the file, the line where known, and the skill that fixes the problem: the old syntax (a flush-left
+`- ` list in `***functional specs***`, an `***acceptance tests***` section in a core spec) names
+`migrate-plain-syntax`, and a functionality without a conformance scenario names `add-test-scenarios`.
 
 ### `parse`
 
@@ -41,23 +46,27 @@ Prints the parse result as one UTF-8 JSON document on stdout. Linked files are l
 ```
 $ plain-parser parse my_spec.plain
 {
-  "format_version": 1,
+  "format_version": 2,
   "modules": [{"module": "...", "tree": {...}}, ...],
-  "resources": {...}
+  "resources": {...},
+  "test_plain_files": {...}
 }
 ```
 
-- `format_version`: `1`.
+- `format_version`: `2`.
 - `modules`: every module in the `requires` chain, in build order: deepest ancestor first, the module itself last.
   Each entry has the module name and its specification tree, in the shape described under *Python* below.
-- `resources`: every linked-resource target in the chain, as written in the spec, mapped to the path it resolves
-  to, relative to the entry module's directory, with `/` as separator: `notes.md` when found next to the module,
-  `../tpl/notes.md` when found in a template directory. The directory part is lexical (symlinks not resolved) and
-  the target is appended as written (`..` kept), so joining the value onto the entry module's directory names the
-  file the parser matched. When no relative path exists (another drive on Windows) the value is the absolute path,
-  still with `/`. Two spellings of one file (`notes.md`, `./notes.md`) are two keys.
-
-Exit `0` on success. On a parser error the JSON is `{"format_version": 1, "type": "<error class>",
+- `resources`: every linked-resource target in the chain and its test plain files, as written in the spec, mapped
+  to the path it resolves to, relative to the entry module's directory, with `/` as separator: `notes.md` when
+  found next to the module, `../tpl/notes.md` when found in a template directory. The directory part is lexical
+  (symlinks not resolved) and the target is appended as written (`..` kept), so joining the value onto the entry
+  module's directory names the file the parser matched. When no relative path exists (another drive on Windows)
+  the value is the absolute path, still with `/`. Two spellings of one file (`notes.md`, `./notes.md`) are two
+  keys.
+- `test_plain_files`: every test plain file read, by its path relative to the entry module's directory with `/`
+  as separator, mapped to `{"module": ..., "func_id": ...}`, the functionality it binds to.
+
+Exit `0` on success. On a parser error the JSON is `{"format_version": 2, "type": "<error class>",
 "message": "…"}` and the exit code is `1`.
 
 ### Options and exit codes
@@ -66,27 +75,37 @@ Modules and `{% include %}` templates are looked up in the `.plain` file's direc
 When `--template-dir` is not given, it is read from the `template-dir` key of `config.yaml` (or the file named by
 `--config-name`), found next to the `.plain` file or in the working directory. Finding it in both is a usage
 error. The config file is read even when `--template-dir` is given, so a malformed one fails the run either way.
-Other config keys are ignored.
+
+Test plain files are the `.plain` files under the test directories: those given with `--test-dir` (repeat the
+option for more than one), else the `test-dir` key of the config file (a string or a list), else `tests` next to
+the `.plain` file. A test directory that does not exist is skipped. Linked resources in a scenario are looked up
+as in a module. The `conformance-tests-script` key of the config file turns on the check that every
+functionality has a conformance scenario. Other config keys are ignored.
 
 | Exit code | Meaning |
 |---|---|
 | `0` | Success |
-| `1` | The module, its chain, or a linked resource is invalid |
+| `1` | The module, its chain, a test plain file, or a linked resource is invalid |
 | `2` | Usage error, including an ambiguous or malformed config file |
 | `130` | Interrupted |
 
 ## Python
 
 ```python
-from plain_parser import plain_file_parser
+from plain_parser import plain_file_parser, project
 
 module_name, plain_source_tree, required_modules = plain_file_parser("my_spec.plain", ["."])
+
+result = project.parse_project("my_spec.plain", ["."], ["tests"], require_conformance_scenarios=True)
+chain = result.chain  # (module name, plain source tree) pairs, deepest ancestor first, the module itself last
 ```
 
-`plain_source_tree` is a dict keyed by section (`definitions`, `implementation reqs`,
-`test reqs`, `functional specs`), each a list of `{"markdown": ..., "linked_resources": [...]}`
-entries. `plain_parser.plain_spec` has helpers for walking functionalities
-(`get_frids`, `get_specifications_for_frid`, `collect_linked_resources`).
+`plain_source_tree` is a dict keyed by section. `definitions`, `implementation reqs` and `test reqs` are lists
+of `{"markdown": ..., "linked_resources": [...]}` entries. `functional specs` is a list of
+`{"id", "markdown", "linked_resources", "conformance_tests", "acceptance_tests"}` entries, one per `# <func-id>`
+heading in file order; each test entry is `{"id", "text", "linked_resources"}` with the scenario text as written.
+`plain_parser.plain_spec` has helpers for walking functionalities by func-id (`get_frids`,
+`get_specifications_for_frid`, `collect_linked_resources`).
 
 ## Development
 

src/plain_parser/__init__.py

@@ -1,6 +1,6 @@
 """Parser for ***plain specification files."""
 
-from plain_parser import concept_utils, exceptions, loaders, plain_file, plain_spec
+from plain_parser import concept_utils, exceptions, loaders, plain_file, plain_spec, project, test_plain_file
 from plain_parser.plain_file import (
     PlainFileParseResult,
     get_filename_from_module_name,
@@ -26,4 +26,6 @@
     "plain_file",
     "plain_file_parser",
     "plain_spec",
+    "project",
+    "test_plain_file",
 ]

src/plain_parser/cli.py

@@ -1,18 +1,22 @@
-"""Command-line entry point: ``plain-parser {check,parse} <file.plain> [--template-dir DIR] [--config-name NAME]``."""
+"""Command-line entry point: ``plain-parser {check,parse} <file.plain> [--template-dir DIR] [--test-dir DIR]...
+[--config-name NAME]``."""
 
 import argparse
 import json
 import os
 import sys
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 
 import yaml
 
-from plain_parser import loaders, plain_file, plain_spec
+from plain_parser import loaders, plain_spec, project
 
 DEFAULT_CONFIG_NAME = "config.yaml"
 TEMPLATE_DIR_CONFIG_KEYS = ("template_dir", "template-dir")
-PARSE_FORMAT_VERSION = 1
+TEST_DIR_CONFIG_KEYS = ("test_dir", "test-dir")
+CONFORMANCE_TESTS_SCRIPT_CONFIG_KEYS = ("conformance_tests_script", "conformance-tests-script")
+DEFAULT_TEST_DIR = "tests"
+PARSE_FORMAT_VERSION = 2
 
 
 class AmbiguousConfigFileError(Exception):
@@ -25,13 +29,13 @@ def build_parser() -> argparse.ArgumentParser:
 
     check_parser = subparsers.add_parser(
         "check",
-        help="Validate a .plain module and its requires chain, and read every linked file: "
+        help="Validate a .plain module, its requires chain and its test plain files, and read every linked file: "
         "binary files and large base64 blobs are rejected.",
     )
     parse_parser = subparsers.add_parser(
         "parse",
-        help="Print the parse result of a .plain module and its requires chain as JSON. Linked files are "
-        "located but never read.",
+        help="Print the parse result of a .plain module, its requires chain and its test plain files as JSON. "
+        "Linked files are located but never read.",
     )
     for subparser in (check_parser, parse_parser):
         subparser.add_argument("plain_file", help="Path to the .plain module.")
@@ -40,11 +44,18 @@ def build_parser() -> argparse.ArgumentParser:
             help="Directory searched for modules and templates after the .plain file's own directory. "
             "Overrides the template-dir value of the config file.",
         )
+        subparser.add_argument(
+            "--test-dir",
+            action="append",
+            metavar="DIR",
+            help="Directory scanned for test plain files; repeat the option for more than one. Overrides the "
+            f"test-dir value of the config file. Defaults to {DEFAULT_TEST_DIR} next to the .plain file.",
+        )
         subparser.add_argument(
             "--config-name",
             default=DEFAULT_CONFIG_NAME,
-            help="Name of the config file to read template-dir from. Looked up in the .plain file's directory "
-            "and the current working directory. Defaults to %(default)s.",
+            help="Name of the config file to read template-dir, test-dir and conformance-tests-script from. "
+            "Looked up in the .plain file's directory and the current working directory. Defaults to %(default)s.",
         )
     return parser
 
@@ -71,17 +82,37 @@ def resolve_config_file(config_name: str, plain_file_path: str) -> str | None:
     return None
 
 
-def template_dir_from_config(config_file: str) -> str | None:
-    """The template-dir value of the config file, resolved against the file's directory. Other keys are ignored."""
+@dataclass
+class Config:
+    template_dir: str | None = None
+    test_dirs: list[str] | None = None  # None when the config does not set test-dir.
+    conformance_tests_enabled: bool = False  # Whether the config names a conformance-tests-script.
+
+
+def _config_value(config: dict, keys: tuple[str, ...]):
+    """The first value set under one of the key spellings, or None."""
+    return next((config[key] for key in keys if config.get(key)), None)
+
+
+def read_config(config_file: str) -> Config:
+    """The values plain-parser reads from the config file, paths resolved against the file's directory. Other keys
+    are ignored."""
     with open(config_file) as f:
         config = yaml.safe_load(f) or {}
 
-    for key in TEMPLATE_DIR_CONFIG_KEYS:
-        value = config.get(key)
-        if value:
-            value = os.path.expanduser(str(value))
-            return value if os.path.isabs(value) else os.path.join(os.path.dirname(config_file), value)
-    return None
+    def resolve(value) -> str:
+        value = os.path.expanduser(str(value))
+        return value if os.path.isabs(value) else os.path.join(os.path.dirname(config_file), value)
+
+    result = Config()
+    template_dir = _config_value(config, TEMPLATE_DIR_CONFIG_KEYS)
+    if template_dir:
+        result.template_dir = resolve(template_dir)
+    test_dir = _config_value(config, TEST_DIR_CONFIG_KEYS)
+    if test_dir:
+        result.test_dirs = [resolve(value) for value in (test_dir if isinstance(test_dir, list) else [test_dir])]
+    result.conformance_tests_enabled = bool(_config_value(config, CONFORMANCE_TESTS_SCRIPT_CONFIG_KEYS))
+    return result
 
 
 @dataclass
@@ -92,20 +123,37 @@ class ParseResult:
     # Each link target in the chain, as written, to the path it resolves to, relative to the entry module's dir
     # with `/` as separator (loaders.relative_resource_path).
     resources: dict[str, str]
-
-
-def parse_spec(plain_file_path: str, template_dir: str | None, load_resources: bool) -> ParseResult:
-    """Parse the module and its requires chain, resolve its links, and walk the entry's FRIDs; raise on the first error.
-
-    With ``load_resources``, each module's linked files are also read and checked (text only, no large base64 blob) right
-    after that module's links are collected, so a resource error is reported before a FRID-walk error.
+    # Each test plain file, by its path relative to the entry module's dir with `/`, to the functionality it binds to.
+    test_plain_files: dict[str, dict] = field(default_factory=dict)
+
+
+def parse_spec(
+    plain_file_path: str,
+    template_dir: str | None,
+    load_resources: bool,
+    test_dirs: list[str] | None = None,
+    require_conformance_scenarios: bool = False,
+) -> ParseResult:
+    """Parse the module, its requires chain and its test plain files, resolve the links, and walk the entry's
+    FRIDs; raise on the first error.
+
+    ``test_dirs`` defaults to ``tests`` next to the module. With ``load_resources``, each module's linked files are
+    also read and checked (text only, no large base64 blob) right after that module's links are collected, so a
+    resource error is reported before a FRID-walk error.
     """
     # Absolute, so the chain parse and the resource paths below search the same directories.
-    template_dirs = [os.path.dirname(os.path.abspath(plain_file_path))]
+    entry_dir = os.path.dirname(os.path.abspath(plain_file_path))
+    template_dirs = [entry_dir]
     if template_dir:
         template_dirs.append(os.path.abspath(template_dir))
+    if test_dirs is None:
+        test_dirs = [os.path.join(entry_dir, DEFAULT_TEST_DIR)]
+    test_dirs = [os.path.abspath(test_dir) for test_dir in test_dirs]
 
-    chain = plain_file.parse_module_chain(os.path.basename(plain_file_path), template_dirs)
+    parsed = project.parse_project(
+        os.path.basename(plain_file_path), template_dirs, test_dirs, require_conformance_scenarios
+    )
+    chain = parsed.chain
 
     # The target joined as written onto its template dir (`..` kept, symlinks not resolved), made relative to the
     # entry module's dir so the result does not depend on where the project sits on the machine.
@@ -124,12 +172,21 @@ def parse_spec(plain_file_path: str, template_dir: str | None, load_resources: b
     for frid in plain_spec.get_frids(top_plain_source_tree):
         plain_spec.get_specifications_for_frid(top_plain_source_tree, frid)
 
-    return ParseResult(chain, resources)
+    test_plain_files = {
+        bound.name: {"module": bound.parsed.module, "func_id": bound.parsed.func_id}
+        for bound in parsed.test_plain_files
+    }
+    return ParseResult(chain, resources, test_plain_files)
 
 
-def check(plain_file_path: str, template_dir: str | None) -> None:
-    """Raise if the module, its requires chain, or its linked resources are invalid."""
-    parse_spec(plain_file_path, template_dir, load_resources=True)
+def check(
+    plain_file_path: str,
+    template_dir: str | None,
+    test_dirs: list[str] | None = None,
+    require_conformance_scenarios: bool = False,
+) -> None:
+    """Raise if the module, its requires chain, its test plain files, or its linked resources are invalid."""
+    parse_spec(plain_file_path, template_dir, True, test_dirs, require_conformance_scenarios)
 
 
 def _error_output(type_name: str, message: str) -> dict:
@@ -146,15 +203,21 @@ def _encode_output(output: dict) -> bytes:
     return (json.dumps(output, indent=2, ensure_ascii=False) + "\n").encode("utf-8")
 
 
-def parse(plain_file_path: str, template_dir: str | None) -> int:
+def parse(
+    plain_file_path: str,
+    template_dir: str | None,
+    test_dirs: list[str] | None = None,
+    require_conformance_scenarios: bool = False,
+) -> int:
     """Print the parse result, or the parser error, as one UTF-8 JSON document on stdout; return the exit code."""
     output: dict
     try:
-        result = parse_spec(plain_file_path, template_dir, load_resources=False)
+        result = parse_spec(plain_file_path, template_dir, False, test_dirs, require_conformance_scenarios)
         output = {
             "format_version": PARSE_FORMAT_VERSION,
             "modules": [{"module": module_name, "tree": tree} for module_name, tree in result.modules],
             "resources": result.resources,
+            "test_plain_files": result.test_plain_files,
         }
         exit_code = 0
     except Exception as e:
@@ -209,10 +272,12 @@ def main(argv: list[str] | None = None) -> int:
 
     try:
         # The config file is resolved even when --template-dir is given, so an ambiguous config is always an error.
-        config_template_dir = _template_dir_from_config(parser, args)
+        config = _read_config(parser, args)
+        template_dir = args.template_dir or config.template_dir
+        test_dirs = args.test_dir or config.test_dirs
         if args.command == "parse":
-            return parse(args.plain_file, args.template_dir or config_template_dir)
-        check(args.plain_file, args.template_dir or config_template_dir)
+            return parse(args.plain_file, template_dir, test_dirs, config.conformance_tests_enabled)
+        check(args.plain_file, template_dir, test_dirs, config.conformance_tests_enabled)
     except KeyboardInterrupt:
         return 130
     except Exception as e:
@@ -223,10 +288,10 @@ def main(argv: list[str] | None = None) -> int:
     return 0
 
 
-def _template_dir_from_config(parser: argparse.ArgumentParser, args: argparse.Namespace) -> str | None:
+def _read_config(parser: argparse.ArgumentParser, args: argparse.Namespace) -> Config:
     try:
         config_file = resolve_config_file(args.config_name, args.plain_file)
-        return template_dir_from_config(config_file) if config_file is not None else None
+        return read_config(config_file) if config_file is not None else Config()
     except AmbiguousConfigFileError as e:
         parser.error(str(e))
     except Exception as e: