ndr_matlab_python_bridge.yaml

# The NDR Bridge Protocol: YAML Specification

# =============================================================================
# 0. THIS FILE IS NORMATIVE
# =============================================================================
# This file is the ONE place the bridge contract is defined. AGENTS.md and
# docs/developer_notes/PYTHON_PORTING_GUIDE.md point here; they do not restate
# the rules. If you are about to write a bridge rule in another document,
# write it here instead and link to it -- a rule stated in three places is a
# rule that will eventually contradict itself.
#
# Two parts of this file are LIVE YAML rather than comments, so that
# tests/test_matlab_bridge_conventions.py can read them and the enforcement
# cannot drift away from the prose:
#   * `status_vocabulary` (Section 6) -- the complete set of legal `status:`
#     values.
#   * `project_metadata` (Section 2) -- the header every bridge file copies.
# `status_vocabulary` is spec-only: do NOT copy it into a package's bridge
# file. Only `project_metadata` is copied.

# =============================================================================
# 1. File Purpose & Placement
# =============================================================================
# Name:     ndr_matlab_python_bridge.yaml
# Location: One file per sub-package directory
#           (e.g., src/ndr/reader/ndr_matlab_python_bridge.yaml).
# Role:     This is the Primary Contract. It defines how MATLAB names and types
#           map to Python. If a function is not here, it does not officially
#           exist in the Python port.

# =============================================================================
# 2. Standard Header
# =============================================================================
# Every bridge file must begin with this metadata to orient the agent:

project_metadata:
  bridge_version: "1.2"
  naming_policy: "Strict MATLAB Mirror"
  indexing_policy: "Semantic Parity (1-based for user concepts, 0-based for internal data)"

# =============================================================================
# 3. The "Active Maintenance" Instruction
# =============================================================================
# When an agent or developer works on a function:
#
# 1. Check:  Does the function/class exist in the YAML?
# 2. Add/Update: If it is missing or the MATLAB signature has changed,
#    update the YAML first.
# 3. Record Hash: Store the MATLAB COMMIT in `matlab_last_sync_hash`
#    (Section 5).
# 4. Notify: The agent MUST explicitly tell the user:
#    "I have updated the ndr_matlab_python_bridge.yaml to include
#    [Function Name]. Please review the interface contract."

# =============================================================================
# 4. One entry per MATLAB function
# =============================================================================
# A MATLAB file gets exactly ONE bridge entry. Duplicate entries are the
# failure mode the completeness check cannot see: that check asks "is this
# file recorded?", and a file recorded twice is recorded. Two entries then
# drift apart -- one saying `porting_deferred` while the other records a real
# port that exists -- and a reader who finds the wrong one first is misled.
#
# Uniqueness is keyed on (`name`, `matlab_path`), NOT on `matlab_path` alone.
# Several entries may legitimately share a `matlab_path`: a class's methods are
# recorded off the one .m file that defines the class.
#
# THE ONE EXCEPTION. One MATLAB function may be recorded twice when it is
# genuinely exposed from two Python modules. To declare that, EVERY entry in
# the group must carry a distinct `python_qualified:` naming its own module
# path. Half a declaration is not a declaration: if one entry names a
# `python_qualified` and its twin does not, that is an accident, not a design,
# and tests/test_matlab_bridge_conventions.py rejects it.
#
# NDR-python currently has no such deliberate pair. The rule exists so that the
# first one is written down rather than discovered.

# =============================================================================
# 5. matlab_last_sync_hash IS A COMMIT
# =============================================================================
# `matlab_last_sync_hash` records a git COMMIT in NDR-matlab -- never a blob
# hash, never a tree, never a tag. Get it with:
#
#     git -C ../NDR-matlab log -n 1 --format=%h -- <matlab_path>
#
# BOTH FORMS OF COMMIT ARE LEGAL (NDI-python#211). The command above gives the
# file's own last-touching commit -- the default, and the more informative of
# the two, because it says "I examined this version of this file". A repo-wide
# commit is equally legal, e.g. NDR-matlab HEAD recorded when a batch of files
# was examined together: a broader claim, "I examined as of this repo state".
# Drift (Section 5a) is sound under both, so neither needs rewriting.
#
# NOT with `git hash-object <file>`. The distinction matters because freshness
# is asked by walking history:
#
#     git -C ../NDR-matlab log <hash>..HEAD -- <matlab_path>
#
# and `git log` DOES NOT ERROR on a blob -- it happily returns commits. A blob
# recorded here therefore reads as merely "stale" rather than as unusable, and
# the wrong diagnosis gets acted on. (`git diff` is the command that rejects a
# blob outright.) NDI-python accumulated 35 blob hashes exactly this way,
# because its spec said "the hash of the .m file" while printing a command that
# yields a commit.
#
# WHAT COUNTS AS OUT OF DATE is Section 5a. The rule there is not NDR's own
# invention: it is one rule for all three bridge repos, settled in
# NDI-python#211 and written down in the DRIFT section of NDI-python's
# `docs/developer_notes/ndi_matlab_python_bridge.yaml`. That is its home.
#
# THIS SPEC DOES NOT DESCRIBE WHAT THE OTHER REPOS DO. An earlier draft here
# characterised NDI-python's rule in a passing sentence; the sentence was a
# claim about a file in somebody else's repository, so it was correct only
# until they edited it -- and it did go stale. A cross-reference cannot: it
# resolves to whatever the other repo says today. Cite them by NAME, never by
# section number, for the same reason -- their numbering is theirs to change,
# and this file has numbered sections of its own for a stray one to land on.
#
# Read the rule there; read how NDR-python enforces it in Section 5a and in
# tests/test_matlab_bridge_completeness.py.
#
# RECOVERING A BAD HASH. If an entry ever holds a blob hash, convert it
# exactly: walk the file's history comparing `git rev-parse <commit>:<path>`
# against the recorded blob, and record the commit that produced it. Never
# "fix" it by bumping to HEAD -- that claims an examination nobody performed.
# If the hash resolves to nothing at all, DELETE the field and say why in the
# `decision_log`. An absent hash claims nothing; a wrong hash claims something
# false.

# =============================================================================
# 5a. DRIFT: detecting upstream MATLAB changes
# =============================================================================
# An entry DRIFTS when NDR-matlab has commits touching its `matlab_path` after
# the recorded hash:
#
#     git -C ../NDR-matlab log <matlab_last_sync_hash>..HEAD -- <matlab_path>
#
# Non-empty means drift, and drift fails CI. This is one rule for all three
# bridge repos: settled in NDI-python#211, stated in the DRIFT section of
# NDI-python's `docs/developer_notes/ndi_matlab_python_bridge.yaml`, and
# enforced here by `tests/test_matlab_bridge_completeness.py`. Restated here
# only far enough to be actionable; where the two ever disagree, theirs is the
# text that governs.
#
# THE CHECK IS FILE-SCOPED, ALWAYS. `-- <matlab_path>` restricts the walk to
# commits touching that ONE file, so unrelated NDR-matlab activity never makes
# an entry drift, whatever form the recorded hash takes.
#
# NOT "the hash equals the file's last-touching commit". That older rule asked
# about the hash's FORM rather than about the file, and so rejected records
# that were TRUE -- an entry recording a repo-wide batch-sync commit failed it
# even when nothing had touched the file since. The only way to clear the red
# build was to rewrite a correct record with a different, equally correct
# value. NDI-python had 103 entries of that shape. A check that demands
# busywork on true records is a check contributors learn to ignore.
#
# When an entry does drift, the file really has changed since it was examined,
# and you must either:
#
#   1. Review the MATLAB diff, port the behavioral change, and bump the hash; or
#   2. Bump the hash and add a short `decision_log` note saying why the MATLAB
#      change is a no-op on the Python side (comment-only, analyzer fix, ...).
#
# This check needs a NON-SHALLOW NDR-matlab checkout. A shallow clone collapses
# every file's history to the commit it fetched, so the drift walk cannot see
# the real history and every entry reads as wrong. CI checks it out with
# `fetch-depth: 0`, and the test fails loudly on a shallow clone rather than
# emitting one false failure per entry.
#
# EVERY ENTRY NAMING A `matlab_path` MUST CARRY A HASH (NDI-python#211
# decision 2, enforced). An entry without one can never drift, so it asserts
# its port is current forever and nothing can contradict it -- the same false
# assurance as a stale hash, but silent instead of red. A `retired` entry sets
# `matlab_path: "N/A"`, names no file, and is exempt.
#
# That gate reads only this repo, needing no NDR-matlab checkout, so it runs in
# every test-matrix job rather than only the bridge job.

# =============================================================================
# 6. status: THE COMPLETE VOCABULARY
# =============================================================================
# `status` says how a MATLAB function is (or is not) represented in Python.
# It is a closed set. `tests/test_matlab_bridge_conventions.py` reads the keys
# below and rejects any other value, so this table and the enforcement cannot
# drift.
#
# EVERY ENTRY STATES ITS STATUS, INCLUDING THE ORDINARY ONE. A plain 1:1 port
# is written `status: regular_port`, not left blank. These files are read by
# people far more often than by the test suite, and an absent field does not
# announce what it means: a reader looking at an entry with no `status` cannot
# tell "this is a normal port" from "nobody filled this in". Spelling it out
# costs one line and removes the guess. This is enforced -- an entry with a
# `matlab_path` and no `status` fails.
#
# Every status EXCEPT `regular_port` needs a `decision_log` explaining it
# (enforced). A divergence without a reason passes the completeness check while
# telling the next reader nothing, so the gap gets re-investigated anyway. A
# regular port needs no justification -- mirroring MATLAB is the default -- so
# requiring 105 boilerplate decision_logs would only devalue the ones that
# carry real reasoning.

status_vocabulary:

  regular_port: >-
    The ordinary case, and by far the most common: a 1:1 Python counterpart
    under the mirrored name. REQUIRES a `python_path` (and `python_class` for
    a class). Needs no `decision_log` -- there is no decision to record, since
    mirroring MATLAB is what the port does by default. Written out rather than
    left blank so that a human reading the YAML can tell a finished entry from
    an unfinished one at a glance.

  ported_differently: >-
    Python HAS this capability, but does the job a DIFFERENT WAY rather than as
    a 1:1 counterpart under the mirrored name: under a different name, folded
    into a class or another function, or through different machinery -- very
    often a third-party dependency doing work MATLAB hand-rolls. The label
    names the manner, not the location: `python_path` already answers WHERE,
    and REQUIRES an answer here (a list when the capability genuinely spans
    modules). When the real work is done by a dependency, the `decision_log`
    names it. Use this and not `matlab_only` whenever a Python caller can do
    the thing -- a reader who sees `matlab_only` concludes Python cannot, which
    is the opposite of the truth.

  porting_deferred: >-
    No Python counterpart today. The gap is known and may be closed later. The
    `decision_log` says why not now, or what has to happen first. This is the
    honest label for "we haven't got to it", and it is not a criticism -- an
    unported MATLAB function that is written down is a managed gap.

  matlab_only: >-
    Exists because MATLAB works that way, and will never get a counterpart by
    design. MATLAB-runtime shims (fopen/fread handle wrappers), MATLAB
    packaging artifacts (Contents.m), MATLAB-interop helpers, and backend
    dispatch that Python's single backend makes moot. Not a deferral: nobody
    is waiting for this.

  retired: >-
    The MATLAB function no longer exists upstream. A tombstone, kept so the
    decision is not re-litigated the next time somebody notices the name in an
    old commit. Set `matlab_path: "N/A"` and say in the `decision_log` which
    NDR-matlab commit removed it.

# RETIRED STATUS VALUES -- do not reintroduce:
#   ported_elsewhere -> ported_differently. "Elsewhere" names a PLACE, but
#                      `python_path` already answers where. Of the nine entries
#                      carrying it, six had not moved anywhere at all: zarr and
#                      numcodecs simply do what MATLAB hand-rolls, and four
#                      native readers cover what +ndr/+reader/imagestack.m
#                      delegates to NANSEN. What a reader cannot recover from
#                      any other field is the MANNER, so the label names that.
#                      Renamed across NDI-python, NDR-python and DID-python
#                      together; see NDR-python issue #21.
#   not_yet_ported  -> porting_deferred  (vague: "not yet" by whom, blocked on what?)
#   not_applicable  -> split by what is actually meant. It conflated three
#                      incompatible claims: "Python does this another way"
#                      (ported_differently), "we haven't done it" (porting_deferred)
#                      and "MATLAB-ism, never will" (matlab_only). Its plain
#                      reading -- "Python cannot / need not do this" -- was
#                      flatly wrong for the majority of entries carrying it.
#   implemented     -> regular_port (one spelling for the ordinary case).
#   ported          -> regular_port (ditto).
#   does_not_exist  -> retired.
#
# NOTE: an ABSENT `status` used to mean "regular port". It no longer does --
# it now means the entry is incomplete, and fails. See the preamble above.

# =============================================================================
# 7. Structure for Classes and Functions
# =============================================================================

# --- Example: Class ---
# - name: base                        # Exact MATLAB Name
#   type: class
#   matlab_path: "+ndr/+reader/base.m"
#   status: regular_port
#   python_path: "ndr/reader/base.py"
#   python_class: "ndr_reader_base"
#   matlab_last_sync_hash: "a4c9e07"  # Latest COMMIT touching base.m (Section 5)
#
#   properties:
#     - name: MightHaveTimeGaps
#       type_matlab: "logical"
#       type_python: "bool"
#       decision_log: "Mirroring property name exactly."
#
#   methods:
#     - name: readchannels_epochsamples
#       input_arguments:
#         - name: channeltype
#           type_matlab: "char"
#           type_python: "str"
#         - name: channel
#           type_matlab: "numeric"
#           type_python: "int | list[int]"
#         - name: epochstreams
#           type_matlab: "cell array of char"
#           type_python: "list[str]"
#         - name: epoch_select
#           type_matlab: "numeric"
#           type_python: "int"
#           decision_log: "Semantic Parity: User provides 1-based ID."
#         - name: s0
#           type_python: "int"
#         - name: s1
#           type_python: "int"
#       output_arguments:
#         - name: data
#           type_matlab: "double | int16 | int32"
#           type_python: "numpy.ndarray"

# --- Example: Standalone Function ---
# - name: ndrpath
#   type: function
#   matlab_path: "+ndr/+fun/ndrpath.m"
#   status: regular_port
#   python_path: "ndr/fun/ndrpath.py"
#   matlab_last_sync_hash: "8f3a2b1"
#   input_arguments: []
#   output_arguments:
#     - name: p
#       type_matlab: "char"
#       type_python: "str"
#   decision_log: "Synchronized with MATLAB main as of 2026-04-11."

# --- Example: Function with no 1:1 counterpart ---
# - name: python_detect
#   type: function
#   matlab_path: "+ndr/+fun/python_detect.m"
#   status: matlab_only
#   matlab_last_sync_hash: "c1d2e3f"
#   decision_log: "MATLAB-only helper that locates a Python interpreter for
#     MATLAB's py.* interop. Python does not need to detect itself."

# =============================================================================
# 8. Building a New Bridge YAML File
# =============================================================================
# When adding a brand-new sub-package to NDR-python (for example
# `src/ndr/newformat/`), create a fresh `ndr_matlab_python_bridge.yaml`
# next to the Python source using the following procedure:
#
# 1. Create the file at `src/ndr/<namespace>/ndr_matlab_python_bridge.yaml`.
# 2. Start with the Standard Header from Section 2 (copy `project_metadata`
#    verbatim; do NOT copy `status_vocabulary`, which is spec-only).
# 3. For each MATLAB class in `+ndr/+<namespace>/`, add a `classes:` entry with:
#      - `name` (MATLAB class name, e.g. `intan_rhd`)
#      - `type: class`
#      - `matlab_path` (e.g. `"+ndr/+reader/intan_rhd.m"`)
#      - `status` (Section 6 -- `regular_port` for an ordinary 1:1 port)
#      - `python_path` (e.g. `"ndr/reader/intan_rhd.py"`)
#      - `python_class` (derived using the Class Name Mirror Rule from
#        `docs/developer_notes/PYTHON_PORTING_GUIDE.md`, e.g.
#        `ndr_reader_intan__rhd`)
#      - `inherits` (if applicable)
#      - `matlab_last_sync_hash` (Section 5)
#      - `properties:` and `methods:` blocks with `input_arguments` /
#        `output_arguments`, each argument carrying `name`, `type_matlab`, and
#        `type_python`.
# 4. For each standalone MATLAB function, add an entry under `functions:` with
#    the same fields (minus `properties` and `methods`). If it has no 1:1
#    counterpart, give it the matching `status` from Section 6 and a
#    `decision_log` saying why.
# 5. Register the package in `PACKAGES` in
#    `tests/test_matlab_bridge_completeness.py`, or nothing under it is guarded.
# 6. Run the bridge checks:
#      NDR_BRIDGE_CHECK_STRICT=1 NDR_MATLAB_PATH=../NDR-matlab \
#        pytest tests/test_matlab_bridge_completeness.py \
#               tests/test_matlab_bridge_conventions.py
# 7. Notify the user with a one-line summary of the new contract so they can
#    review it before any Python implementation is written.
#
# Refer to `src/ndr/reader/ndr_matlab_python_bridge.yaml` for a concrete,
# non-trivial example covering both `classes:` with inherited methods and
# `static_methods:` blocks.

# =============================================================================
# 9. Summary of Field Rules
# =============================================================================
# Field                | Rule
# ---------------------|----------------------------------------------------------
# name                 | Strict Match. Case-sensitive match to the MATLAB .m file.
# matlab_path          | Path relative to NDR-matlab's repo root, or "N/A" for a
#                      | `retired` entry. Must name a file that exists.
# matlab_last_sync_hash| A COMMIT in NDR-matlab -- either the file's own
#                      | last-touching commit or a repo-wide one. Must not have
#                      | DRIFTED (Section 5a). See Section 5. Never a blob.
# status               | REQUIRED on every entry. One of Section 6's vocabulary;
#                      | `regular_port` for an ordinary 1:1 port. Requires a
#                      | `decision_log` for every value except `regular_port`.
# python_path          | Where the Python counterpart lives. Required for
#                      | `regular_port` and for `ported_differently`. A plain port names one
#                      | file; a `ported_differently` entry MAY name a list when
#                      | the capability is genuinely reached through more than
#                      | one module (see +ndr/+reader/imagestack.m).
# python_qualified     | Dotted module path. Required on EVERY member of a
#                      | deliberate two-module pair (Section 4).
# input_arguments      | Used to generate Pydantic @validate_call.
# output_arguments     | Defines the order of the Return Tuple.
# type_python          | Use Python 3.10+ syntax (e.g., str | int).
# decision_log         | Mandatory for any divergence and for every `status`
#                      | except `regular_port`.
#                      | Should include the sync date (e.g., "Synchronized with
#                      | MATLAB main as of 2026-04-11.").