feat: enhance JSON schema generation to allow empty strings for optional fields

Author: arnoox

src/extensions/score_metamodel/README.md

@@ -81,6 +81,127 @@ JSON Schema cannot express:
 | `check_id_contains_feature` | `id_contains_feature.py` | Need IDs must contain the feature abbreviation from the file path |
 | `check_standards` | `standards.py` | Standard compliance link validation |
 
+### Coverage comparison
+
+Schema column: **yes** = implemented, **feasible** = could be added, **--** = not possible.
+
+| Rule | Schema (`sn_schemas.py` + sphinx-needs) | S-Core metamodel (`checks/`) | Notes |
+|---|:---:|:---:|---|
+| ID required | yes | -- | `needs_id_required` (sphinx-needs built-in) |
+| ID basic regex | yes | -- | `needs_id_regex` (sphinx-needs built-in) |
+| Dead link detection | yes | -- | `allow_dead_links` (sphinx-needs built-in) |
+| Mandatory field presence | yes | yes | Both enforce `required` |
+| Mandatory field regex | yes | yes | Same pattern from metamodel |
+| Optional field regex | yes | yes | Schema: only if field present |
+| Mandatory link presence | yes | yes | Schema: `minItems: 1` in local |
+| Mandatory link target type | yes | yes | Schema: `validate.network` |
+| Mandatory link ID regex | feasible | yes | Can add `items.pattern` in local; TODO in code |
+| Optional link target type | feasible | yes (info) | Split into separate schema with `severity: "info"` |
+| Optional link ID regex | feasible | yes (info) | Same split-severity approach |
+| Mixed regex+plain link type | -- | yes | `ValidateSchemaType` has no `anyOf`/`oneOf` |
+| ID structure (parts count) | feasible | yes | Per-type pattern from `parts` field; cannot check file-path part |
+| Prohibited words | feasible | yes | Negative lookahead regex on `title`; less precise than Python |
+| Graph constraints | -- | yes | Cross-need traversals beyond JSON Schema |
+| Undefined extra options | -- | yes | `unevaluatedProperties` would reject sphinx-needs internal fields |
+
+#### Rule explanations
+
+**ID required** --
+Every need directive must have a manually set ID (e.g. `.. feat_req:: feat_req__my_feature__001`).
+Enforced by sphinx-needs' `needs_id_required = True` in `__init__.py`.
+
+**ID basic regex** --
+The ID must match `^[A-Za-z0-9_-]{6,}` (at least 6 alphanumeric/underscore/hyphen characters).
+Enforced by sphinx-needs' `needs_id_regex` in `__init__.py`. The build stops if a need
+has an invalid ID.
+
+**Dead link detection** --
+A link like `satisfies: nonexistent_need_id` that points to a need that does not exist
+triggers a sphinx-needs warning. Controlled per link type via `allow_dead_links` in
+`needs_extra_links`.
+
+**Mandatory field presence** --
+A `feat_req` must have a `status` field. If it is missing, both the schema
+(`"required": ["status"]`) and the Python check flag it.
+
+**Mandatory field regex** --
+The `status` field on `feat_req` must match `^(valid|invalid)$`. Both the schema
+(`"pattern": "^(valid|invalid)$"`) and the Python check validate this. Writing
+`status: approved` is rejected.
+
+**Optional field regex** --
+`document` has `optional_options: { author: ^.*$ }`. If `author` is present, it must
+match the pattern. If absent, no error. The schema includes it in `properties` but
+not in `required`.
+
+**Mandatory link presence** --
+`feat_req` has `mandatory_links: { satisfies: stkh_req }`. At least one target must
+be provided. The schema enforces this with `"satisfies": {"type": "array", "minItems": 1}`
+in `validate.local`.
+
+**Mandatory link target type** --
+`feat_req.satisfies` must point to a need of type `stkh_req`. The schema enforces
+this with `validate.network`: each linked need is checked for
+`{"type": {"const": "stkh_req"}}`. If a `feat_req` links to a `comp` via `satisfies`,
+the schema rejects it.
+
+**Mandatory link ID regex** (feasible) --
+`feat` has `mandatory_links: { includes: ^logic_arc_int(_op)*__.+$ }`. The link
+target IDs (strings like `logic_arc_int__something`) must match this regex.
+Currently the schema only enforces that at least one link exists (`minItems: 1`),
+not the ID pattern. *Feasible*: add `"items": {"pattern": "^logic_arc_int(_op)*__.+$"}`
+to the local schema. There is a TODO in the code for this.
+
+**Optional link target type** (feasible) --
+`tool_req` has `optional_links: { satisfies: gd_req, stkh_req }`. If provided, targets
+should be `gd_req` or `stkh_req`. The Python check validates this but treats violations
+as informational (non-fatal). The schema currently skips this because all schema entries
+use `severity: "violation"` and there is no way to set a different severity for one
+rule within the same schema entry. *Feasible*: create a second schema entry for the
+same need type with `severity: "info"` that only checks optional link targets.
+
+**Optional link ID regex** (feasible) --
+Same as above, but for regex-based link IDs on optional links (e.g.
+`optional_links: { links: ^.*$ }` on `tsf`). Same severity-split approach would work.
+
+**Mixed regex+plain link type** (not possible) --
+`workproduct` has `optional_links: { complies: std_wp, ^std_req__aspice_40__iic.*$ }`.
+A `complies` target is valid if it is either a need of type `std_wp` OR has an ID
+matching the regex. The `validate.network` `items` schema applies to ALL linked needs
+identically, so it cannot express "match type X *or* match regex Y".
+sphinx-needs' `ValidateSchemaType` does not support `anyOf`/`oneOf`.
+These mixed fields are validated only by the Python check.
+
+**ID structure (parts count)** (feasible) --
+`feat_req` has `parts: 3`, meaning its ID must have 3 segments separated by `__`
+(e.g. `feat_req__my_feature__001`). The Python check (`check_id_format`) splits on
+`__` and counts parts. *Feasible*: generate a per-type regex like
+`^feat_req__[^_]+(__[^_]+){1}$` in the schema. However, the Python check also
+validates that the ID contains the feature abbreviation from the file path
+(`check_id_contains_feature`), which depends on runtime context and cannot be
+expressed in a schema.
+
+**Prohibited words** (feasible) --
+The metamodel forbids words like "shall", "must", "will" in need titles (for
+requirement types). The Python check splits the title into words and checks each one.
+*Feasible*: add a negative lookahead regex on the `title` field, e.g.
+`^(?!.*\b(shall|must|will)\b).*$`. This is less precise than the Python check
+(which normalizes case, strips punctuation) but catches most violations.
+
+**Graph constraints** (not possible) --
+`graph_checks` in the metamodel define rules like "an ASIL_B need must link to at
+least one non-QM requirement via `satisfies`". This requires traversing the need
+graph across multiple levels, which is fundamentally beyond what JSON Schema can
+express. Only the Python check (`check_metamodel_graph`) can do this.
+
+**Undefined extra options** (not possible) --
+The Python check (`check_extra_options`) warns when a need has fields not defined
+in the metamodel (e.g. a typo like `saftey` instead of `safety`). In theory,
+`unevaluatedProperties: false` could reject unknown fields. In practice, sphinx-needs
+adds many internal fields to needs (e.g. `docname`, `lineno`, `is_external`, computed
+fields from dynamic functions) that are not in the metamodel. Enabling this would
+cause false positives on every need.
+
 ## File layout
 
 ```

src/extensions/score_metamodel/sn_schemas.py

@@ -136,13 +136,14 @@ def _build_local_validator(
         if field in IGNORE_FIELDS:
             continue
         required.append(field)
-        properties[field] = get_field_pattern_schema(field, pattern)
+        properties[field] = get_field_pattern_schema(field, pattern, is_optional=False)
 
     # Optional fields: if present, must match the regex pattern
+    # Allow empty strings to align with Python checker behavior
     for field, pattern in optional_fields.items():
         if field in IGNORE_FIELDS:
             continue
-        properties[field] = get_field_pattern_schema(field, pattern)
+        properties[field] = get_field_pattern_schema(field, pattern, is_optional=True)
 
     # Mandatory links (regex): must have at least one entry
     # TODO: regex pattern matching on link IDs is not yet enabled
@@ -265,28 +266,47 @@ def add_network_entry(field: str, target_types: list[str]) -> None:
     return type_schema
 
 
-def get_field_pattern_schema(field: str, pattern: str) -> dict[str, Any]:
+def get_field_pattern_schema(field: str, pattern: str, is_optional: bool = False) -> dict[str, Any]:
     """Return the appropriate JSON schema for a field's regex pattern.
 
     Array-valued fields (like ``tags``) get an array-of-strings schema;
     scalar fields get a plain string schema.
+
+    For optional fields, the schema allows empty strings to align with the
+    Python metamodel checker's behavior (which treats empty strings as absent).
     """
     if field in SN_ARRAY_FIELDS:
-        return get_array_pattern_schema(pattern)
-    return get_pattern_schema(pattern)
+        return get_array_pattern_schema(pattern, is_optional=is_optional)
+    return get_pattern_schema(pattern, is_optional=is_optional)
+
 
+def get_pattern_schema(pattern: str, is_optional: bool = False) -> dict[str, Any]:
+    """Return a JSON schema that validates a string against a regex pattern.
 
-def get_pattern_schema(pattern: str) -> dict[str, str]:
-    """Return a JSON schema that validates a string against a regex pattern."""
+    For optional fields, allows either an empty string OR a string matching
+    the pattern, matching the Python checker's behavior where empty strings
+    are treated as "absent" and not validated.
+    """
+    if is_optional:
+        # Allow empty strings for optional fields (Python checker treats "" as absent)
+        # Use regex alternation to match either empty string or the original pattern
+        return {
+            "type": "string",
+            "pattern": f"^$|{pattern}",
+        }
     return {
         "type": "string",
         "pattern": pattern,
     }
 
 
-def get_array_pattern_schema(pattern: str) -> dict[str, Any]:
-    """Return a JSON schema that validates an array where each item matches a regex."""
+def get_array_pattern_schema(pattern: str, is_optional: bool = False) -> dict[str, Any]:
+    """Return a JSON schema that validates an array where each item matches a regex.
+
+    For optional fields, allows empty strings in the array to align with the
+    Python checker's behavior.
+    """
     return {
         "type": "array",
-        "items": get_pattern_schema(pattern),
+        "items": get_pattern_schema(pattern, is_optional=is_optional),
     }

src/extensions/score_metamodel/tests/test_sn_schemas.py

@@ -45,6 +45,16 @@ def test_preserves_complex_regex(self) -> None:
         assert result["type"] == "string"
         assert result["pattern"] == pattern
 
+    def test_optional_allows_empty_string(self) -> None:
+        result = get_pattern_schema("^https://github.com/.*$", is_optional=True)
+        assert result == {"type": "string", "pattern": "^$|^https://github.com/.*$"}
+
+    def test_mandatory_does_not_allow_empty_string(self) -> None:
+        result = get_pattern_schema("^[A-Z]+$", is_optional=False)
+        assert result == {"type": "string", "pattern": "^[A-Z]+$"}
+        # Should not have alternation with empty string
+        assert "^$|" not in result.get("pattern", "")
+
 
 # =============================================================================
 # Tests for get_array_pattern_schema
@@ -64,6 +74,11 @@ def test_items_match_get_pattern_schema(self) -> None:
         result = get_array_pattern_schema(pattern)
         assert result["items"] == get_pattern_schema(pattern)
 
+    def test_optional_array_allows_empty_string_items(self) -> None:
+        result = get_array_pattern_schema("^tag_.*$", is_optional=True)
+        assert result["type"] == "array"
+        assert result["items"] == {"type": "string", "pattern": "^$|^tag_.*$"}
+
 
 # =============================================================================
 # Tests for get_field_pattern_schema
@@ -85,6 +100,16 @@ def test_unknown_field_returns_string_schema(self) -> None:
         result = get_field_pattern_schema("some_custom_field", "^.*$")
         assert result["type"] == "string"
 
+    def test_optional_scalar_field_allows_empty_string(self) -> None:
+        result = get_field_pattern_schema("mitigation_issue", "^https://github.com/.*$", is_optional=True)
+        assert result == {"type": "string", "pattern": "^$|^https://github.com/.*$"}
+
+    def test_mandatory_scalar_field_does_not_allow_empty_string(self) -> None:
+        result = get_field_pattern_schema("status", "^(valid|invalid)$", is_optional=False)
+        assert result == {"type": "string", "pattern": "^(valid|invalid)$"}
+        # Should not have alternation with empty string
+        assert "^$|" not in result.get("pattern", "")
+
 
 # =============================================================================
 # Tests for _classify_links
@@ -157,6 +182,8 @@ def test_optional_fields_not_required(self) -> None:
         result = _build_local_validator({}, optional, {}, {})
         assert "comment" not in result["required"]
         assert "comment" in result["properties"]
+        # Optional fields should allow empty strings via pattern alternation
+        assert result["properties"]["comment"] == {"type": "string", "pattern": "^$|^.*$"}
 
     def test_ignored_fields_excluded(self) -> None:
         mandatory = {field: "^.*$" for field in IGNORE_FIELDS}