Toolkit: fix reviewed Defender workbook conversion and additive V3.1 packaging

Tools/Create-Azure-Sentinel-Solution/V3/README-V3_1.md

@@ -247,8 +247,15 @@ V3.1 applies these hard guards in local, catalog, and positional pipeline modes:
 |---|---|
 | Effective solution/package version, even without XDR content | `>= 3.2.0` and `< 4.0.0` |
 | Effective XDR Custom Detection content version | `>= 3.1.0` and `< 4.0.0` |
+| Effective Defender Workbook `xdrMetadata.version` | Canonical `major.minor.patch` in `[3.1.0, 4.0.0)` |
+| Ordinary Sentinel Workbook metadata content version | Independent numeric version; no solution-major ceiling (for example, `4.0.0` is valid with solution `3.2.1`) |
 | Every other packaged content-item release version | `< 4.0.0`; existing `1.x`/`2.x` versions remain valid |
 
+Existing Sentinel Workbook content and metadata versions are retained unchanged;
+adding XDR content does not require downgrading them to match the solution major.
+This distinction does not relax solution, Custom Detection, or Defender Workbook
+release-version checks.
+
 The guard reports the content type/path, effective version, and required range.
 It never promotes versions automatically. Explicit local bumps are evaluated
 before output: for example, `major` from `3.2.0` selects `4.0.0` and is rejected.

Tools/Create-Azure-Sentinel-Solution/common/versionPolicy.ps1

@@ -10,7 +10,7 @@ function Assert-V31ReleaseVersion {
     if (-not $v31VersionPolicy) { return }
 
     $minimum = if ($ContentKind -eq 'Solution') { [version]'3.2.0.0' } elseif ($ContentKind -eq 'CustomDetection') { [version]'3.1.0.0' } else { [version]'0.0.0.0' }
-    $range = if ($ContentKind -eq 'Solution') { '>= 3.2.0 and < 4.0.0' } elseif ($ContentKind -eq 'CustomDetection') { '>= 3.1.0 and < 4.0.0' } else { '< 4.0.0' }
+    $range = if ($ContentKind -eq 'Solution') { '>= 3.2.0 and < 4.0.0' } elseif ($ContentKind -eq 'CustomDetection') { '>= 3.1.0 and < 4.0.0' } elseif ($ContentKind -eq 'Workbook') { 'an independent numeric content version' } else { '< 4.0.0' }
     $parsed = $null
     $valid = $Version -cmatch '\A[0-9]+\.[0-9]+(?:\.[0-9]+){0,2}\z' -and
         [version]::TryParse($Version, [ref]$parsed)
@@ -19,7 +19,10 @@ function Assert-V31ReleaseVersion {
     }
     if ($valid) {
         $normalized = [version]::new($parsed.Major, $parsed.Minor, [Math]::Max(0, $parsed.Build), [Math]::Max(0, $parsed.Revision))
-        $valid = $normalized -ge $minimum -and $normalized -lt [version]'4.0.0.0'
+        # Sentinel Workbook metadata versions predate and are independent of solution release bounds.
+        # Defender Workbooks use their separate Assert-DefenderWorkbookReleaseVersion guard.
+        $valid = $normalized -ge $minimum -and
+            ($ContentKind -eq 'Workbook' -or $normalized -lt [version]'4.0.0.0')
     }
     if (-not $valid) {
         $message = "V3.1 version policy: $ContentKind '$ContentPath' has effective release version '$Version'; required $range (numeric release version). No version is automatically promoted."

Tools/SentinelToXDRMigration/README.md

@@ -117,6 +117,31 @@ resolution, and unresolved review findings remain blockers. The converter record
 source items in `workbooks-manifest.json`, separate from detection `manifest.json`.
 Queries removed as proven scope-only helpers remain in provenance.
 
+The KQL rewriter treats `Type` conservatively: literal `Type` labels explicitly
+projected by a query are retained, and a bare `Type` grouping in a one-line
+`let Alias = NativeTable | summarize ... by ...` can be preserved by adding
+that table's name as a `Type` value before aggregation. Other `Type` references
+remain review findings; they are not replaced with `ActionType` or removed.
+
+Workbook dependency review also recognizes source-free literal `datatable`
+queries limited to literal cells, direct `project` expressions, and `order by`.
+Complete `range Column from Start to Stop step Step` queries are also recognized
+when all three operands are signed 64-bit integer literals or finite real
+literals (decimal/exponent notation), the step is nonzero, and its direction
+matches the endpoints (equal endpoints are allowed). Range keywords are lowercase;
+the column must be a plain ASCII identifier. No range functions, parameters,
+arithmetic, casts, comments, additional statements, or pipeline/union tails are
+accepted by this exception. The original query is preserved unchanged.
+These literal queries require an explicit hash-bound review decision but no
+telemetry table, workload, or connector prerequisite; other table-free KQL remains
+unresolved. V3.1 prerequisite scanning already treats the range operator as built-in
+and continues discovering function calls and telemetry branches in larger queries.
+This static exception does not verify runtime visibility or parity. The identity
+mappings for `MessageEvents`, `MessageUrlInfo`, `MessagePostDeliveryEvents`, and
+`FileMaliciousContentInfo` are grounded in their Microsoft Learn Advanced
+Hunting table references; they do not assert runtime visibility or add a schema
+where none is catalogued.
+
 The explicit solution Data `XDR Workbooks` array selects deployable content.
 An empty array selects none; packaging never repopulates it by scanning files.
 V3.1 emits workbook installation and Content Hub registration together under
@@ -135,6 +160,16 @@ alternatives, and function candidates. Optional `--results <metadata-file>
 --workspace-id <approved-workspace-id>` imports approved lab evidence; planning
 and import never execute queries or authenticate.
 
+Connector selections in solution Data accept `Data Connectors/<file>` and,
+when the solution directory is directly under `Solutions`, the exact
+`Solutions/<current-solution-directory>/Data Connectors/<file>` repository-relative
+form. Plans return solution-relative POSIX `sourcePath` values for both forms;
+source Data selections are never rewritten. References must name existing files
+within the current solution's `Data Connectors` directory. Other solutions,
+absolute paths, backslashes, empty/dot/traversal segments, and symlinks are
+rejected. Review regenerates the same canonical plan and checks connector hashes,
+while retaining the hash of the original solution Data.
+
 Apply explicit decisions with `workbook-review --solution <path> --run-id <run-id>
 --plan <plan-file> --decisions <decision-file> --plan-sha256 <hash> --actor <reviewer>
 --reference <review-reference> --output-dir <new-run-local-directory>`.
@@ -197,6 +232,15 @@ resources block packaging. Full workbook registration retains function prerequis
 the installed notebook retains only the identity/version quartet. ARM ordering
 does not establish AH propagation, query correctness, or provider/UI acceptance.
 
+The V3.1 prerequisite scanner excludes a source query only when current source
+file/query hashes and its JSON pointer prove it belongs to a removable scope
+parameter, and the target no longer contains or references that helper. A
+`removed-scope-helper` status alone is insufficient; retained helpers and
+non-helper `resources` queries still block. Source and target scanning use the
+shared workbook mapping catalog and reviewed scalar `Granularity` and
+`union withsource=Table (print ...)` contracts. Real saved aliases and dynamic
+`table()`/`totable()` sources still require review or remain blockers.
+
 **Provider contract unverified:** generated templates and successful local tests
 are candidate evidence, not accepted installation or publication evidence.
 Independent, approved lab installation, registration/provenance readback, repeat

Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/data/mapping_catalog.json

@@ -714,7 +714,7 @@
       "dropColumns": [],
       "applyGlobalDropColumns": false,
       "discriminator": null,
-      "notes": "Offline conversion mapping; target visibility and runtime semantics remain unverified."
+      "notes": "Identity mapping for the Microsoft Defender XDR Advanced Hunting table documented at https://learn.microsoft.com/en-us/defender-xdr/advanced-hunting-filemaliciouscontentinfo-table; visibility and runtime semantics remain unverified."
     },
     "GCPDNS": {
       "ahTable": "GCPDNS",
@@ -813,6 +813,30 @@
       "recommendAhTable": "GraphAPIAuditEvents",
       "notes": "Offline conversion mapping; target visibility and runtime semantics remain unverified."
     },
+    "MessageEvents": {
+      "ahTable": "MessageEvents",
+      "tier": "1to1",
+      "columnRenames": {},
+      "dropColumns": [],
+      "discriminator": null,
+      "notes": "Identity mapping for the Microsoft Defender XDR Advanced Hunting table documented at https://learn.microsoft.com/en-us/defender-xdr/advanced-hunting-messageevents-table; visibility and runtime semantics remain unverified."
+    },
+    "MessagePostDeliveryEvents": {
+      "ahTable": "MessagePostDeliveryEvents",
+      "tier": "1to1",
+      "columnRenames": {},
+      "dropColumns": [],
+      "discriminator": null,
+      "notes": "Identity mapping for the Microsoft Defender XDR Advanced Hunting table documented at https://learn.microsoft.com/en-us/defender-xdr/advanced-hunting-messagepostdeliveryevents-table; visibility and runtime semantics remain unverified."
+    },
+    "MessageUrlInfo": {
+      "ahTable": "MessageUrlInfo",
+      "tier": "1to1",
+      "columnRenames": {},
+      "dropColumns": [],
+      "discriminator": null,
+      "notes": "Identity mapping for the Microsoft Defender XDR Advanced Hunting table documented at https://learn.microsoft.com/en-us/defender-xdr/advanced-hunting-messageurlinfo-table; visibility and runtime semantics remain unverified."
+    },
     "NTANetAnalytics": {
       "ahTable": "NTANetAnalytics",
       "tier": "1to1",

Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/import-manifest.json

@@ -27,7 +27,7 @@
     },
     {
       "component": "data/mapping_catalog.json",
-      "sha256": "5a76b154eb5a077219e900392eefce12f8a2e8d18c49e723441c1c4c9d3660c7",
+      "sha256": "6e81e0029e971302354e607e3f103b25aae282629205f8bb44637d45cae7d159",
       "originalSha256": "6a3c2c8a164923ec461208f33919518e36e34242106a3a8826650c8fcde60067"
     },
     {
@@ -55,7 +55,7 @@
     },
     {
       "component": "migrator.py",
-      "sha256": "694a75d820016928c127eccb51b933dae8eb5dbd95b7b9840a74dfe3ddb90c82",
+      "sha256": "6172c2b8ecb0069b5702403b62b254f68ce5e24a06ca5b9bec35c436e06c3a93",
       "originalSha256": "4f7d9d0c126d283a1c92e7429e89c01085b08c239e7dce66b8953482b0a5e92f"
     },
     {
@@ -65,7 +65,7 @@
     },
     {
       "component": "rewrite.py",
-      "sha256": "a77ff6aaeca6c919d0ed002085f877514a47fd670b68d6d1ac2179b45640c8c2",
+      "sha256": "9b53e3be9877c178d0b1aa8c255ae96da451f30b956e71f4b7e30b08bafc8ab0",
       "originalSha256": "0c4138edf041221f0bdfc4ee00e4417d4c9d505ff8ffec7ec29d62409ef8732a"
     },
     {
@@ -75,7 +75,7 @@
     },
     {
       "component": "workbook.py",
-      "sha256": "238701e7fec987ec6e7bb9ae89eaf98647c09e03802db9aac889a1a5772f28ac",
+      "sha256": "d23ddd2ada78e4b9da717cf9a79e58d96f31a9c5f84d25727a46129cfda020b7",
       "originalSha256": "eb931b23333994a56e94d3c8975c7663db258b9b8032b0ff171b9e456e1b3942"
     },
     {

Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/migrator.py

@@ -60,13 +60,19 @@ def convert(kql: str, tables: Optional[List[str]] = None,
         raise ValueError("Live resolution is not supported by the offline converter.")
 
     res = rewrite.rewrite(kql, tables)
-    status = _status_of(tables, res)
+    source_free_literal = (
+        not tables and rewrite.is_source_free_literal_query(kql)
+    )
+    if source_free_literal:
+        res["deterministic"] = True
+        res["overallTier"] = "1to1"
+    status = "converted" if source_free_literal else _status_of(tables, res)
     tier = res["overallTier"]
     mapping_tier = tier if tier in _CONTRACT_TIERS else "blocked"
 
     unsupported = [{"construct": f["name"], "kind": f["kind"], "reason": f["reason"]}
                    for f in res["flags"]]
-    if tables is None:
+    if tables is None and not source_free_literal:
         unsupported.append({
             "construct": "data-sources", "kind": "unresolved",
             "reason": resolve_error or "Data sources could not be resolved "