diff --git a/Tools/Create-Azure-Sentinel-Solution/V3/README-V3_1.md b/Tools/Create-Azure-Sentinel-Solution/V3/README-V3_1.md index 3519173ee78..a69057ed501 100644 --- a/Tools/Create-Azure-Sentinel-Solution/V3/README-V3_1.md +++ b/Tools/Create-Azure-Sentinel-Solution/V3/README-V3_1.md @@ -48,6 +48,8 @@ The 12 positional pipeline parameters (positions 0–11) are unchanged. The optional named `-SkipAttributionLookup` switch is available in both local and pipeline parameter sets. For CI callers that do not forward optional arguments, set `SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1` in the process environment. +Both controls apply only when attribution is optional; emitted XDR detections +or Defender Workbooks require attribution and cannot skip missing-ID lookup. ## Independent content selection (V3.1 default) @@ -114,6 +116,33 @@ uninstall operation. There is no new all-off runtime check. A selection can leave prerequisites unmet: the booleans do not rewrite dependency semantics, guarantee Content Hub acceptance, or provision missing external dependencies. +### Native Defender Workbooks without a Sentinel parent + +Already-native Advanced Hunting workbooks use the explicit +`native-workbook-plan` / `native-workbook-review` / `native-workbook-import` authoring route documented in +[the migration toolkit README](../../SentinelToXDRMigration/README.md#native-defender-workbook-authoringimport-no-sentinel-parent). +The raw source may be supplied under solution-local `Workbook Sources/Native/` +while running the authoring commands, but it need not be retained or shipped. +Select only the authored `XDR Workbooks` candidate; no duplicate source artifact +is packaged. `contentProvenance.origin: native-xdr` selects native review with +historical source path/hash and a canonical reviewed Notebook payload hash, +instead of migrated-source converter proof. It does not weaken the existing +migrated workbook contract. + +The V3.1 emitter invokes the shared native Python validator before any template +mutation, requiring Python 3.11/3.12 with the toolkit dependencies. +`SENTINEL_XDR_PYTHON` selects the interpreter (default: `python`); the subprocess +uses this emitter's toolkit checkout, not another editable installation. +Unknown functions/tables, payload/query changes, and stale/incomplete reviews +block packaging. Source deletion after import does not invalidate a native +candidate. +Native and migrated entries share managed install/registration, independent +Sentinel selection, default-false `DeployDefenderWorkbook`, and the installed +provenance quartet. Native sources/reviews are not installed or ZIP entries. +No Sentinel workbook, converter receipt or runtime acceptance is required or +manufactured for the native authoring route. Provider acceptance still requires +the existing independent approved evidence. + ### Defender Workbook parser prerequisites Authored workbooks declare reviewed function prerequisites in @@ -301,9 +330,9 @@ These guards do not govern ARM `apiVersion`, `contentSchemaVersion`, ARM templat saved-search API revision numbers. Legacy `V3/createSolutionV3.ps1` and `pipeline/createSolutionV4.ps1` retain their previous version behavior. -## Optional customer usage attribution +## Customer usage attribution -Set optional `trackingId` in the solution's **SolutionMetadata.json** to the +Set `trackingId` in the solution's **SolutionMetadata.json** to the **complete customer usage attribution tracking ID** copied from Partner Center **offer > plan > Technical configuration**, for example: @@ -364,24 +393,34 @@ Version catalog lookup and `GetPackageVersion` behavior are unchanged. Attribution requests use a 20-second request/connection timeout and, on PowerShell 7.4+, an explicit 20-second stalled-read timeout, with no retries or redirects (continuous downloads are not capped at 20 seconds). -HTTP status, timeout, network/read, JSON, selection and marker failures produce -classified `CUSTOMER USAGE ATTRIBUTION:` warnings and continue **without** -attribution or metadata changes. A successful lookup logs publisher, offer, +For packages emitting XDR detections or Defender Workbooks, attribution is +**required**. HTTP status, timeout, network/read, JSON, selection and marker +failures produce a classified, terminating `CUSTOMER USAGE ATTRIBUTION:` error +before package files or the ZIP are written, leaving metadata unchanged. +Verify the correct publisher/offer identity and unambiguous published plan/marker, +or supply the complete ID from Partner Center and rebuild. There is no +incomplete-package override. When attribution is optional (no emitted XDR +detections or Defender Workbooks), lookup failures retain the warning-only +behavior and continue **without** attribution or metadata changes. +A successful lookup logs publisher, offer, plan ID, template URL (without query credentials), tracking ID and updated path as `CUSTOMER USAGE ATTRIBUTION SOURCE:`. It is not reported as a warning. -For offline attribution, use `-SkipAttributionLookup` or set -`SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1`; explicit IDs still emit markers. Use +For optional offline attribution, use `-SkipAttributionLookup` or set +`SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1`; explicit IDs still emit markers. +Neither control suppresses lookup when required attribution has a missing ID. +A valid explicit ID avoids attribution network access even for required packages. Use `-VersionMode local` as well to avoid the **independent** version-catalog lookup. These controls do not disable other packaging prerequisites or validators. The migration CLI inherits the same environment control, so no redundant CLI flag is required. Its packaging report preserves lookup failures in `warnings` and successful source messages in `attributionSources`. -If no ID is available, nested XDR packaging still warns that Partner Center -cannot automatically add tracking for functional nested deployments. -Obtain the ID from offer > plan > Technical configuration, add `trackingId`, -and rebuild. Missing attribution does not imply Marketplace upload rejection. +Partner Center cannot automatically add tracking for functional nested +deployments. If required lookup cannot resolve an ID, packaging fails rather +than emitting a markerless package. Obtain the ID from offer > plan > Technical +configuration, add `trackingId`, and rebuild. This packaging guard is not a +claim about Marketplace upload rejection. See [Microsoft Marketplace Azure apps customer usage attribution](https://learn.microsoft.com/partner-center/marketplace-offers/azure-partner-customer-usage-attribution#microsoft-marketplace-azure-apps). The migration CLI uses `package-v3-1` and writes `packaging.v3_1.json` with @@ -399,10 +438,11 @@ PYTHONPATH=Tools/SentinelToXDRMigration:Tools/SentinelToXDRMigration/tests pytho The attribution tests exercise both local and positional pipeline entry points with isolated synthetic solutions, verify generated JSON and ZIP contents, and cover missing/invalid IDs and nested XDR preservation. Catalog/template requests -are mocked; default test subprocesses disable lookup, so unit tests never depend +are mocked; optional test subprocesses disable lookup, so unit tests never depend on live Marketplace availability or mutate real solution metadata. Cases include automatic persistence/replay, explicit overrides, ambiguous identities/plans/ -markers, remote failures, offline controls, byte preservation, local write +markers, required lookup despite both offline controls, terminating failures +before artifact output, optional offline behavior, byte preservation, local write failures/concurrent edits and cached-ID retention during version bumps. They stub the unrelated full repository validators and isolate repository-root resolution. They do not diff --git a/Tools/Create-Azure-Sentinel-Solution/common/AttributionMetadata.cs b/Tools/Create-Azure-Sentinel-Solution/common/AttributionMetadata.cs index b4bdf8e4a8e..73f04c142ba 100644 --- a/Tools/Create-Azure-Sentinel-Solution/common/AttributionMetadata.cs +++ b/Tools/Create-Azure-Sentinel-Solution/common/AttributionMetadata.cs @@ -77,7 +77,12 @@ public static void Save(string path, byte[] expected, string trackingId) typeof(File).GetMethod("SetUnixFileMode", new[] { typeof(string), mode.GetType() }) .Invoke(null, new object[] { staging, mode }); } - File.Move(staging, path, true); + // Windows overwrite-move fails while the destination is open; + // Replace supports the writer-excluding, delete-sharing handle. + if (Path.DirectorySeparatorChar == '\\') + File.Replace(staging, path, null); + else + File.Move(staging, path, true); } } finally diff --git a/Tools/Create-Azure-Sentinel-Solution/common/commonFunctions.ps1 b/Tools/Create-Azure-Sentinel-Solution/common/commonFunctions.ps1 index 47d498e688d..8987602ecd3 100644 --- a/Tools/Create-Azure-Sentinel-Solution/common/commonFunctions.ps1 +++ b/Tools/Create-Azure-Sentinel-Solution/common/commonFunctions.ps1 @@ -3313,6 +3313,11 @@ function GeneratePackage( $baseMainTemplate.metadata.author = $(if ($contentToImport.Author) { $contentToImport.Author } else { "" }) $baseMainTemplate.metadata.comments = $baseMainTemplate.metadata.comments -replace "{{SolutionName}}", $solutionName + if ($IncludeXdrDetections) { + # Resolve required attribution after content/version preparation, before any package output. + Add-CustomerUsageAttribution -SolutionMetadataPath $SolutionMetadataPath -Template $global:baseMainTemplate -WarnIfMissing ($xdrDetectionCount -gt 0 -or $defenderWorkbookCount -gt 0) -SkipAttributionLookup:$SkipAttributionLookup + } + $repoRoot = $(git rev-parse --show-toplevel) $solutionFolderName = $solutionName $solutionFolder = "$repoRoot/Solutions/$solutionFolderName" @@ -3326,11 +3331,6 @@ function GeneratePackage( $mainTemplateOutputPath = "$solutionFolder/mainTemplate.json" $createUiDefinitionOutputPath = "$solutionFolder/createUiDefinition.json" - if ($IncludeXdrDetections) { - # Persist only after content/version preparation, after metadata filtering and before output. - Add-CustomerUsageAttribution -SolutionMetadataPath $SolutionMetadataPath -Template $global:baseMainTemplate -WarnIfMissing ($xdrDetectionCount -gt 0 -or $defenderWorkbookCount -gt 0) -SkipAttributionLookup:$SkipAttributionLookup - } - try { $baseMainTemplate | ConvertTo-Json -Depth $packageSerializationDepth | Out-File $mainTemplateOutputPath -Encoding utf8 diff --git a/Tools/Create-Azure-Sentinel-Solution/common/customerUsageAttribution.ps1 b/Tools/Create-Azure-Sentinel-Solution/common/customerUsageAttribution.ps1 index 0812707bd06..56ed3c8ad72 100644 --- a/Tools/Create-Azure-Sentinel-Solution/common/customerUsageAttribution.ps1 +++ b/Tools/Create-Azure-Sentinel-Solution/common/customerUsageAttribution.ps1 @@ -141,6 +141,7 @@ function Add-CustomerUsageAttribution { [string]$SolutionMetadataPath, [Parameter(Mandatory = $true)] [psobject]$Template, + # Retain the existing parameter name; nested XDR content requires attribution. [bool]$WarnIfMissing = $false, [switch]$SkipAttributionLookup ) @@ -152,19 +153,22 @@ function Add-CustomerUsageAttribution { $trackingId = $metadata.trackingId $discovered = $null if ($null -eq $trackingId -or ($trackingId -is [string] -and [string]::IsNullOrWhiteSpace($trackingId))) { - if (-not $SkipAttributionLookup -and $env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP -ne '1') { + if ($WarnIfMissing -or (-not $SkipAttributionLookup -and $env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP -ne '1')) { try { $discovered = Resolve-PublishedCustomerUsageAttribution -Metadata $metadata $trackingId = $discovered.TrackingId } catch { + if ($WarnIfMissing) { + throw "CUSTOMER USAGE ATTRIBUTION: Required published lookup failed [$($_.Exception.Message)]. Packaging stopped; SolutionMetadata.json is unchanged. Verify publisherId and offerId identify the correct published offer with one unambiguous plan/marker, or supply the complete trackingId from Partner Center > offer > plan > Technical configuration in '$SolutionMetadataPath' and rebuild. Attribution lookup cannot be skipped for nested XDR content." + } Write-Warning "CUSTOMER USAGE ATTRIBUTION: Published lookup failed [$($_.Exception.Message)]. No attribution marker will be emitted; SolutionMetadata.json is unchanged. Supply the complete trackingId from Partner Center > offer > plan > Technical configuration and rebuild." } } } if ($null -eq $trackingId -or ($trackingId -is [string] -and [string]::IsNullOrWhiteSpace($trackingId))) { if ($WarnIfMissing) { - Write-Warning "CUSTOMER USAGE ATTRIBUTION: No attribution marker will be emitted because trackingId is absent or blank in '$SolutionMetadataPath'. Partner Center cannot automatically add tracking when the template contains functional nested deployments (including XDR). Obtain the COMPLETE customer usage attribution tracking ID from Partner Center > offer > plan > Technical configuration, add trackingId to SolutionMetadata.json, and rebuild. Do not use a Partner ID, plan ID, or a newly generated GUID. See https://learn.microsoft.com/partner-center/marketplace-offers/azure-partner-customer-usage-attribution#microsoft-marketplace-azure-apps" + throw "CUSTOMER USAGE ATTRIBUTION: Required trackingId is still absent or blank in '$SolutionMetadataPath'. Packaging stopped; SolutionMetadata.json is unchanged. Verify publisherId and offerId, or supply the complete trackingId from Partner Center > offer > plan > Technical configuration and rebuild. Attribution lookup cannot be skipped for nested XDR content." } return } diff --git a/Tools/Create-Azure-Sentinel-Solution/common/defenderWorkbooks.ps1 b/Tools/Create-Azure-Sentinel-Solution/common/defenderWorkbooks.ps1 index 666ec7ed657..3f2f3e6dfd7 100644 --- a/Tools/Create-Azure-Sentinel-Solution/common/defenderWorkbooks.ps1 +++ b/Tools/Create-Azure-Sentinel-Solution/common/defenderWorkbooks.ps1 @@ -44,6 +44,47 @@ function Assert-DefenderWorkbookReleaseVersion { } } +function Assert-NativeDefenderWorkbookDocument { + param([psobject]$Document, [string]$SolutionRoot) + + $python = if ($env:SENTINEL_XDR_PYTHON) { $env:SENTINEL_XDR_PYTHON } else { 'python' } + $start = [System.Diagnostics.ProcessStartInfo]::new() + $start.FileName = $python + $start.UseShellExecute = $false + $start.RedirectStandardInput = $true + $start.RedirectStandardOutput = $true + $start.RedirectStandardError = $true + $start.StandardInputEncoding = [System.Text.UTF8Encoding]::new($false) + $start.StandardOutputEncoding = [System.Text.UTF8Encoding]::new($false) + foreach ($argument in @('-m', 'sentinel_xdr_migration.native_workbooks', $SolutionRoot)) { + $start.ArgumentList.Add($argument) + } + $start.Environment['PYTHONPATH'] = [System.IO.Path]::GetFullPath( + (Join-Path $PSScriptRoot '../../SentinelToXDRMigration')) + $start.Environment['PYTHONIOENCODING'] = 'utf-8' + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $start + try { + [void]$process.Start() + $stdout = $process.StandardOutput.ReadToEndAsync() + $stderr = $process.StandardError.ReadToEndAsync() + $process.StandardInput.Write(($Document | ConvertTo-Json -Depth 100 -Compress)) + $process.StandardInput.Close() + $process.WaitForExit() + $output = $stdout.GetAwaiter().GetResult() + $diagnostic = $stderr.GetAwaiter().GetResult() + if ($process.ExitCode -ne 0) { + throw "Native Defender Workbook validation failed: $output $diagnostic" + } + $receipt = $output | ConvertFrom-Json -ErrorAction Stop + if ($null -eq $receipt -or $receipt.errors -isnot [array] -or $receipt.errors.Count -ne 0) { + throw 'Native Defender Workbook validation returned no valid receipt.' + } + } finally { + $process.Dispose() + } +} + function Assert-DefenderWorkbookDocument { param([psobject]$Document, [string]$SolutionRoot, [string]$SolutionName, [string]$ContentPath, [switch]$PassThruSource) @@ -63,6 +104,21 @@ function Assert-DefenderWorkbookDocument { } Assert-DefenderWorkbookReleaseVersion -Version $metadata.version $provenance = $Document.contentProvenance + if ($null -ne $provenance -and $provenance.origin -ceq 'native-xdr') { + Assert-NativeDefenderWorkbookDocument -Document $Document -SolutionRoot $SolutionRoot + if ($PassThruSource) { + $nativePayload = $Document | ConvertTo-Json -Depth 100 -Compress | + ConvertFrom-Json -ErrorAction Stop + $nativePayload.PSObject.Properties.Remove('xdrMetadata') + $nativePayload.PSObject.Properties.Remove('contentProvenance') + return $nativePayload + } + return + } + if ($null -ne $provenance.PSObject.Properties['origin'] -and + $provenance.origin -cne 'sentinel-migration') { + throw "Defender Workbook '$ContentPath' has an unknown origin." + } if ($provenance -isnot [pscustomobject] -or $provenance.dependencyReviewRequired -isnot [bool] -or $provenance.dependencyReviewRequired -or $provenance.conversion -isnot [pscustomobject] -or @@ -704,6 +760,11 @@ function Get-DefenderWorkbookParserDependencies { $sourceNotebook = Assert-DefenderWorkbookDocument -Document $Document -SolutionRoot $SolutionRoot ` -SolutionName $SolutionName -ContentPath 'prerequisite scan' -PassThruSource $provenance = $Document.contentProvenance + if ($provenance.origin -ceq 'native-xdr') { + # The shared native validator rejects functions and non-native tables; + # native imports have no savedSearch/parser deployment prerequisites. + return + } $declared = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal) $property = $provenance.PSObject.Properties['requiredWorkspaceFunctions'] if ($null -ne $property -and $property.Value -isnot [array]) { @@ -1061,9 +1122,15 @@ function Add-XdrDefenderWorkbooksToSolution { $contentPackages = @($Template.resources | Where-Object { $_.type -ieq 'Microsoft.OperationalInsights/workspaces/providers/contentPackages' }) - if ($contentPackages.Count -ne 1 -or - $null -eq $contentPackages[0].properties.dependencies.PSObject.Properties['criteria']) { - throw 'Defender Workbook packaging requires exactly one contentPackages dependency criteria resource.' + if ($contentPackages.Count -ne 1 -or $null -eq $contentPackages[0].properties) { + throw 'Defender Workbook packaging requires exactly one contentPackages resource.' + } + $dependencyProperty = $contentPackages[0].properties.PSObject.Properties['dependencies'] + if ($null -ne $dependencyProperty -and + ($null -eq $dependencyProperty.Value -or + $null -eq $dependencyProperty.Value.PSObject.Properties['criteria'] -or + $dependencyProperty.Value.criteria -isnot [array])) { + throw 'Defender Workbook packaging requires valid contentPackages dependency criteria.' } $resources = @() $criteria = @() @@ -1134,7 +1201,11 @@ function Add-XdrDefenderWorkbooksToSolution { ) # Exempt only reviewed prerequisites and their registration faces from DeployParser. Register-SolutionRequiredResources -Resources $requiredResources - $Template.resources += $resources + $Template.resources = @($Template.resources) + @($resources) + if ($null -eq $dependencyProperty) { + $contentPackages[0].properties | Add-Member -NotePropertyName dependencies -NotePropertyValue ( + [pscustomobject]@{ operator = 'AND'; criteria = @() }) + } $existing = @($contentPackages[0].properties.dependencies.criteria | Where-Object { $_.kind -ine 'DefenderWorkbook' }) $contentPackages[0].properties.dependencies.criteria = @($existing) + @($criteria) Write-Warning 'Defender Workbook install/registration is an UNVERIFIED candidate contract. Approved lab installation, registration readback, identity/provenance and repeat-install/update evidence are required; generation does not establish provider acceptance.' diff --git a/Tools/Create-Azure-Sentinel-Solution/pipeline/README.md b/Tools/Create-Azure-Sentinel-Solution/pipeline/README.md index da250af0815..23060d62fef 100644 --- a/Tools/Create-Azure-Sentinel-Solution/pipeline/README.md +++ b/Tools/Create-Azure-Sentinel-Solution/pipeline/README.md @@ -214,7 +214,7 @@ Create a file and place it in the base path of solution `https://raw.githubuser ### Generate Solution Package -- The automated pipeline calls `Tools/Create-Azure-Sentinel-Solution/V3/createSolutionV3_1.ps1`. V3.1 also supports local hybrid Sentinel/XDR packaging; see the [V3.1 guide](../V3/README-V3_1.md), including optional `trackingId` customer usage attribution in SolutionMetadata.json. +- The automated pipeline calls `Tools/Create-Azure-Sentinel-Solution/V3/createSolutionV3_1.ps1`. V3.1 also supports local hybrid Sentinel/XDR packaging; see the [V3.1 guide](../V3/README-V3_1.md), including customer usage attribution in SolutionMetadata.json. Attribution is required for emitted XDR detections or Defender Workbooks: missing-ID lookup cannot be skipped, and lookup failure stops packaging. > **IMPORTANT:** `V3/createSolutionV3.ps1` remains the legacy Sentinel-only local entry point. The older `pipeline/createSolutionV4.ps1` is not the current package automation entry point. - Core business logic for V3 and V3.1 is in `Tools/Create-Azure-Sentinel-Solution/common/commonFunctions.ps1`. - For any of the WorkbookMetadata change make use of './Tools/Create-Azure-Sentinel-Solution/V2/WorkbookMetadata/WorkbooksMetadata.json' file. diff --git a/Tools/SentinelToXDRMigration/README.md b/Tools/SentinelToXDRMigration/README.md index 810dbf5653b..70c3c8122da 100644 --- a/Tools/SentinelToXDRMigration/README.md +++ b/Tools/SentinelToXDRMigration/README.md @@ -256,6 +256,164 @@ adaptations. Refreshing this slice is a deliberate maintainer change requiring reviewed import evidence and the differential workbook fixture tests, not an automatic runtime fetch. +### Native Defender workbook authoring/import (no Sentinel parent) + +Already-native `Notebook/1.0` content has an explicit offline authoring route. +It is **not** migration: no Sentinel sibling, converter replay, conversion +success receipt, workflow-stage receipt or runtime qualification is invented. +Migrated workbooks retain their existing source inventory and deterministic +converter proof unchanged. + +Provide the original raw JSON to the plan/review/import commands at +`Workbook Sources/Native/.json` (nested paths are supported). This input +must be available while generating and importing the review, but it need not be +committed, retained in the solution, or copied into the package. Its path and +exact-byte SHA256 remain historical import evidence and preserve the existing +stable identity assignment; validation does not reread or require that source. +The review also binds `canonicalPayloadSha256`: SHA256 of canonical UTF-8 JSON +(sorted keys, compact separators, non-ASCII preserved) for the complete Notebook +after excluding only top-level `xdrMetadata` and `contentProvenance`. This +protects notices, layout, queries, and all other Notebook fields, while strict +query hashes and dependency discovery are independently rechecked. Do not +select the raw source in `Workbooks` or `XDR Workbooks`; it is not a deployment +artifact. The import command creates +`XDR Workbooks/.json` and appends that candidate once to the existing +`Data/Solution_*.json` `XDR Workbooks` selection, preserving other selections +and the solution version. A native-only solution needs no `Workbooks` directory. + +Use Python 3.11/3.12 and the toolkit's existing dependencies. For an isolated +toolkit checkout, set `PYTHONPATH` rather than reinstalling an editable package: + +```powershell +$Toolkit = 'C:\E5Workbooks\Azure-Sentinel-toolkit-native-workbooks' +$Python = 'C:\Users\wiking\.copilot\session-state\368f9173-fbe4-4420-a1b1-2fe04724ed6b\files\xdr-toolkit-venv\Scripts\python.exe' +$env:PYTHONPATH = Join-Path $Toolkit 'Tools\SentinelToXDRMigration' +$env:PYTHONIOENCODING = 'utf-8' +$env:SENTINEL_XDR_PYTHON = $Python +$Solution = 'C:\content\Solutions\Example' +$Source = 'Workbook Sources/Native/example.json' + +& $Python -m sentinel_xdr_migration.cli native-workbook-plan ` + --solution $Solution --source $Source | + Set-Content -LiteralPath '.\native-review.json' -Encoding utf8 + +# Review/edit native-review.json explicitly; do not auto-accept proposals. +& $Python -m sentinel_xdr_migration.cli native-workbook-review ` + --solution $Solution --source $Source --review '.\native-review.json' | + Set-Content -LiteralPath '.\native-reviewed.json' -Encoding utf8 +& $Python -m sentinel_xdr_migration.cli native-workbook-import ` + --solution $Solution --source $Source --review '.\native-reviewed.json' +& $Python -m sentinel_xdr_migration.cli native-workbook-validate --solution $Solution +``` + +The plan is the review input, not a success receipt. The review command checks +and echoes complete current decisions; import independently rechecks them. +Neither command auto-accepts proposals. Supply nonempty `actor`, +`reference`, and `displayName`. For **every** current query pointer (including +parameter and nested queries), supply `decision: "accept"` and nonempty +`evidence` describing the exact query's complete input/dependency review. +Keep the proposed `origin`, `source.path`, `source.sha256`, query `path`, +`canonicalPayloadSha256`, `querySha256`, `tables`, `blockers`, `requiredWorkloads`, and +`requiredDataConnectors` unchanged. The native route supports catalogued +non-Sentinel tables and standard built-in underscore ASIM parsers (`_ASim_*`, +`_Im_*`) recognized by the shared workbook classifier, including bare and called +forms. It does not support arbitrary underscore/custom saved functions or +connector/saved-function prerequisites. Reviewers cannot override discovery, label an unknown function +as a table, accept a blocker, or supply arbitrary empty lineage. Unsupported +functions/tables, dynamic/external/cross-resource sources and non-AH providers +require a separate maintainer/content design decision, not a forced import. +Empty table lineage is allowed for recognized built-in ASIM parser queries as +well as the existing strict literal `datatable`/numeric `range` recognizers. +ASIM parser names are not tables: only explicitly discovered catalogued tables +contribute workload records. An ASIM-only query therefore has empty `tables`, +`requiredWorkloads` and `requiredDataConnectors`; these mean no statically +inferred dependencies, not proof of no prerequisites. The toolkit does not infer +hidden parser inputs, workloads or connector coverage, and offline acceptance +does not qualify parser availability or execution in the target tenant. +The plan exits nonzero when it contains technical blockers. + +Authored provenance is additive: + +```json +{ + "contentProvenance": { + "origin": "native-xdr", + "source": { + "path": "Workbook Sources/Native/example.json", + "sha256": "" + }, + "canonicalPayloadSha256": "", + "nativeReview": { + "origin": "native-xdr", + "source": {"path": "Workbook Sources/Native/example.json", "sha256": ""}, + "canonicalPayloadSha256": "", + "displayName": "Example", + "actor": "", + "reference": "", + "requiredWorkloads": [{"workload": "mdo", "dataTypes": ["EmailEvents"]}], + "requiredDataConnectors": [], + "queries": [{ + "path": "/items/0/content/query", + "querySha256": "", + "tables": ["EmailEvents"], + "decision": "accept", + "evidence": "", + "blockers": [] + }] + }, + "requiredWorkloads": [{"workload": "mdo", "dataTypes": ["EmailEvents"]}], + "requiredDataConnectors": [], + "dependencyReviewRequired": false + } +} +``` + +There is no `conversion` object in native provenance. `xdrMetadata` uses the +normal schema/version/display name fields. Canonical UUIDv5 assignment is +`uuid5(NAMESPACE_URL, "native-xdr-workbook::")`; +the instance is the existing +`uuid5(NAMESPACE_URL, "xdr-workbook-instance::")`. +Relocating a checkout, reimporting, changing query content or changing release +does not change these IDs; renaming the solution directory/source path does. +Import refuses differing existing candidates rather than silently overwriting +them. A deliberate source edit requires a fresh review and removal of the old +local candidate and its `XDR Workbooks` selection before reimport, retaining the +same identity for the same path. + +Native validation compares the **whole original Notebook** with the authored +payload after removing only the two additive authoring fields. Thus query +strings (including CRLF/Unicode), items, notices, tabs and other source keys +remain unchanged. Missing/stale sources or query coverage, duplicate JSON +keys/IDs/selections, traversal and symlinks fail closed. Native validation is +integrated with `validate_workbooks`, the existing local static XDR adapter, +and the V3.1 emitter. The PowerShell emitter reuses this Python validator +against its in-memory native document and requires that interpreter's +dependencies; a missing interpreter/dependency is an error, never a waiver. +`SENTINEL_XDR_PYTHON` selects the interpreter; absent it, `python` is used. +The subprocess imports only the emitter's toolkit checkout. + +For candidate package regression testing, use the correct V3.1 entry point +from the **content repository root**, with a separate toolkit root: + +```powershell +Set-Location 'C:\content' +& (Join-Path $Toolkit 'Tools\Create-Azure-Sentinel-Solution\V3\createSolutionV3_1.ps1') ` + -SolutionDataFolderPath (Join-Path $Solution 'Data') ` + -ToolkitRootPath $Toolkit -VersionMode local -VersionBump none ` + -ArmValidationMethod lab-deployment +``` + +This generates an **unaccepted candidate** with ARM lab validation pending; +it performs no deployment and is not a replacement for production release +validation/acceptance. Do not use legacy `createSolutionV3.ps1`, fabricate an +ARM/ZIP or claim lab/runtime qualification. V3.1 uses the same managed-ID +install/registration contract and default-false `DeployDefenderWorkbook` +switch for native and migrated entries. Installed Notebook provenance remains +only the package/content identity/version quartet; source paths, hashes, +native reviews and authoring metadata are not installed. Registration keeps +the existing reviewed dependency contract. Original Sentinel selections remain +independent and no native source is deployed twice. + ### Toolkit-owner managed update policy Toolkit-managed Defender Workbooks and XDR detections are updated by stable ID. @@ -1998,17 +2156,22 @@ XDR uses only its top-level YAML `version`, including when registration is disabled. Registration metadata and content product IDs use that same version. Data `XDR Detection Version`, solution version, and source provenance are not fallbacks in V3.1. See "Independent XDR release version" above for older files. -Optional `trackingId` in SolutionMetadata.json is the complete Partner Center +`trackingId` in SolutionMetadata.json is the complete Partner Center customer usage attribution ID, not a bare GUID, Partner ID, or plan ID. When absent/blank, V3.1 attempts anonymous **public Marketplace catalog/template** discovery (not authenticated Partner Center API access) using exact publisher and offer identity and one unambiguous plan/marker. It saves the validated full ID into authoritative SolutionMetadata.json and emits the current marker shape. -Explicit IDs win without lookup. Set `SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1` for -offline attribution; the CLI inherits it without an additional flag. -Lookup failures leave metadata unchanged and preserve prominent warnings in the -packaging report; successful provenance is recorded separately in -`attributionSources`. Local metadata write failures stop packaging. +Explicit IDs win without lookup. Attribution is required when XDR detections +or Defender Workbooks are emitted: neither `-SkipAttributionLookup` nor +`SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1` suppresses missing-ID lookup. +Lookup failure or ambiguity leaves metadata unchanged and terminates packaging +before package files/ZIP output. Verify publisher/offer identity and the +published plan/marker, or supply the correct full ID and rebuild. +For optional attribution only, the CLI inherits the environment control for +offline skips and preserves warning-only lookup failures in the packaging +report. Successful provenance is recorded separately in `attributionSources`. +Local metadata write failures stop packaging. Partner Center cannot auto-add tracking for functional nested deployments. If lookup cannot resolve it, obtain the ID from offer > plan > Technical configuration, add it, and rebuild. Missing attribution does not imply upload diff --git a/Tools/SentinelToXDRMigration/agent-workflows/end-to-end-migration.md b/Tools/SentinelToXDRMigration/agent-workflows/end-to-end-migration.md index 0fa86b45d1a..18e4aca8398 100644 --- a/Tools/SentinelToXDRMigration/agent-workflows/end-to-end-migration.md +++ b/Tools/SentinelToXDRMigration/agent-workflows/end-to-end-migration.md @@ -1349,9 +1349,13 @@ AR/CD count matching, correct `E5Flavor` conditions, disabled CDs, unique IDs, and explained validation results. Complete the packaging stage only with `packager=V3.1`, the generated `packaging.v3_1.json`, and its template, UI, parameters, and ZIP paths. Existing package files are not current-run evidence. -Do not relabel historical V4 evidence. Rebuild with V3.1. Surface missing -customer usage attribution warnings; optional `trackingId` must be copied -unchanged from Partner Center into SolutionMetadata.json, never synthesized. +Do not relabel historical V4 evidence. Rebuild with V3.1. XDR detections and +Defender Workbooks require customer usage attribution: a missing `trackingId` +triggers published Marketplace lookup even with offline skip controls. +Lookup failure or ambiguity stops packaging before artifact output without +changing metadata. Verify the offer identity or supply the complete Partner +Center ID unchanged in SolutionMetadata.json; never synthesize it. +Optional-attribution packages retain offline skips and warning-only failures. V3.1 copies only the first authored tactic and its compatible nested techniques into the ARM payload, without changing the complete YAML/provenance. Disclose this intentional classification loss; packaging is not classification/alert diff --git a/Tools/SentinelToXDRMigration/schema/xdr-workbook.schema.json b/Tools/SentinelToXDRMigration/schema/xdr-workbook.schema.json index 3441a69c598..7825fb41cc8 100644 --- a/Tools/SentinelToXDRMigration/schema/xdr-workbook.schema.json +++ b/Tools/SentinelToXDRMigration/schema/xdr-workbook.schema.json @@ -21,13 +21,59 @@ }, "contentProvenance": { "type": "object", - "required": ["source", "conversion", "requiredWorkloads", "requiredDataConnectors", "dependencyReviewRequired"], + "required": ["source", "requiredWorkloads", "requiredDataConnectors", "dependencyReviewRequired"], + "allOf": [{ + "if": {"required": ["origin"], "properties": {"origin": {"const": "native-xdr"}}}, + "then": { + "required": ["nativeReview", "canonicalPayloadSha256"], + "not": {"anyOf": [{"required": ["conversion"]}, {"required": ["requiredWorkspaceFunctions"]}]}, + "properties": {"source": {"properties": {"path": {"pattern": "^Workbook Sources/Native/[^\\\\]+\\.[jJ][sS][oO][nN]$"}}}} + }, + "else": { + "required": ["conversion"], + "not": {"required": ["nativeReview"]}, + "properties": {"source": {"properties": {"path": {"pattern": "^Workbooks/[^\\\\]+\\.[jJ][sS][oO][nN]$"}}}} + } + }], "properties": { + "origin": {"enum": ["native-xdr", "sentinel-migration"]}, + "canonicalPayloadSha256": {"$ref": "#/$defs/sha256"}, + "nativeReview": { + "type": "object", + "required": ["origin", "source", "canonicalPayloadSha256", "displayName", "actor", "reference", "requiredWorkloads", "requiredDataConnectors", "queries"], + "properties": { + "origin": {"const": "native-xdr"}, + "source": {"type": "object"}, + "canonicalPayloadSha256": {"$ref": "#/$defs/sha256"}, + "displayName": {"type": "string", "minLength": 1}, + "actor": {"type": "string", "minLength": 1}, + "reference": {"type": "string", "minLength": 1}, + "requiredWorkloads": {"type": "array"}, + "requiredDataConnectors": {"type": "array"}, + "queries": { + "type": "array", + "items": { + "type": "object", + "required": ["path", "querySha256", "tables", "decision", "evidence", "blockers"], + "properties": { + "path": {"type": "string", "pattern": "^/"}, + "querySha256": {"$ref": "#/$defs/sha256"}, + "tables": {"type": "array", "uniqueItems": true, "items": {"type": "string", "minLength": 1}}, + "decision": {"const": "accept"}, + "evidence": {"type": "string", "minLength": 1}, + "blockers": {"const": []} + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false + }, "source": { "type": "object", "required": ["path", "sha256"], "properties": { - "path": {"type": "string", "pattern": "^Workbooks/[^\\\\]+\\.[jJ][sS][oO][nN]$"}, + "path": {"type": "string"}, "sha256": {"$ref": "#/$defs/sha256"}, "contentId": {"type": ["string", "null"]}, "originalMetadataVersion": {"type": ["string", "null"]} diff --git a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/data/xdr-workbook.schema.json b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/data/xdr-workbook.schema.json index 3441a69c598..7825fb41cc8 100644 --- a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/data/xdr-workbook.schema.json +++ b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/data/xdr-workbook.schema.json @@ -21,13 +21,59 @@ }, "contentProvenance": { "type": "object", - "required": ["source", "conversion", "requiredWorkloads", "requiredDataConnectors", "dependencyReviewRequired"], + "required": ["source", "requiredWorkloads", "requiredDataConnectors", "dependencyReviewRequired"], + "allOf": [{ + "if": {"required": ["origin"], "properties": {"origin": {"const": "native-xdr"}}}, + "then": { + "required": ["nativeReview", "canonicalPayloadSha256"], + "not": {"anyOf": [{"required": ["conversion"]}, {"required": ["requiredWorkspaceFunctions"]}]}, + "properties": {"source": {"properties": {"path": {"pattern": "^Workbook Sources/Native/[^\\\\]+\\.[jJ][sS][oO][nN]$"}}}} + }, + "else": { + "required": ["conversion"], + "not": {"required": ["nativeReview"]}, + "properties": {"source": {"properties": {"path": {"pattern": "^Workbooks/[^\\\\]+\\.[jJ][sS][oO][nN]$"}}}} + } + }], "properties": { + "origin": {"enum": ["native-xdr", "sentinel-migration"]}, + "canonicalPayloadSha256": {"$ref": "#/$defs/sha256"}, + "nativeReview": { + "type": "object", + "required": ["origin", "source", "canonicalPayloadSha256", "displayName", "actor", "reference", "requiredWorkloads", "requiredDataConnectors", "queries"], + "properties": { + "origin": {"const": "native-xdr"}, + "source": {"type": "object"}, + "canonicalPayloadSha256": {"$ref": "#/$defs/sha256"}, + "displayName": {"type": "string", "minLength": 1}, + "actor": {"type": "string", "minLength": 1}, + "reference": {"type": "string", "minLength": 1}, + "requiredWorkloads": {"type": "array"}, + "requiredDataConnectors": {"type": "array"}, + "queries": { + "type": "array", + "items": { + "type": "object", + "required": ["path", "querySha256", "tables", "decision", "evidence", "blockers"], + "properties": { + "path": {"type": "string", "pattern": "^/"}, + "querySha256": {"$ref": "#/$defs/sha256"}, + "tables": {"type": "array", "uniqueItems": true, "items": {"type": "string", "minLength": 1}}, + "decision": {"const": "accept"}, + "evidence": {"type": "string", "minLength": 1}, + "blockers": {"const": []} + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false + }, "source": { "type": "object", "required": ["path", "sha256"], "properties": { - "path": {"type": "string", "pattern": "^Workbooks/[^\\\\]+\\.[jJ][sS][oO][nN]$"}, + "path": {"type": "string"}, "sha256": {"$ref": "#/$defs/sha256"}, "contentId": {"type": ["string", "null"]}, "originalMetadataVersion": {"type": ["string", "null"]} diff --git a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/import-manifest.json b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/import-manifest.json index 7c664b9d023..f3fbb834726 100644 --- a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/import-manifest.json +++ b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/_workbook_core/import-manifest.json @@ -32,7 +32,7 @@ }, { "component": "data/xdr-workbook.schema.json", - "sha256": "32735770ee130a61cf0841adecf72a72c32ad10d808c425583642fbbcd7b6c2e" + "sha256": "41c1d158185a7a7e6ae16fcf0717c9b6e14fb0ee0b6065b9fa486f2098a0ff07" }, { "component": "dataframes.py", diff --git a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/cli.py b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/cli.py index d0c7b2a5ddf..99e5d30df21 100644 --- a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/cli.py +++ b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/cli.py @@ -177,6 +177,15 @@ def main(argv: list[str] | None = None) -> int: description="Convert Microsoft Sentinel analytic rules into XDR Custom Detection YAML." ) subparsers = parser.add_subparsers(dest="command", required=True) + for native_command in ("native-workbook-plan", "native-workbook-review", + "native-workbook-import", "native-workbook-validate"): + native_parser = subparsers.add_parser( + native_command, help="Offline native AH authoring; never conversion or runtime qualification.") + native_parser.add_argument("--solution", required=True) + if native_command != "native-workbook-validate": + native_parser.add_argument("--source", required=True, help="Workbook Sources/Native/ solution-relative JSON.") + if native_command in ("native-workbook-review", "native-workbook-import"): + native_parser.add_argument("--review", required=True, help="Complete reviewed current native plan JSON.") dependency_parser = subparsers.add_parser( "dependency-plan", help="Propose YAML dependencies offline, or verify them from approved lab metadata.", @@ -1172,6 +1181,10 @@ def main(argv: list[str] | None = None) -> int: selection = None try: isolated_permission_commands = { + "native-workbook-plan", + "native-workbook-review", + "native-workbook-import", + "native-workbook-validate", "plan-delegated-app-permission-update", "verify-delegated-app-permission-update", "workbook-dependency-plan", @@ -1541,6 +1554,22 @@ def main(argv: list[str] | None = None) -> int: result = plan_solution_dependencies( args.solution, results_path=args.results, workspace_id=args.workspace_id, ) + elif args.command.startswith("native-workbook-"): + from .native_workbooks import ( + plan_native_workbook, review_native_workbook, import_native_workbook, load_json, + ) + from .workbooks import validate_workbooks + if args.command == "native-workbook-plan": + result = plan_native_workbook(args.solution, args.source) + elif args.command in ("native-workbook-review", "native-workbook-import"): + review_path = Path(args.review) + if any(p.is_symlink() for p in (review_path, *review_path.parents)): + raise ValueError("Native review must not use symlinks.") + operation = (review_native_workbook if args.command == "native-workbook-review" + else import_native_workbook) + result = operation(args.solution, args.source, load_json(review_path.read_bytes())) + else: + result = validate_workbooks(args.solution, selected_only=True) elif args.command == "workbook-dependency-plan": from .workbook_dependencies import plan_workbook_dependencies result = plan_workbook_dependencies( @@ -1956,6 +1985,12 @@ def main(argv: list[str] | None = None) -> int: or result.get("validationStatus") == "blocked" ): return 1 + if args.command == "native-workbook-validate" and result["invalid"]: + return 1 + if args.command == "native-workbook-import" and result["validation"]["invalid"]: + return 1 + if args.command == "native-workbook-plan" and any(q["blockers"] for q in result["queries"]): + return 1 return 0 diff --git a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/native_workbooks.py b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/native_workbooks.py new file mode 100644 index 00000000000..8259faa4e9d --- /dev/null +++ b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/native_workbooks.py @@ -0,0 +1,309 @@ +"""Offline native Advanced Hunting import; no conversion or runtime claims.""" +from __future__ import annotations + +from copy import deepcopy +import json +import re +import sys +from pathlib import PurePosixPath +from uuid import NAMESPACE_URL, uuid5 + +from ._workbook_core import workbook, mapping_catalog, rewrite +from .catalog import query_table_references +from .dependencies import TABLE_WORKLOADS, workload_records +from .workbooks import ( + _root, _path, _document, _sha, _write, _solution_data, _release, _load_json as load_json, +) + +SOURCE_PREFIX = "Workbook Sources/Native/" +ORIGIN = "native-xdr" + + +def _source(root, relative): + path = _path(root, relative) + if not relative.startswith(SOURCE_PREFIX) or path.suffix.lower() != ".json" or not path.is_file(): + raise ValueError(f"Native source must be an existing {SOURCE_PREFIX} JSON file.") + document = _document(path) + if any(key in document for key in ("xdrMetadata", "contentProvenance")): + raise ValueError("Native source must be raw Notebook content, not an authored candidate.") + return path, document + + +def _payload(document): + return {key: value for key, value in document.items() + if key not in ("contentProvenance", "xdrMetadata")} + + +def _canonical_payload_sha256(document): + canonical = json.dumps( + _payload(document), sort_keys=True, separators=(",", ":"), + ensure_ascii=False, allow_nan=False, + ).encode("utf-8") + return _sha(canonical) + + +def _valid_source_identity(source): + if not isinstance(source, dict) or set(source) != {"path", "sha256"}: + return False + relative, digest = source["path"], source["sha256"] + return ( + isinstance(relative, str) + and relative.startswith(SOURCE_PREFIX) + and "\\" not in relative + and relative.lower().endswith(".json") + and all(part not in ("", ".", "..") for part in relative.split("/")) + and isinstance(digest, str) + and re.fullmatch(r"[0-9a-f]{64}", digest) is not None + ) + + +def identities(root, relative): + content = str(uuid5(NAMESPACE_URL, f"native-xdr-workbook:{root.name}:{relative}")) + instance = str(uuid5(NAMESPACE_URL, f"xdr-workbook-instance:{root.name}:{content}")) + return content, instance + + +def _query(surface): + raw = surface["node"]["query"] + if not isinstance(raw, str) or surface["queryType"] != workbook.AH: + raise ValueError("Every native query must be a string using advancedHunting.") + code = workbook._stripped_code(raw) + for match in re.finditer(r"\b(?:workspace|app|resource|cluster|database|table|totable|externaldata)\s*[\[(]", code, re.I): + if not workbook._is_union_withsource_output_binding(code, match.start()): + raise ValueError("Native import does not support dynamic, external or cross-resource sources.") + known = set(TABLE_WORKLOADS) | set(mapping_catalog.load_mapping_catalog()["tables"]) + asim_names = {name for name in re.findall(r"\b[A-Za-z_]\w*", code) + if workbook.is_standard_builtin_asim_parser(name)} + tables = {r["table"] for r in query_table_references(raw, known_tables=known | asim_names)} + # Include quoted/union sources and reject unknown calls/bare sources rather + # than interpreting a reviewer's assertion as proof that they are tables. + quoted_calls, quoted_tables = workbook._quoted_calls_and_table_sources(raw) + tables |= quoted_tables | workbook.union_source_names(raw) + calls = set(re.findall(r"(?=3.2.0,<4.0.0.") + content, instance = identities(root, source) + document["xdrMetadata"] = { + "contentId": content, "instanceId": instance, "version": version, + "schemaVersion": "1.0.0", "displayName": review["displayName"], + } + document["contentProvenance"] = { + "origin": ORIGIN, "source": deepcopy(review["source"]), "nativeReview": deepcopy(review), + "canonicalPayloadSha256": review["canonicalPayloadSha256"], + "requiredWorkloads": deepcopy(review["requiredWorkloads"]), + "requiredDataConnectors": deepcopy(review["requiredDataConnectors"]), + "dependencyReviewRequired": False, + } + relative = "XDR Workbooks/" + source.removeprefix(SOURCE_PREFIX) + target = _path(root, relative) + raw = (json.dumps(document, indent=2, ensure_ascii=False) + "\n").encode() + if target.exists() and target.read_bytes() != raw: + raise ValueError("Existing native candidate differs; refusing overwrite.") + # Validate all existing candidates/identities before adding this selection. + from .workbooks import _files, validate_workbooks, authored_schema_errors + errors = authored_schema_errors(document) + validate_native_document(root, document) + if errors: + raise ValueError("; ".join(errors)) + existing_validation = validate_workbooks(root, selected_only=False) + invalid = [r for r in existing_validation["results"] + if not r["valid"] and r["errors"] != ["Source workbook has no authored candidate."]] + if invalid: + raise ValueError("Existing authored candidates must validate before native import: " + str(invalid)) + for existing in _files(root, "XDR Workbooks"): + if existing == target: + continue + metadata = _document(existing).get("xdrMetadata", {}) + if {content, instance} & {metadata.get("contentId"), metadata.get("instanceId")}: + raise ValueError("Duplicate authored workbook identity.") + references = data.get("XDR Workbooks", []) + if not isinstance(references, list) or any(not isinstance(ref, str) for ref in references): + raise ValueError("XDR Workbooks selection must be an array of paths.") + if len({ref.casefold() for ref in references}) != len(references): + raise ValueError("Duplicate workbook selection.") + if any(ref.casefold() == relative.casefold() and ref != relative for ref in references): + raise ValueError("Case-insensitive workbook selection collision.") + if relative not in references: + references.append(relative) + data["XDR Workbooks"] = references + data_path = next(_path(root, "Data").glob("Solution_*.json")) + previous_data = data_path.read_bytes() + data_raw = (json.dumps(data, indent=2, ensure_ascii=False) + "\n").encode() + if _sha(path.read_bytes()) != review["source"]["sha256"]: + raise ValueError("Native source changed before import.") + target.parent.mkdir(parents=True, exist_ok=True) + if not target.exists(): + _write(target, raw, None) + if data_raw != previous_data: + _write(data_path, data_raw, previous_data) + return {"path": relative, "contentId": content, "instanceId": instance, + "runtimeQualified": False, "validation": validate_workbooks(root, selected_only=True)} + + +def main(): + # Private stdin adapter for PowerShell: validate the actual in-memory + # document without scratch files or an independently maintained validator. + if not (3, 11) <= sys.version_info[:2] < (3, 13): + raise ValueError("Native workbook validation requires Python 3.11 or 3.12.") + root = sys.argv[1] + document = load_json(sys.stdin.read()) + from .workbooks import authored_schema_errors + errors = authored_schema_errors(document) + validate_native_document(root, document) + print(json.dumps({"errors": errors})) + return int(bool(errors)) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/workbooks.py b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/workbooks.py index 48fc887f8f1..ec530a2350c 100644 --- a/Tools/SentinelToXDRMigration/sentinel_xdr_migration/workbooks.py +++ b/Tools/SentinelToXDRMigration/sentinel_xdr_migration/workbooks.py @@ -30,6 +30,17 @@ def _sha(value: bytes) -> str: return hashlib.sha256(value).hexdigest() +def _load_json(text): + def unique(pairs): + value = {} + for key, item in pairs: + if key in value: + raise ValueError(f"Duplicate JSON key: {key}") + value[key] = item + return value + return json.loads(text, object_pairs_hook=unique) + + def _root(solution) -> Path: path = Path(solution).expanduser() if any(p.is_symlink() for p in (path, *path.parents)) or not path.is_dir(): @@ -72,7 +83,7 @@ def _files(root: Path, directory: str) -> list[Path]: def _document(path: Path) -> dict: - value = json.loads(path.read_bytes()) + value = _load_json(path.read_bytes()) if not isinstance(value, dict): raise ValueError("Workbook JSON must be an object.") if value.get("version") != "Notebook/1.0" or not isinstance(value.get("items"), list): @@ -406,19 +417,22 @@ def convert_workbooks(solution, *, config=None, report_directory=None, overwrite return summary +def authored_schema_errors(document): + schema = json.loads(package_files("sentinel_xdr_migration._workbook_core").joinpath( + "data/xdr-workbook.schema.json").read_text("utf-8")) + return [f"{'/'.join(map(str, e.absolute_path))}: {e.message}" + for e in Draft202012Validator(schema).iter_errors(document)] + + def validate_workbooks(solution, *, selected_only=False) -> dict: root = _root(solution) inventory = {r["path"]: r for r in inventory_workbooks(root)} files = selected_workbook_files(root) if selected_only else _files(root, "XDR Workbooks") results, identities, instances = [], set(), set() source_paths = set() - schema = json.loads(package_files("sentinel_xdr_migration._workbook_core").joinpath( - "data/xdr-workbook.schema.json").read_text("utf-8")) - validator = Draft202012Validator(schema) for path in files: document = _document(path) - schema_errors = [f"{'/'.join(map(str, e.absolute_path))}: {e.message}" - for e in validator.iter_errors(document)] + schema_errors = authored_schema_errors(document) if schema_errors: provenance = document.get("contentProvenance") if isinstance(provenance, dict) and isinstance(provenance.get("source"), dict): @@ -442,7 +456,7 @@ def validate_workbooks(solution, *, selected_only=False) -> dict: identity, instance = _guid(metadata.get("contentId")), _guid(metadata.get("instanceId")) if not identity or not instance or identity == instance: errors.append("Canonical and instance identities must be distinct GUIDs.") - if identity in identities or instance in instances: + if identity in identities or instance in instances or identity in instances or instance in identities: errors.append("Duplicate authored workbook identity.") identities.add(identity) instances.add(instance) @@ -452,6 +466,15 @@ def validate_workbooks(solution, *, selected_only=False) -> dict: errors.append("Workbook displayName is required.") if any(key in provenance for key in ("packageId", "packageVersion", "contentId", "contentVersion")): errors.append("Deployment provenance quartet must not be authored.") + if provenance.get("origin") == "native-xdr": + from .native_workbooks import validate_native_document + errors.extend(validate_native_document(root, document)) + if source.get("path") in source_paths: + errors.append("Duplicate workbook source provenance.") + source_paths.add(source.get("path")) + results.append({"path": path.relative_to(root).as_posix(), + "sha256": _sha(path.read_bytes()), "valid": not errors, "errors": errors}) + continue row = inventory.get(source.get("path")) if not row or row["sha256"] != source.get("sha256"): errors.append("Source provenance is missing or stale.") diff --git a/Tools/SentinelToXDRMigration/tests/test_customer_usage_attribution.py b/Tools/SentinelToXDRMigration/tests/test_customer_usage_attribution.py index 21763e9dc17..62500154533 100644 --- a/Tools/SentinelToXDRMigration/tests/test_customer_usage_attribution.py +++ b/Tools/SentinelToXDRMigration/tests/test_customer_usage_attribution.py @@ -65,24 +65,43 @@ def helper(self, tracking_id=ABSENT, warn: bool = True) -> subprocess.CompletedP "$template | ConvertTo-Json -Depth 30" ) - def test_absent_null_and_blank_continue_with_actionable_warning(self) -> None: + def test_required_absent_null_and_blank_fail_with_actionable_error(self) -> None: for value in (ABSENT, None, "", " \t"): with self.subTest(value=value): + self.mock_lookup(failure="throw [System.TimeoutException]::new(); ") result = self.helper(value) - self.assertEqual(0, result.returncode, result.stderr) - for text in ("No attribution marker", "Partner Center", "Technical configuration", - "SolutionMetadata.json", "rebuild", "https://learn.microsoft.com/"): - self.assertIn(text, result.stdout) + self.assertNotEqual(0, result.returncode) + for text in ("Packaging stopped", "Partner Center", "Technical", "configuration", + "SolutionMetadata.json", "rebuild", "publisherId", "offerId"): + self.assertIn(text, result.stderr) + self.ps_prelude = "" result = self.helper(warn=False) self.assertEqual({"resources": []}, json.loads(result.stdout)) def test_complete_ids_are_preserved_and_marker_shape_is_exact(self) -> None: + self.mock_lookup(failure="throw 'NETWORK MUST NOT RUN'; ") for tracking_id in (TRACKING_ID, "pid-contoso-myoffer-partnercenter"): with self.subTest(tracking_id=tracking_id): result = self.helper(tracking_id) self.assertEqual(0, result.returncode, result.stderr) self.assertEqual({"resources": [self.marker(tracking_id)]}, json.loads(result.stdout)) + def test_required_lookup_ignores_each_skip_control_and_persists_marker(self) -> None: + for value in (ABSENT, None, "", " \t"): + for skip, environment in ((False, False), (True, False), (False, True), (True, True)): + with self.subTest(value=value, skip=skip, environment=environment): + self.write_metadata(value) + self.mock_lookup() + if environment: + self.ps_prelude += "$env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP = '1'; " + extra = "-WarnIfMissing $true" + (" -SkipAttributionLookup" if skip else "") + result = self.lookup_helper(extra) + self.assertEqual(0, result.returncode, result.stdout + result.stderr) + self.assertIn("LOOKUP_CALLS=2", result.stdout) + self.assertEqual(TRACKING_ID, json.loads(self.metadata.read_text())["trackingId"]) + self.assertEqual({"resources": [self.marker(TRACKING_ID)]}, + json.loads((self.repository / "result.json").read_text())) + @staticmethod def marker(tracking_id: str) -> dict: return { @@ -451,16 +470,18 @@ def test_write_failure_and_concurrent_edit_are_hard_errors(self) -> None: def test_lookup_local_and_pipeline_use_authoritative_metadata_and_keep_cached_id_on_bump(self) -> None: entry = self.prepare_integration() for pipeline in (False, True): - for skip in (False, True): - with self.subTest(pipeline=pipeline, skip=skip): + for skip, environment in ((False, False), (True, False), (False, True)): + with self.subTest(pipeline=pipeline, skip=skip, environment=environment): self.write_metadata() self.mock_lookup() + if environment: + self.ps_prelude += "$env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP = '1'; " result = self.package(entry, pipeline, skip_lookup=skip) self.assertEqual(0, result.returncode, result.stdout + result.stderr) template = json.loads((self.solution / "Package/mainTemplate.json").read_text()) markers = [r for r in template["resources"] if r.get("name") == TRACKING_ID] - self.assertEqual(0 if skip else 1, len(markers)) - self.assertEqual(None if skip else TRACKING_ID, + self.assertEqual(1, len(markers)) + self.assertEqual(TRACKING_ID, json.loads(self.metadata.read_text()).get("trackingId")) self.write_metadata() self.mock_lookup() @@ -488,6 +509,7 @@ def test_local_and_positional_pipeline_preserve_nested_xdr_and_authoritative_id( entry = self.prepare_integration() self.write_metadata(TRACKING_ID) before = self.data.read_bytes(), self.metadata.read_bytes() + self.mock_lookup(failure="throw 'NETWORK MUST NOT RUN'; ") for pipeline in (False, True): with self.subTest(pipeline=pipeline): result = self.package(entry, pipeline) @@ -518,20 +540,49 @@ def test_local_and_positional_pipeline_preserve_nested_xdr_and_authoritative_id( self.assertEqual(template, json.loads(archive.read("mainTemplate.json").decode("utf-8-sig"))) self.assertEqual(before, (self.data.read_bytes(), self.metadata.read_bytes())) - def test_entry_points_warn_for_missing_id_and_fail_for_invalid_id(self) -> None: + def test_required_lookup_failures_stop_entry_points_before_artifacts_or_metadata_writes(self) -> None: entry = self.prepare_integration() for pipeline in (False, True): - with self.subTest(pipeline=pipeline): - self.write_metadata() - result = self.package(entry, pipeline) - self.assertEqual(0, result.returncode, result.stdout + result.stderr) - self.assertIn("CUSTOMER USAGE ATTRIBUTION:", result.stdout) - template = json.loads((self.solution / "Package/mainTemplate.json").read_text(encoding="utf-8-sig")) - self.assertEqual(1, len([r for r in template["resources"] if r["type"] == "Microsoft.Resources/deployments"])) - self.write_metadata("bad/id") - result = self.package(entry, pipeline) - self.assertNotEqual(0, result.returncode) - self.assertIn("Invalid trackingId", result.stderr) + for failure in ("network", "ambiguity", "invalid supplied ID"): + with self.subTest(pipeline=pipeline, failure=failure): + self.write_metadata("bad/id" if failure == "invalid supplied ID" else ABSENT) + before = self.metadata.read_bytes() + if failure == "ambiguity": + self.mock_lookup(catalog={"items": []}) + else: + self.mock_lookup(failure="throw [System.TimeoutException]::new('secret?sig=hidden'); ") + self.ps_prelude += "$env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP = '1'; " + result = self.package(entry, pipeline, skip_lookup=True) + self.assertNotEqual(0, result.returncode) + self.assertIn("Invalid trackingId" if failure == "invalid supplied ID" + else "Required published lookup failed", result.stderr) + self.assertNotIn("hidden", result.stdout + result.stderr) + self.assertNotIn("Package Generated Successfully", result.stdout) + self.assertEqual(before, self.metadata.read_bytes()) + self.assertFalse((self.solution / "Package").exists()) + self.assertEqual([], list(self.solution.glob("*.attribution-*"))) + + def test_optional_packaging_preserves_offline_skips_and_warning_only_failures(self) -> None: + entry = self.prepare_integration() + data = json.loads(self.data.read_text()) + del data["XDR Detections"] + self.data.write_text(json.dumps(data)) + for pipeline in (False, True): + for route in ("switch", "environment", "failure"): + with self.subTest(pipeline=pipeline, route=route): + self.write_metadata() + before = self.metadata.read_bytes() + self.mock_lookup(failure="throw [System.TimeoutException]::new(); ") + if route == "environment": + self.ps_prelude += "$env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP = '1'; " + result = self.package(entry, pipeline, skip_lookup=route == "switch") + self.assertEqual(0, result.returncode, result.stdout + result.stderr) + self.assertEqual(route == "failure", "Published lookup failed" in result.stdout) + self.assertEqual(before, self.metadata.read_bytes()) + template = json.loads((self.solution / "Package/mainTemplate.json").read_text()) + self.assertFalse(any(r["type"] == "Microsoft.Resources/deployments" + for r in template["resources"])) + self.assertTrue((self.solution / "Package/3.2.0.zip").exists()) if __name__ == "__main__": diff --git a/Tools/SentinelToXDRMigration/tests/test_defender_workbooks_packaging.py b/Tools/SentinelToXDRMigration/tests/test_defender_workbooks_packaging.py index b6f1582ec44..86c99512e3f 100644 --- a/Tools/SentinelToXDRMigration/tests/test_defender_workbooks_packaging.py +++ b/Tools/SentinelToXDRMigration/tests/test_defender_workbooks_packaging.py @@ -107,7 +107,8 @@ def tearDown(self) -> None: def write(self) -> None: self.target.write_text(json.dumps(self.notebook), encoding="utf-8") - def invoke(self, *, success: bool = True, package: bool = False, unchanged: bool = False) -> dict: + def invoke(self, *, success: bool = True, package: bool = False, unchanged: bool = False, + tracking_id: str | None = "pid-workbook-regression") -> dict: payload = self.root / "input.json" payload.write_text(json.dumps({ "data": self.data, "template": self.template, "ui": self.ui, @@ -120,7 +121,7 @@ def invoke(self, *, success: bool = True, package: bool = False, unchanged: bool ) if package: metadata = self.solution / "SolutionMetadata.json" - metadata.write_text("{}", encoding="utf-8") + metadata.write_text(json.dumps({"trackingId": tracking_id}), encoding="utf-8") emit = ( f"function git {{ $global:LASTEXITCODE=0; return {literal(self.root)} }}; " "$global:baseMainTemplate=$template; $global:baseCreateUiDefinition=$ui; " @@ -1001,11 +1002,22 @@ def test_generate_package_writes_independent_test_parameters_and_zip(self) -> No with zipfile.ZipFile(package / "3.2.7.zip") as archive: template = json.loads(archive.read("mainTemplate.json")) ui = json.loads(archive.read("createUiDefinition.json")) - self.assertEqual(4, len(template["resources"])) + self.assertEqual(5, len(template["resources"])) + marker = next(r for r in template["resources"] if r.get("name") == "pid-workbook-regression") + self.assertNotIn("condition", marker) + self.assertEqual([], marker["properties"]["template"]["resources"]) self.assertEqual("[steps('contentSelection').DeployDefenderWorkbook]", ui["parameters"]["outputs"]["DeployDefenderWorkbook"]) self.assertEqual(original, self.target.read_bytes()) self.assertIn("**XDR Workbooks:** 1", ui["parameters"]["config"]["basics"]["description"]) + def test_workbook_only_missing_attribution_cannot_skip_lookup(self) -> None: + result = self.invoke(package=True, success=False, tracking_id=None) + self.assertIn("Required published lookup failed", result["error"]) + self.assertIn("catalog identity", result["error"]) + self.assertFalse((self.solution / "Package").exists()) + self.assertEqual({"trackingId": None}, + json.loads((self.solution / "SolutionMetadata.json").read_text())) + def test_empty_selection_never_scans_or_emits_flag(self) -> None: self.data["XDR Workbooks"] = [] result = self.invoke() diff --git a/Tools/SentinelToXDRMigration/tests/test_native_workbooks.py b/Tools/SentinelToXDRMigration/tests/test_native_workbooks.py new file mode 100644 index 00000000000..fb3bfbc5d10 --- /dev/null +++ b/Tools/SentinelToXDRMigration/tests/test_native_workbooks.py @@ -0,0 +1,472 @@ +from __future__ import annotations + +from copy import deepcopy +import importlib.util +import json +from pathlib import Path +import shutil +import subprocess +import sys +from uuid import uuid4 +import zipfile + +import pytest + +from sentinel_xdr_migration import cli, native_workbooks as native, workbooks +from sentinel_xdr_migration._workbook_core import workbook + +ROOT = Path(__file__).resolve().parents[3] +SOURCE = "Workbook Sources/Native/native.json" + + +@pytest.fixture +def solution(): + workspace = Path.cwd() / f".native-workbook-test-{uuid4()}" + root = workspace / "Solutions" / "Sample" + (root / "Data").mkdir(parents=True) + (root / "Workbook Sources" / "Native").mkdir(parents=True) + (root / "Data" / "Solution_Sample.json").write_text(json.dumps({ + "Name": "Sample", "Author": "Synthetic - test@example.invalid", + "Version": "3.2.0", "Metadata": "SolutionMetadata.json", + "Description": "Synthetic native workbook import regression.", "XDR Workbooks": [], + }), encoding="utf-8") + (root / "SolutionMetadata.json").write_text(json.dumps({ + "name": "Sample", "version": "3.2.0", "trackingId": "pid-native-regression", + "support": {"name": "Synthetic", "email": "test@example.invalid", "tier": "Community"}, + "categories": {"domains": [], "verticals": []}, + }), encoding="utf-8") + (root / SOURCE).write_text(json.dumps({ + "version": "Notebook/1.0", "isLocked": False, "fallbackResourceIds": [], + "$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json", + "items": [ + {"type": 1, "content": {"json": "Notice: an ISOC workspace is required for incident visuals."}}, + {"type": 3, "content": { + "queryType": "advancedHunting", + "query": "// Preserve CRLF, spacing and Unicode: café\r\nEmailEvents | take 1\r\n", + }}, + {"type": 9, "content": {"parameters": [{ + "name": "Fixture", "queryType": "advancedHunting", + "query": 'datatable(Value:string)["literal"]', + }]}}, + ], + }, ensure_ascii=False), encoding="utf-8") + try: + yield root + finally: + shutil.rmtree(workspace) + + +def reviewed(solution): + plan = native.plan_native_workbook(solution, SOURCE) + plan.update(actor="fixture reviewer", reference="offline fixture review") + for query in plan["queries"]: + query.update(decision="accept", evidence="Read exact fixture KQL and complete input tables.") + return plan + + +def imported(solution): + return native.import_native_workbook(solution, SOURCE, reviewed(solution)) + + +def candidate(solution): + return solution / "XDR Workbooks" / "native.json" + + +def write_source_query(solution, query): + document = json.loads((solution / SOURCE).read_bytes()) + document["items"][1]["content"]["query"] = query + (solution / SOURCE).write_text(json.dumps(document), encoding="utf-8") + + +def test_native_import_preserves_notebook_and_stable_identity(solution): + original = (solution / SOURCE).read_bytes() + first = imported(solution) + raw = candidate(solution).read_bytes() + document = json.loads(raw) + assert document["contentProvenance"]["origin"] == "native-xdr" + assert "conversion" not in document["contentProvenance"] + payload = {k: v for k, v in document.items() if k not in ("xdrMetadata", "contentProvenance")} + assert payload == json.loads(original) + assert (solution / SOURCE).read_bytes() == original + assert not (solution / "Workbooks").exists() + assert first["validation"]["invalid"] == 0 + assert imported(solution)["contentId"] == first["contentId"] + assert candidate(solution).read_bytes() == raw + other = solution.parent.parent / "relocated" / "Sample" + shutil.copytree(solution, other) + assert native.identities(other, SOURCE) == (first["contentId"], first["instanceId"]) + # Identity is source-path based, not query-, file-hash- or release-based. + write_source_query(solution, "EmailEvents | take 2") + assert native.identities(solution, SOURCE) == (first["contentId"], first["instanceId"]) + assert workbooks.validate_workbooks(solution, selected_only=True)["invalid"] == 0 + (solution / SOURCE).unlink() + assert workbooks.validate_workbooks(solution, selected_only=True)["invalid"] == 0 + document = json.loads(candidate(solution).read_bytes()) + document["items"][1]["content"]["query"] += " | take 2" + candidate(solution).write_text(json.dumps(document), encoding="utf-8") + assert workbooks.validate_workbooks(solution, selected_only=True)["invalid"] == 1 + + +@pytest.mark.parametrize("mutation", [ + "missing-query", "duplicate-query", "stale-query", "stale-source", "empty-evidence", + "stale-payload", "defer", "missing-actor", "wrong-tables", "wrong-workload", "connectors", +]) +def test_review_is_complete_current_and_explicit(solution, mutation): + plan = reviewed(solution) + if mutation == "missing-query": + plan["queries"].pop() + elif mutation == "duplicate-query": + plan["queries"][1] = deepcopy(plan["queries"][0]) + elif mutation == "stale-query": + plan["queries"][0]["querySha256"] = "0" * 64 + elif mutation == "stale-source": + plan["source"]["sha256"] = "0" * 64 + elif mutation == "stale-payload": + plan["canonicalPayloadSha256"] = "0" * 64 + elif mutation == "empty-evidence": + plan["queries"][0]["evidence"] = " " + elif mutation == "defer": + plan["queries"][0]["decision"] = "defer" + elif mutation == "missing-actor": + plan["actor"] = "" + elif mutation == "wrong-tables": + plan["queries"][0]["tables"] = [] + elif mutation == "wrong-workload": + plan["requiredWorkloads"][0]["workload"] = "mde" + else: + plan["requiredDataConnectors"] = [{"connectorId": "invented", "dataTypes": ["EmailEvents"]}] + before = (solution / "Data" / "Solution_Sample.json").read_bytes() + with pytest.raises(ValueError): + native.import_native_workbook(solution, SOURCE, plan) + assert not candidate(solution).exists() + assert (solution / "Data" / "Solution_Sample.json").read_bytes() == before + + +@pytest.mark.parametrize("query", [ + "UnknownTable | take 1", "UnknownFunction() | take 1", + "EmailEvents | join (UnknownTable | take 1) on NetworkMessageId", + "EmailEvents | extend value=UnknownFunction(1)", + "union EmailEvents, UnknownTable", + "EmailEvents | join (SecurityIncident | take 1) on Title", + "let f=(){UnknownTable | take 1}; f()", + "table('EmailEvents') | take 1", "table({Table}) | take 1", + "print x=1", "range n from 1 to toscalar(UnknownFunction()) step 1", + 'datatable(x:string)["literal"] | join (UnknownTable) on x', + "workspace('other').EmailEvents | take 1", + "_CustomFunction() | take 1", "_CustomSource | take 1", + "_Im_Test | take 1", "_ASim_Test() | take 1", + "ASimAuthentication() | take 1", + "_Im_Authentication | extend x=_CustomFunction()", + "_Im_Authentication | join (UnknownFunction()) on EventType", + "_Im_Authentication | union SecurityEvent", + "_Im_Authentication | extend x=['_CustomFunction']()", + "union _Im_Authentication, _CustomSource", + "_ASim_Authentication() | union externaldata(x:string)['https://example.invalid/data']", + "_Im_Authentication | union cluster('other').database('db').EmailEvents", + 'print x="_Im_Authentication"', + "print _Im_Authentication=1", + "// _Im_Authentication\nprint x=1", +]) +def test_unknown_functions_tables_and_unproven_empty_lineage_block(solution, query): + write_source_query(solution, query) + plan = reviewed(solution) + assert plan["queries"][0]["blockers"] + # Removing blockers/claiming empty tables cannot override current discovery. + plan["queries"][0]["blockers"] = [] + with pytest.raises(ValueError): + native.import_native_workbook(solution, SOURCE, plan) + + +@pytest.mark.parametrize("query,tables", [ + ("range n from 1 to 5 step 1", []), + ('datatable(x:string)["EmailEvents"]', []), + ("EmailEvents | take 1", ["EmailEvents"]), + ("let x = EmailEvents | take 1; x | count", ["EmailEvents"]), + ("EmailEvents | join kind=inner (EmailUrlInfo) on NetworkMessageId", ["EmailEvents", "EmailUrlInfo"]), + ('let c=toscalar(EmailEvents | count); union withsource=Table (print x=c), (print x=c)', ["EmailEvents"]), +]) +def test_strict_recognizers_and_native_source_shapes(solution, query, tables): + write_source_query(solution, query) + plan = reviewed(solution) + assert plan["queries"][0]["tables"] == tables + assert not plan["queries"][0]["blockers"] + assert native.import_native_workbook(solution, SOURCE, plan)["validation"]["invalid"] == 0 + + +@pytest.mark.parametrize("query,tables", [ + ("_Im_Authentication | take 1", []), + ("_ASim_Authentication() | take 1", []), + ("_ASim_NetworkSession | count", []), + ("_ASim_NetworkSession(starttime=ago(1d)) | count", []), + ("_im_Dns() | count", []), + ("let events = _Im_Authentication; events | count", []), + ("union _Im_Authentication, _ASim_NetworkSession()", []), + ("['_ASim_Authentication']() | take 1", []), + ("_Im_Authentication | union EmailEvents | count", ["EmailEvents"]), +]) +def test_builtin_asim_roundtrip_does_not_invent_hidden_dependencies(solution, query, tables): + write_source_query(solution, query) + plan = reviewed(solution) + assert plan["queries"][0]["tables"] == tables + assert plan["queries"][0]["blockers"] == [] + assert plan["requiredWorkloads"] == ( + [{"workload": "mdo", "dataTypes": ["EmailEvents"]}] if tables else []) + assert plan["requiredDataConnectors"] == [] + assert native.review_native_workbook(solution, SOURCE, plan) == plan + result = native.import_native_workbook(solution, SOURCE, plan) + assert result["runtimeQualified"] is False + assert result["validation"]["invalid"] == 0 + document = json.loads(candidate(solution).read_bytes()) + assert document["items"][1]["content"]["query"] == query + assert document["contentProvenance"]["requiredWorkloads"] == plan["requiredWorkloads"] + (solution / SOURCE).unlink() + assert workbooks.validate_workbooks(solution, selected_only=True)["invalid"] == 0 + + +@pytest.mark.parametrize("mutation", ["stale-query", "invented-tables", "invented-workload"]) +def test_builtin_asim_review_cannot_override_hash_or_dependencies(solution, mutation): + write_source_query(solution, "_Im_Authentication | take 1") + plan = reviewed(solution) + if mutation == "stale-query": + write_source_query(solution, "_ASim_Authentication() | take 1") + elif mutation == "invented-tables": + plan["queries"][0]["tables"] = ["IdentityLogonEvents"] + else: + plan["requiredWorkloads"] = [{"workload": "mdi", "dataTypes": ["IdentityLogonEvents"]}] + with pytest.raises(ValueError): + native.import_native_workbook(solution, SOURCE, plan) + assert not candidate(solution).exists() + + +@pytest.mark.parametrize("source", [ + "../native.json", "Workbook Sources/Native/../../escape.json", + "Workbook Sources/Native/./native.json", "Workbook Sources/Native//native.json", + "Workbooks/native.json", "Workbook Sources\\Native\\native.json", + "Workbook Sources/Native/missing.json", "C:/outside.json", +]) +def test_unsafe_or_missing_source_paths_fail(solution, source): + with pytest.raises(ValueError): + native.plan_native_workbook(solution, source) + + +def test_symlink_source_is_rejected(solution): + link = solution / "Workbook Sources" / "Native" / "link.json" + try: + link.symlink_to(solution / SOURCE) + except OSError as exc: + pytest.skip(f"Symlink privilege unavailable: {exc}") + with pytest.raises(ValueError, match="Symlink"): + native.plan_native_workbook(solution, "Workbook Sources/Native/link.json") + + +def test_bad_selection_and_existing_identity_are_not_mutated_by_import(solution): + data_file = solution / "Data" / "Solution_Sample.json" + data = json.loads(data_file.read_bytes()) + data["XDR Workbooks"] = ["XDR Workbooks/missing.json"] + data_file.write_text(json.dumps(data), encoding="utf-8") + previous = data_file.read_bytes() + with pytest.raises(ValueError, match="Missing workbook"): + imported(solution) + assert data_file.read_bytes() == previous + assert not candidate(solution).exists() + data["XDR Workbooks"] = [] + data_file.write_text(json.dumps(data), encoding="utf-8") + imported(solution) + duplicate = candidate(solution).parent / "duplicate.json" + duplicate.write_bytes(candidate(solution).read_bytes()) + candidate(solution).unlink() + data["XDR Workbooks"] = ["XDR Workbooks/duplicate.json"] + data_file.write_text(json.dumps(data), encoding="utf-8") + with pytest.raises(ValueError, match="Duplicate"): + imported(solution) + assert not candidate(solution).exists() + + +@pytest.mark.parametrize("provider", [0, "logAnalytics", ""]) +def test_native_provider_is_not_converted(solution, provider): + path = solution / SOURCE + document = json.loads(path.read_bytes()) + document["items"][1]["content"]["queryType"] = provider + path.write_text(json.dumps(document), encoding="utf-8") + assert native.plan_native_workbook(solution, SOURCE)["queries"][0]["blockers"] + with pytest.raises(ValueError, match="advancedHunting"): + imported(solution) + + +@pytest.mark.parametrize("mutation", [ + "query", "notice", "hash", "identity", "duplicate", "migration-origin", + "payload-hash", "dependency", "review", "review-hash", +]) +def test_authored_tampering_fails_closed(solution, mutation): + imported(solution) + document = json.loads(candidate(solution).read_bytes()) + if mutation == "query": + document["items"][1]["content"]["query"] += " | take 2" + elif mutation == "notice": + document["items"][0]["content"]["json"] = "changed" + elif mutation == "hash": + document["contentProvenance"]["source"]["sha256"] = "0" * 64 + elif mutation == "payload-hash": + document["contentProvenance"]["canonicalPayloadSha256"] = "0" * 64 + elif mutation == "dependency": + document["contentProvenance"]["requiredWorkloads"] = [] + elif mutation == "review": + document["contentProvenance"]["nativeReview"]["queries"][0]["evidence"] = "" + elif mutation == "review-hash": + document["contentProvenance"]["nativeReview"]["canonicalPayloadSha256"] = "0" * 64 + elif mutation == "identity": + document["xdrMetadata"]["contentId"] = str(uuid4()) + elif mutation == "migration-origin": + document["contentProvenance"].pop("origin") + else: + (candidate(solution).parent / "duplicate.json").write_bytes(candidate(solution).read_bytes()) + candidate(solution).write_text(json.dumps(document), encoding="utf-8") + assert workbooks.validate_workbooks(solution)["invalid"] > 0 + + +@pytest.mark.parametrize("query", ["EmailEvents | take 1", "_Im_Authentication | take 1"]) +def test_cli_roundtrip_and_duplicate_json_keys(solution, capsys, query): + write_source_query(solution, query) + assert cli.main(["native-workbook-plan", "--solution", str(solution), "--source", SOURCE]) == 0 + plan = json.loads(capsys.readouterr().out) + plan.update(actor="CLI reviewer", reference="CLI regression") + for query in plan["queries"]: + query.update(decision="accept", evidence="Explicit current query review.") + review = solution / "review.json" + review.write_text(json.dumps(plan), encoding="utf-8") + assert cli.main(["native-workbook-review", "--solution", str(solution), "--source", SOURCE, + "--review", str(review)]) == 0 + assert json.loads(capsys.readouterr().out) == plan + assert cli.main(["native-workbook-import", "--solution", str(solution), "--source", SOURCE, + "--review", str(review)]) == 0 + capsys.readouterr() + assert cli.main(["native-workbook-validate", "--solution", str(solution)]) == 0 + write_source_query(solution, "UnknownFunction()") + assert cli.main(["native-workbook-validate", "--solution", str(solution)]) == 0 + document = json.loads(candidate(solution).read_bytes()) + document["items"][1]["content"]["query"] = "UnknownFunction()" + candidate(solution).write_text(json.dumps(document), encoding="utf-8") + assert cli.main(["native-workbook-validate", "--solution", str(solution)]) == 1 + review.write_text('{"queries": [], "queries": []}', encoding="utf-8") + assert cli.main(["native-workbook-import", "--solution", str(solution), "--source", SOURCE, + "--review", str(review)]) == 2 + + +@pytest.mark.skipif(not shutil.which("pwsh"), reason="PowerShell 7 required") +@pytest.mark.parametrize("query", [ + "EmailEvents | take 1", "_Im_Authentication | take 1", "_ASim_NetworkSession() | count", +]) +def test_native_only_v31_package_requires_no_sentinel_parent(solution, monkeypatch, query): + monkeypatch.setenv("SENTINEL_XDR_PYTHON", sys.executable) + write_source_query(solution, query) + imported(solution) + (solution / SOURCE).unlink() + content_root = solution.parent.parent + subprocess.run(["git", "init", "--quiet", str(content_root)], check=True) + result = subprocess.run([ + "pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-File", + str(ROOT / "Tools" / "Create-Azure-Sentinel-Solution" / "V3" / "createSolutionV3_1.ps1"), + "-SolutionDataFolderPath", str(solution / "Data"), "-ToolkitRootPath", str(ROOT), + "-VersionMode", "local", "-VersionBump", "none", + "-ArmValidationMethod", "lab-deployment", "-SkipAttributionLookup", + ], cwd=content_root, capture_output=True, text=True, encoding="utf-8") + assert result.returncode == 0, result.stdout + result.stderr + template = json.loads((solution / "Package" / "mainTemplate.json").read_bytes()) + assert not (solution / "Workbooks").exists() + assert "DeployWorkbook" not in template["parameters"] + assert len([r for r in template["resources"] + if r.get("properties", {}).get("contentKind") == "DefenderWorkbook"]) == 1 + assert not [r for r in template["resources"] + if r.get("properties", {}).get("contentKind") == "Workbook"] + from sentinel_xdr_migration.packaging import _verify_workbook_package + _verify_workbook_package(solution, workbooks.selected_workbook_files(solution), template, + next((solution / "Package").glob("*.zip"))) + + +@pytest.mark.skipif(not shutil.which("pwsh"), reason="PowerShell 7 required") +def test_real_v31_mixed_package_and_static_validator(solution, monkeypatch): + monkeypatch.setenv("SENTINEL_XDR_PYTHON", sys.executable) + sentinel = solution / "Workbooks" / "sentinel.json" + sentinel.parent.mkdir() + source = {"version": "Notebook/1.0", "fromTemplateId": str(uuid4()), "items": [ + {"type": 3, "content": {"queryType": 0, "resourceType": workbook.LA, + "query": "DeviceEvents | take 1"}}, + ]} + sentinel.write_text(json.dumps(source), encoding="utf-8") + metadata_dir = solution.parent.parent / "Workbooks" + metadata_dir.mkdir() + (metadata_dir / "WorkbooksMetadata.json").write_text(json.dumps([{ + "templateRelativePath": "sentinel.json", "workbookKey": source["fromTemplateId"], + "title": "Sentinel fixture", "description": "Synthetic Sentinel workbook", + "version": "3.2.0", "dataTypesDependencies": ["DeviceEvents"], + "dataConnectorsDependencies": [], + }]), encoding="utf-8") + conversion = workbooks.convert_workbooks(solution, report_directory=solution / "Logs", + config={"workbookResolutions": {"Workbooks/sentinel.json": { + "/items/0/content/query": { + "querySha256": workbook.query_hash(source["items"][0]["content"]["query"]), + "tables": ["DeviceEvents"], "evidence": "Reviewed synthetic source", + }}}}) + assert conversion["converted"] == 1 + data_file = solution / "Data" / "Solution_Sample.json" + data = json.loads(data_file.read_bytes()) + data.update({"Workbooks": ["Workbooks/sentinel.json"], + "XDR Workbooks": ["XDR Workbooks/sentinel.json"]}) + data["Metadata"] = data.pop("Metadata") + data_file.write_text(json.dumps(data), encoding="utf-8") + migrated = (solution / "XDR Workbooks" / "sentinel.json").read_bytes() + imported(solution) + (solution / SOURCE).unlink() + assert (solution / "XDR Workbooks" / "sentinel.json").read_bytes() == migrated + assert workbooks.validate_workbooks(solution)["invalid"] == 0 + subprocess.run(["git", "init", "--quiet", str(solution.parent.parent)], check=True) + arguments = [ + str(ROOT / "Tools" / "Create-Azure-Sentinel-Solution" / "V3" / "createSolutionV3_1.ps1"), + "-SolutionDataFolderPath", str(solution / "Data"), "-ToolkitRootPath", str(ROOT), + "-VersionMode", "local", "-VersionBump", "none", + "-ArmValidationMethod", "lab-deployment", "-SkipAttributionLookup"] + def literal(value): + return "'" + value.replace("'", "''") + "'" + invocation = "& " + literal(arguments[0]) + " " + " ".join( + arg if arg.startswith("-") else literal(arg) for arg in arguments[1:]) + command = ["pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", + "try { " + invocation + " } catch { Write-Output $_.ScriptStackTrace; throw }"] + result = subprocess.run(command, cwd=solution.parent.parent, capture_output=True, text=True, encoding="utf-8") + assert result.returncode == 0, result.stdout + result.stderr + template = json.loads((solution / "Package" / "mainTemplate.json").read_bytes()) + resources = template["resources"] + installs = [r for r in resources if r["type"] == "Microsoft.Resources/deployments" + and r["name"].startswith("DefenderWorkbook-")] + registrations = [r for r in resources if r.get("properties", {}).get("contentKind") == "DefenderWorkbook"] + assert len(installs) == len(registrations) == 2 + sentinel_registrations = [r for r in resources if r.get("properties", {}).get("contentKind") == "Workbook"] + assert len(sentinel_registrations) == 1 + assert len([r for r in sentinel_registrations[0]["properties"]["mainTemplate"]["resources"] + if r["type"] == "Microsoft.Insights/workbooks"]) == 1 + document = json.loads(candidate(solution).read_bytes()) + install = next(r for r in installs if + r["properties"]["template"]["resources"]["workbook"]["properties"]["id"] == document["xdrMetadata"]["instanceId"]) + notebook = install["properties"]["template"]["resources"]["workbook"]["properties"]["content"] + assert notebook["items"] == document["items"] + assert "xdrMetadata" not in notebook + assert set(notebook["contentProvenance"]) == {"packageId", "packageVersion", "contentId", "contentVersion"} + assert template["parameters"]["DeployDefenderWorkbook"]["defaultValue"] is False + assert json.loads(data_file.read_bytes())["Version"] == "3.2.0" + from sentinel_xdr_migration.packaging import _verify_workbook_package + zip_path = next((solution / "Package").glob("*.zip")) + _verify_workbook_package(solution, workbooks.selected_workbook_files(solution), template, zip_path) + with zipfile.ZipFile(zip_path) as archive: + assert any(name.endswith("mainTemplate.json") for name in archive.namelist()) + spec = importlib.util.spec_from_file_location( + "native_static_adapter", ROOT / ".script" / "local-validation" / "validate-xdr-content.py") + adapter = importlib.util.module_from_spec(spec) + spec.loader.exec_module(adapter) + static = adapter.validate_solution_xdr_workbooks(solution, solution.parent.parent, ROOT) + assert all(row["passed"] for row in static["results"]), static + # The same V3.1 entry point must reject a tampered native query, not merely + # rely on the caller having run validation previously. + document["items"][1]["content"]["query"] += " | take 2" + candidate(solution).write_text(json.dumps(document), encoding="utf-8") + failed = subprocess.run(command, cwd=solution.parent.parent, capture_output=True, text=True, encoding="utf-8") + assert failed.returncode != 0 + assert "Native Defender Workbook validation failed" in failed.stdout + failed.stderr