Toolkit: native XDR workbook imports and required Marketplace attribution

Tools/Create-Azure-Sentinel-Solution/V3/README-V3_1.md

@@ -48,6 +48,8 @@ The 12 positional pipeline parameters (positions 0–11) are unchanged.
 The optional named `-SkipAttributionLookup` switch is available in both local
 and pipeline parameter sets. For CI callers that do not forward optional
 arguments, set `SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1` in the process environment.
+Both controls apply only when attribution is optional; emitted XDR detections
+or Defender Workbooks require attribution and cannot skip missing-ID lookup.
 
 ## Independent content selection (V3.1 default)
 
@@ -114,6 +116,33 @@ uninstall operation. There is no new all-off runtime check. A selection can
 leave prerequisites unmet: the booleans do not rewrite dependency semantics,
 guarantee Content Hub acceptance, or provision missing external dependencies.
 
+### Native Defender Workbooks without a Sentinel parent
+
+Already-native Advanced Hunting workbooks use the explicit
+`native-workbook-plan` / `native-workbook-review` / `native-workbook-import` authoring route documented in
+[the migration toolkit README](../../SentinelToXDRMigration/README.md#native-defender-workbook-authoringimport-no-sentinel-parent).
+The raw source may be supplied under solution-local `Workbook Sources/Native/`
+while running the authoring commands, but it need not be retained or shipped.
+Select only the authored `XDR Workbooks` candidate; no duplicate source artifact
+is packaged. `contentProvenance.origin: native-xdr` selects native review with
+historical source path/hash and a canonical reviewed Notebook payload hash,
+instead of migrated-source converter proof. It does not weaken the existing
+migrated workbook contract.
+
+The V3.1 emitter invokes the shared native Python validator before any template
+mutation, requiring Python 3.11/3.12 with the toolkit dependencies.
+`SENTINEL_XDR_PYTHON` selects the interpreter (default: `python`); the subprocess
+uses this emitter's toolkit checkout, not another editable installation.
+Unknown functions/tables, payload/query changes, and stale/incomplete reviews
+block packaging. Source deletion after import does not invalidate a native
+candidate.
+Native and migrated entries share managed install/registration, independent
+Sentinel selection, default-false `DeployDefenderWorkbook`, and the installed
+provenance quartet. Native sources/reviews are not installed or ZIP entries.
+No Sentinel workbook, converter receipt or runtime acceptance is required or
+manufactured for the native authoring route. Provider acceptance still requires
+the existing independent approved evidence.
+
 ### Defender Workbook parser prerequisites
 
 Authored workbooks declare reviewed function prerequisites in
@@ -301,9 +330,9 @@ These guards do not govern ARM `apiVersion`, `contentSchemaVersion`, ARM templat
 saved-search API revision numbers. Legacy `V3/createSolutionV3.ps1` and
 `pipeline/createSolutionV4.ps1` retain their previous version behavior.
 
-## Optional customer usage attribution
+## Customer usage attribution
 
-Set optional `trackingId` in the solution's **SolutionMetadata.json** to the
+Set `trackingId` in the solution's **SolutionMetadata.json** to the
 **complete customer usage attribution tracking ID** copied from Partner Center
 **offer > plan > Technical configuration**, for example:
 
@@ -364,24 +393,34 @@ Version catalog lookup and `GetPackageVersion` behavior are unchanged.
 Attribution requests use a 20-second request/connection timeout and, on
 PowerShell 7.4+, an explicit 20-second stalled-read timeout, with no retries or
 redirects (continuous downloads are not capped at 20 seconds).
-HTTP status, timeout, network/read, JSON, selection and marker failures produce
-classified `CUSTOMER USAGE ATTRIBUTION:` warnings and continue **without**
-attribution or metadata changes. A successful lookup logs publisher, offer,
+For packages emitting XDR detections or Defender Workbooks, attribution is
+**required**. HTTP status, timeout, network/read, JSON, selection and marker
+failures produce a classified, terminating `CUSTOMER USAGE ATTRIBUTION:` error
+before package files or the ZIP are written, leaving metadata unchanged.
+Verify the correct publisher/offer identity and unambiguous published plan/marker,
+or supply the complete ID from Partner Center and rebuild. There is no
+incomplete-package override. When attribution is optional (no emitted XDR
+detections or Defender Workbooks), lookup failures retain the warning-only
+behavior and continue **without** attribution or metadata changes.
+A successful lookup logs publisher, offer,
 plan ID, template URL (without query credentials), tracking ID and updated path
 as `CUSTOMER USAGE ATTRIBUTION SOURCE:`. It is not reported as a warning.
 
-For offline attribution, use `-SkipAttributionLookup` or set
-`SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1`; explicit IDs still emit markers. Use
+For optional offline attribution, use `-SkipAttributionLookup` or set
+`SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1`; explicit IDs still emit markers.
+Neither control suppresses lookup when required attribution has a missing ID.
+A valid explicit ID avoids attribution network access even for required packages. Use
 `-VersionMode local` as well to avoid the **independent** version-catalog lookup.
 These controls do not disable other packaging prerequisites or validators.
 The migration CLI inherits the same environment control, so no redundant CLI
 flag is required. Its packaging report preserves lookup failures in `warnings`
 and successful source messages in `attributionSources`.
 
-If no ID is available, nested XDR packaging still warns that Partner Center
-cannot automatically add tracking for functional nested deployments.
-Obtain the ID from offer > plan > Technical configuration, add `trackingId`,
-and rebuild. Missing attribution does not imply Marketplace upload rejection.
+Partner Center cannot automatically add tracking for functional nested
+deployments. If required lookup cannot resolve an ID, packaging fails rather
+than emitting a markerless package. Obtain the ID from offer > plan > Technical
+configuration, add `trackingId`, and rebuild. This packaging guard is not a
+claim about Marketplace upload rejection.
 See [Microsoft Marketplace Azure apps customer usage attribution](https://learn.microsoft.com/partner-center/marketplace-offers/azure-partner-customer-usage-attribution#microsoft-marketplace-azure-apps).
 
 The migration CLI uses `package-v3-1` and writes `packaging.v3_1.json` with
@@ -399,10 +438,11 @@ PYTHONPATH=Tools/SentinelToXDRMigration:Tools/SentinelToXDRMigration/tests pytho
 The attribution tests exercise both local and positional pipeline entry points
 with isolated synthetic solutions, verify generated JSON and ZIP contents, and
 cover missing/invalid IDs and nested XDR preservation. Catalog/template requests
-are mocked; default test subprocesses disable lookup, so unit tests never depend
+are mocked; optional test subprocesses disable lookup, so unit tests never depend
 on live Marketplace availability or mutate real solution metadata. Cases include
 automatic persistence/replay, explicit overrides, ambiguous identities/plans/
-markers, remote failures, offline controls, byte preservation, local write
+markers, required lookup despite both offline controls, terminating failures
+before artifact output, optional offline behavior, byte preservation, local write
 failures/concurrent edits and cached-ID retention during version bumps.
 They stub the unrelated
 full repository validators and isolate repository-root resolution. They do not

Tools/Create-Azure-Sentinel-Solution/common/AttributionMetadata.cs

@@ -77,7 +77,12 @@ public static void Save(string path, byte[] expected, string trackingId)
                     typeof(File).GetMethod("SetUnixFileMode", new[] { typeof(string), mode.GetType() })
                         .Invoke(null, new object[] { staging, mode });
                 }
-                File.Move(staging, path, true);
+                // Windows overwrite-move fails while the destination is open;
+                // Replace supports the writer-excluding, delete-sharing handle.
+                if (Path.DirectorySeparatorChar == '\\')
+                    File.Replace(staging, path, null);
+                else
+                    File.Move(staging, path, true);
             }
         }
         finally

Tools/Create-Azure-Sentinel-Solution/common/commonFunctions.ps1

@@ -3313,6 +3313,11 @@ function GeneratePackage(
     $baseMainTemplate.metadata.author = $(if ($contentToImport.Author) { $contentToImport.Author } else { "" })
     $baseMainTemplate.metadata.comments = $baseMainTemplate.metadata.comments -replace "{{SolutionName}}", $solutionName
 
+    if ($IncludeXdrDetections) {
+        # Resolve required attribution after content/version preparation, before any package output.
+        Add-CustomerUsageAttribution -SolutionMetadataPath $SolutionMetadataPath -Template $global:baseMainTemplate -WarnIfMissing ($xdrDetectionCount -gt 0 -or $defenderWorkbookCount -gt 0) -SkipAttributionLookup:$SkipAttributionLookup
+    }
+
     $repoRoot = $(git rev-parse --show-toplevel)
     $solutionFolderName = $solutionName
     $solutionFolder = "$repoRoot/Solutions/$solutionFolderName"
@@ -3326,11 +3331,6 @@ function GeneratePackage(
     $mainTemplateOutputPath = "$solutionFolder/mainTemplate.json"
     $createUiDefinitionOutputPath = "$solutionFolder/createUiDefinition.json"
 
-    if ($IncludeXdrDetections) {
-        # Persist only after content/version preparation, after metadata filtering and before output.
-        Add-CustomerUsageAttribution -SolutionMetadataPath $SolutionMetadataPath -Template $global:baseMainTemplate -WarnIfMissing ($xdrDetectionCount -gt 0 -or $defenderWorkbookCount -gt 0) -SkipAttributionLookup:$SkipAttributionLookup
-    }
-
     try {
         $baseMainTemplate | ConvertTo-Json -Depth $packageSerializationDepth | Out-File $mainTemplateOutputPath -Encoding utf8
 

Tools/Create-Azure-Sentinel-Solution/common/customerUsageAttribution.ps1

@@ -141,6 +141,7 @@ function Add-CustomerUsageAttribution {
         [string]$SolutionMetadataPath,
         [Parameter(Mandatory = $true)]
         [psobject]$Template,
+        # Retain the existing parameter name; nested XDR content requires attribution.
         [bool]$WarnIfMissing = $false,
         [switch]$SkipAttributionLookup
     )
@@ -152,19 +153,22 @@ function Add-CustomerUsageAttribution {
     $trackingId = $metadata.trackingId
     $discovered = $null
     if ($null -eq $trackingId -or ($trackingId -is [string] -and [string]::IsNullOrWhiteSpace($trackingId))) {
-        if (-not $SkipAttributionLookup -and $env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP -ne '1') {
+        if ($WarnIfMissing -or (-not $SkipAttributionLookup -and $env:SENTINEL_SKIP_ATTRIBUTION_LOOKUP -ne '1')) {
             try {
                 $discovered = Resolve-PublishedCustomerUsageAttribution -Metadata $metadata
                 $trackingId = $discovered.TrackingId
             }
             catch {
+                if ($WarnIfMissing) {
+                    throw "CUSTOMER USAGE ATTRIBUTION: Required published lookup failed [$($_.Exception.Message)]. Packaging stopped; SolutionMetadata.json is unchanged. Verify publisherId and offerId identify the correct published offer with one unambiguous plan/marker, or supply the complete trackingId from Partner Center > offer > plan > Technical configuration in '$SolutionMetadataPath' and rebuild. Attribution lookup cannot be skipped for nested XDR content."
+                }
                 Write-Warning "CUSTOMER USAGE ATTRIBUTION: Published lookup failed [$($_.Exception.Message)]. No attribution marker will be emitted; SolutionMetadata.json is unchanged. Supply the complete trackingId from Partner Center > offer > plan > Technical configuration and rebuild."
             }
         }
     }
     if ($null -eq $trackingId -or ($trackingId -is [string] -and [string]::IsNullOrWhiteSpace($trackingId))) {
         if ($WarnIfMissing) {
-            Write-Warning "CUSTOMER USAGE ATTRIBUTION: No attribution marker will be emitted because trackingId is absent or blank in '$SolutionMetadataPath'. Partner Center cannot automatically add tracking when the template contains functional nested deployments (including XDR). Obtain the COMPLETE customer usage attribution tracking ID from Partner Center > offer > plan > Technical configuration, add trackingId to SolutionMetadata.json, and rebuild. Do not use a Partner ID, plan ID, or a newly generated GUID. See https://learn.microsoft.com/partner-center/marketplace-offers/azure-partner-customer-usage-attribution#microsoft-marketplace-azure-apps"
+            throw "CUSTOMER USAGE ATTRIBUTION: Required trackingId is still absent or blank in '$SolutionMetadataPath'. Packaging stopped; SolutionMetadata.json is unchanged. Verify publisherId and offerId, or supply the complete trackingId from Partner Center > offer > plan > Technical configuration and rebuild. Attribution lookup cannot be skipped for nested XDR content."
         }
         return
     }

Tools/Create-Azure-Sentinel-Solution/common/defenderWorkbooks.ps1

@@ -44,6 +44,47 @@ function Assert-DefenderWorkbookReleaseVersion {
     }
 }
 
+function Assert-NativeDefenderWorkbookDocument {
+    param([psobject]$Document, [string]$SolutionRoot)
+
+    $python = if ($env:SENTINEL_XDR_PYTHON) { $env:SENTINEL_XDR_PYTHON } else { 'python' }
+    $start = [System.Diagnostics.ProcessStartInfo]::new()
+    $start.FileName = $python
+    $start.UseShellExecute = $false
+    $start.RedirectStandardInput = $true
+    $start.RedirectStandardOutput = $true
+    $start.RedirectStandardError = $true
+    $start.StandardInputEncoding = [System.Text.UTF8Encoding]::new($false)
+    $start.StandardOutputEncoding = [System.Text.UTF8Encoding]::new($false)
+    foreach ($argument in @('-m', 'sentinel_xdr_migration.native_workbooks', $SolutionRoot)) {
+        $start.ArgumentList.Add($argument)
+    }
+    $start.Environment['PYTHONPATH'] = [System.IO.Path]::GetFullPath(
+        (Join-Path $PSScriptRoot '../../SentinelToXDRMigration'))
+    $start.Environment['PYTHONIOENCODING'] = 'utf-8'
+    $process = [System.Diagnostics.Process]::new()
+    $process.StartInfo = $start
+    try {
+        [void]$process.Start()
+        $stdout = $process.StandardOutput.ReadToEndAsync()
+        $stderr = $process.StandardError.ReadToEndAsync()
+        $process.StandardInput.Write(($Document | ConvertTo-Json -Depth 100 -Compress))
+        $process.StandardInput.Close()
+        $process.WaitForExit()
+        $output = $stdout.GetAwaiter().GetResult()
+        $diagnostic = $stderr.GetAwaiter().GetResult()
+        if ($process.ExitCode -ne 0) {
+            throw "Native Defender Workbook validation failed: $output $diagnostic"
+        }
+        $receipt = $output | ConvertFrom-Json -ErrorAction Stop
+        if ($null -eq $receipt -or $receipt.errors -isnot [array] -or $receipt.errors.Count -ne 0) {
+            throw 'Native Defender Workbook validation returned no valid receipt.'
+        }
+    } finally {
+        $process.Dispose()
+    }
+}
+
 function Assert-DefenderWorkbookDocument {
     param([psobject]$Document, [string]$SolutionRoot, [string]$SolutionName, [string]$ContentPath, [switch]$PassThruSource)
 
@@ -63,6 +104,21 @@ function Assert-DefenderWorkbookDocument {
     }
     Assert-DefenderWorkbookReleaseVersion -Version $metadata.version
     $provenance = $Document.contentProvenance
+    if ($null -ne $provenance -and $provenance.origin -ceq 'native-xdr') {
+        Assert-NativeDefenderWorkbookDocument -Document $Document -SolutionRoot $SolutionRoot
+        if ($PassThruSource) {
+            $nativePayload = $Document | ConvertTo-Json -Depth 100 -Compress |
+                ConvertFrom-Json -ErrorAction Stop
+            $nativePayload.PSObject.Properties.Remove('xdrMetadata')
+            $nativePayload.PSObject.Properties.Remove('contentProvenance')
+            return $nativePayload
+        }
+        return
+    }
+    if ($null -ne $provenance.PSObject.Properties['origin'] -and
+        $provenance.origin -cne 'sentinel-migration') {
+        throw "Defender Workbook '$ContentPath' has an unknown origin."
+    }
     if ($provenance -isnot [pscustomobject] -or
         $provenance.dependencyReviewRequired -isnot [bool] -or $provenance.dependencyReviewRequired -or
         $provenance.conversion -isnot [pscustomobject] -or
@@ -704,6 +760,11 @@ function Get-DefenderWorkbookParserDependencies {
     $sourceNotebook = Assert-DefenderWorkbookDocument -Document $Document -SolutionRoot $SolutionRoot `
         -SolutionName $SolutionName -ContentPath 'prerequisite scan' -PassThruSource
     $provenance = $Document.contentProvenance
+    if ($provenance.origin -ceq 'native-xdr') {
+        # The shared native validator rejects functions and non-native tables;
+        # native imports have no savedSearch/parser deployment prerequisites.
+        return
+    }
     $declared = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::Ordinal)
     $property = $provenance.PSObject.Properties['requiredWorkspaceFunctions']
     if ($null -ne $property -and $property.Value -isnot [array]) {
@@ -1061,9 +1122,15 @@ function Add-XdrDefenderWorkbooksToSolution {
     $contentPackages = @($Template.resources | Where-Object {
         $_.type -ieq 'Microsoft.OperationalInsights/workspaces/providers/contentPackages'
     })
-    if ($contentPackages.Count -ne 1 -or
-        $null -eq $contentPackages[0].properties.dependencies.PSObject.Properties['criteria']) {
-        throw 'Defender Workbook packaging requires exactly one contentPackages dependency criteria resource.'
+    if ($contentPackages.Count -ne 1 -or $null -eq $contentPackages[0].properties) {
+        throw 'Defender Workbook packaging requires exactly one contentPackages resource.'
+    }
+    $dependencyProperty = $contentPackages[0].properties.PSObject.Properties['dependencies']
+    if ($null -ne $dependencyProperty -and
+        ($null -eq $dependencyProperty.Value -or
+         $null -eq $dependencyProperty.Value.PSObject.Properties['criteria'] -or
+         $dependencyProperty.Value.criteria -isnot [array])) {
+        throw 'Defender Workbook packaging requires valid contentPackages dependency criteria.'
     }
     $resources = @()
     $criteria = @()
@@ -1134,7 +1201,11 @@ function Add-XdrDefenderWorkbooksToSolution {
     )
     # Exempt only reviewed prerequisites and their registration faces from DeployParser.
     Register-SolutionRequiredResources -Resources $requiredResources
-    $Template.resources += $resources
+    $Template.resources = @($Template.resources) + @($resources)
+    if ($null -eq $dependencyProperty) {
+        $contentPackages[0].properties | Add-Member -NotePropertyName dependencies -NotePropertyValue (
+            [pscustomobject]@{ operator = 'AND'; criteria = @() })
+    }
     $existing = @($contentPackages[0].properties.dependencies.criteria | Where-Object { $_.kind -ine 'DefenderWorkbook' })
     $contentPackages[0].properties.dependencies.criteria = @($existing) + @($criteria)
     Write-Warning 'Defender Workbook install/registration is an UNVERIFIED candidate contract. Approved lab installation, registration readback, identity/provenance and repeat-install/update evidence are required; generation does not establish provider acceptance.'