[kingwil]

Summary

Correct native XDR workbook validation so the authored XDR Workbooks Notebook is the sole required deployable artifact. The raw Notebook is an authoring input for plan/review/import; it may be removed after import and is never copied into or required by the package. This PR does not ship a duplicate Workbook Source.

The native artifact retains the truthful original source path and exact-byte SHA256 as historical import evidence and keeps the existing source-path-based UUIDv5 identities. Validation does not read that path. Instead, plan, nativeReview, and provenance require the same canonicalPayloadSha256: SHA256 over canonical UTF-8 JSON (sorted keys, compact separators, non-ASCII preserved) of the complete Notebook excluding only top-level xdrMetadata and contentProvenance. This catches changes to notices, tabs, layout, or any other payload content. Validation independently recomputes current query hashes, strict lineage, and workload dependencies; stale or incomplete reviews and hash mismatches still fail closed.

Changes

• Keep migrated-workbook source inventory and validation behavior unchanged.
• Validate native artifact content against its reviewed canonical payload hash rather than rereading an external source.
• Include the canonical payload hash in native plans, reviews, authored provenance, and both mirrored schema copies; refresh the imported-schema hash in import-manifest.json.
• Make V3.1 PowerShell PassThruSource return the already-validated stripped native Notebook, not load the historical source file.
• Document the exact CLI route, transient source lifecycle, historical source hash versus reviewed payload hash, and candidate-only packaging.
• Add regressions for source removal in native-only and mixed V3.1 packaging; query, notice, dependency, review and hash tampering; and stable identity behavior.

The existing already-approved O365 Notebook can be updated by adding only the computed canonical hash to its native review/provenance metadata. Preserve its IDs, query text, tabs/notices, historical source identity, and all 237 approved query-review entries; no rereview is required when the canonical payload is unchanged. No content worktree or real O365 package build was changed/run for this toolkit correction.

Verification

• Native workbook tests: 58 passed, including native-only and mixed native/migrated V3.1 packaging with the raw source removed.
• Existing migrated PowerShell source-passthrough regression: 1 passed (4 subtests).
• Both authored schema copies match; import-manifest.json SHA256 matches the imported schema.

No runtime qualification or deployment is claimed.

Required Marketplace attribution guard

This PR also fixes the V3.1 packaging path that could honor offline skip controls and successfully emit nested XDR packages without their customer usage attribution marker. The existing anonymous published Marketplace catalog/template resolver is reused unchanged; no new lookup implementation, plan-selection configuration, incomplete-package override, or content-package rebuild is introduced.

• For emitted XDR detections or Defender Workbooks, the existing WarnIfMissing signal now enforces required attribution. Missing/null/blank trackingId triggers automatic lookup even with -SkipAttributionLookup, SENTINEL_SKIP_ATTRIBUTION_LOOKUP=1, or both. Successful discovery saves the exact validated ID in authoritative SolutionMetadata.json and emits the unconditional root marker into JSON/ZIP.
• Valid explicit IDs still avoid attribution network requests. Legacy optional-attribution packages retain both explicit offline skips and warning-only remote-failure behavior.
• Genuine lookup failure or ambiguity terminates required packaging before the Package directory, template, parameters, UI, or ZIP are written. Metadata stays unchanged. Errors explain correcting publisher/offer identity and published plan/marker ambiguity, or supplying the complete correct Partner Center ID and rebuilding, rather than asking for an ID before attempting discovery.
• Fix a tightly coupled Windows persistence issue exposed by the mocked success regressions: use atomic File.Replace while retaining the writer-excluding snapshot handle; overwrite-move fails with that handle on Windows. Non-Windows persistence, byte preservation, concurrent-edit rejection and failure cleanup remain unchanged.
• Update directly related V3.1, pipeline and migration documentation. Native reviewed-payload/source-hash validation and migrated-source validation remain intact.

Additional verification

Python 3.12 venv and PowerShell 7, synthetic isolated solutions only:

• python -m unittest test_customer_usage_attribution test_defender_workbooks_packaging: 52 passed. Mocked catalog/template HTTP covers automatic metadata+marker population despite each bypass route, explicit-ID offline replay, local and positional pipeline entry points, terminating failures/ambiguity before artifact output, unchanged metadata on failure, optional offline behavior, byte-preserving persistence and write/conflict failures. Workbook-only required-attribution failure is also covered.
• python -m pytest Tools\SentinelToXDRMigration\tests\test_native_workbooks.py -q: 58 passed, including native-only and mixed native/migrated packaging and tampering/source-removal checks.
• Focused test_packaging.PackagingTests selection: 32 passed.
• 23 version-policy cases passed in the combined attribution/workbook/version-policy run; an initially incorrect new workbook-marker assertion in that run was corrected, and all attribution/workbook tests were rerun successfully above.
• git diff --check: passed.

The initial unfiltered attribution/packaging run encountered two unrelated existing test limitations, not waived production guards: test_runs_v3_1_and_returns_workflow_evidence treats the E5 substring in the checkout path C:\E5Workbooks as a forbidden command argument; test_cli_runs_real_packager_for_same_and_cross_worktree_roots completes its synthetic checks but its cleanup hits a Windows open logging-file handle (WinError 32). These tests were not edited; the remaining 32 packaging cases pass. No full content suite, content-worktree edits, real solution package rebuild, production network mutation, runtime qualification or deployment was performed for this fix.

Revision: standard built-in ASIM in native workbooks

Commit db86d16ddbc addresses the requested standard built-in underscore ASIM support, aligned with the existing workbook classifier and detection-converter behavior.

• Native planning, explicit review, import, validation and V3.1 packaging now accept recognized _Im_* / _ASim_* parsers, including bare _Im_Authentication, bare and called _ASim_ forms, and mixed parser/native-table queries. Query text is preserved.
• Parser names are excluded from table lineage. ASIM-only queries may have empty discovered table/workload/connector lists; these mean no statically inferred dependencies, not proof of no prerequisites. Hidden parser inputs, workload mappings and connector coverage are not invented. Explicit native-table inputs retain their workload records.
• Arbitrary underscore/custom/nonstandard functions, unknown/Sentinel tables, dynamic/external/cross-resource sources and non-AH providers remain blocked. An ASIM token in a comment, string or output column cannot unlock unproven empty lineage. Complete explicit review, query/source/canonical payload hashes and stale/fabricated dependency checks remain enforced.
• Update native-authoring documentation to distinguish accepted built-in parsers from unsupported saved-function prerequisites and clarify dependency/qualification limits.

Revision verification

Existing Python 3.12 venv and PowerShell 7, targeted tests and synthetic isolated solutions only:

• Native workbook suite: 87 passed, including CLI round trips, ASIM-only V3.1 packaging with the raw source removed, mixed explicit-table dependency retention, rejected custom/external sources, and stale/fabricated review checks.
• Closest existing workbook dependency/classifier, detection-converter ASIM and unchanged KQL gate regressions: 3 passed, 6 subtests passed (301 unrelated tests deselected).
• git diff --check: passed.

No content worktrees were modified, no real solutions were rebuilt, and no full validation, runtime qualification or deployment was performed. User-reported live ASIM observations motivate compatibility; these offline toolkit regressions do not establish target-tenant runtime availability or dependency coverage.

[oferInbar]

Overall the native XDR workbook import flow (native-workbook-plan / review / import / validate, canonicalPayloadSha256 verification after transient Workbook Sources/Native cleanup) and the fail-closed Marketplace attribution (trackingId / pid-<trackingId>-partnercenter) enforcement look great.

Requested Revision — Standard ASIM Functions (_ prefix)

In Tools/SentinelToXDRMigration/sentinel_xdr_migration/native_workbooks.py (around lines 84–87):

bad = sorted({*analysis["asimFunctions"], *analysis["externalDataSources"]} | (
    set(analysis["unknownFunctions"]) if analysis["unknownFunctions"] else set()
))
if bad:
    blockers.append(f"Unreviewable native functions/sources: {', '.join(bad)}")

We have verified in live testing that standard built-in Log Analytics ASIM functions starting with "_" (such as _ASim_* and _Im_*) work well in Defender XDR Advanced Hunting, Custom Detections, and Defender Workbooks.

Please revise native_workbooks.py so that standard "_"-prefixed Log Analytics / ASIM functions (_ASim_*, _Im_*, etc.) are allowed rather than rejected as unreviewable native functions (matching workbooks.py / converter.py handling of built-in "_" ASIM functions). Everything else in this PR is good to go!

[kingwil]

Pushed ASIM changes

[aketht-msft]

@kingwil - This is Aketh and I am a current Microsoft employee from 1P activity ingestion team.

Can you kindly share your MSFT alias, if you are a MSFT employee or reach out to me (my alias - aketht)?