Skip to content

ASIM: Add Proofpoint POD Message EmailEvent parsers - #15409

Open
adbhogeGit wants to merge 11 commits into
masterfrom
adbhoge-microsoft-proofpoint-message-parser
Open

adbhogeGit wants to merge 11 commits into
masterfrom
adbhoge-microsoft-proofpoint-message-parser

Conversation

@adbhogeGit

@adbhogeGit adbhogeGit commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Required items, please complete

Change(s):

  • Register the exact ProofpointPODMessage_CL schema from the CCP connector definition in the repository KQL-validation table catalog.
  • Add focused offline assertions comparing the catalog fixture with the connector schema and preserving the source-row and recipient-array contracts.
  • Document hosted ASIM validator metadata/support limitations. The repository GitHub action generated and committed the parser ARM outputs; no ARM templates were manually edited.

Reason for Change(s):

  • Implements Azure DevOps task 5510371.
  • Fixes KQL CI KS204 caused by the missing custom-table definition. The remaining hosted ASIM data/filter failures are validator catalog/support gaps, not an evidence-backed parser mapping defect. No validation was weakened.

Version Updated:

  • N/A — no Detection or Analytic Rule templates changed.

Testing Completed:

  • PASS: 370 offline parser contract, KQL semantic/type, fixture, and registration checks. Command:
    .\Parsers\ASimEmailEvent\Tests\ProofpointPODMessage.Tests.ps1
  • PASS: 84 repository KQL validation tests. Command:
    dotnet test .script\tests\KqlvalidationsTests\Kqlvalidations.Tests.csproj --filter "DisplayName~Proofpoint" --no-build --logger "console;verbosity=minimal"
  • PASS on current head 89a900d: KQL Validations, ASIM template/ingest-time validations, ARM generation, CodeQL, syntax, solution, connector, detection, and other checks.
  • No user-workspace queries, deployment, or ingestion were performed. Live validation remains deferred; the supplied workspace did not contain ProofpointPODMessage_CL. The repository's standard shared-canary CI ingestion completed.

Checked that the validations are passing and have addressed any issues that are present:

  • Current KQL validation passes on 89a900d; this resolves the earlier KS204 missing-table-schema failure. The catalog fixture mirrors the connector's exact current CCP table schema.
  • Current shared-canary run 40008308726 fails in ASIM data and filtering checks. The data tester reports EmailToRecipients arrays with “All inner values must be of type []”: ASIM/dev/ASimTester/ASimTester.csv declares DynamicType=array but leaves ArrayValuesType blank, although source recipients are string arrays. The ASIM tester catalog was not changed.
  • The same run's filtering test reports Schema: EmailEvent - Not an existing schema or not supported by the validations script, before testing parser parameters. ASimFilteringTest.py's supported-schema catalog omits EmailEvent. Shared EmailEvent validator support is being implemented in a separate PR; this PR does not modify that code.
  • All other checks for current head 89a900d69164ca3cc5c646cd6711b2a078027e8e have passed. The PR remains open and ready for review, not merged. Live user-workspace validation remains deferred.

Aditi Bhoge and others added 5 commits October 21, 2026 23:31
Add parameterless and filtering parsers, unifier registrations, fixtures, focused offline validation, and generated ARM assets. Live validation is deferred by user direction.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Remove generated ARM changes and retain source-only offline checks and documentation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
@contentautomationbot

Copy link
Copy Markdown

ASIM parsers have been changed. ARM templates were regenerated from the updated KQL function YAML files.
To find the new ARM templates, pull your branch.

@adbhogeGit
adbhogeGit marked this pull request as ready for review October 23, 2026 18:01
@adbhogeGit
adbhogeGit requested review from a team as code owners October 23, 2026 18:01
@adbhogeGit
adbhogeGit requested a balanced review from Copilot October 23, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The reported KQL catalog failures, deferred live validation, and unverified source mappings require maintainer review before merging.

No open issues 🎉

What changed in this PR

Adds Proofpoint POD message support to ASIM EmailEvent 1.0.0, making ProofpointPODMessage_CL available through the standard EmailEvent unifiers.

Changes:

  • Adds parameterless and filtering parsers that preserve one event per source message, with optional source details.
  • Registers both parsers in the unifiers and includes generated ARM deployment artifacts.
  • Adds synthetic fixtures, offline validation checks, changelogs, and documentation of mapping limitations and deferred live validation.
File Description
Sample Data/​ASIM/​ProofpointPODMessage_CL_Schema.csv Defines the source fixture schema.
Sample Data/​ASIM/​Proofpoint_POD_EmailEvent_IngestedLogs.csv Adds four synthetic message scenarios.
Parsers/​ASimEmailEvent/​Tests/​ProofpointPODMessage.Tests.ps1 Checks contracts, KQL types, filter bindings, and registrations offline.
Parsers/​ASimEmailEvent/​README.md Documents mappings, filters, limitations, and validation status.
Parsers/​ASimEmailEvent/​Parsers/​vimEmailEventProofpointPODMessage.yaml Adds the filtering Proofpoint parser.
Parsers/​ASimEmailEvent/​Parsers/​imEmailEvent.yaml Registers the filtering parser and forwards parameters.
Parsers/​ASimEmailEvent/​Parsers/​ASimEmailEventProofpointPODMessage.yaml Adds source-to-EmailEvent normalization.
Parsers/​ASimEmailEvent/​Parsers/​ASimEmailEvent.yaml Registers the parameterless parser.
Parsers/​ASimEmailEvent/​CHANGELOG/​vimEmailEventProofpointPODMessage.md Records the filtering parser’s initial release.
Parsers/​ASimEmailEvent/​CHANGELOG/​imEmailEvent.md Records the filtering unifier update.
Parsers/​ASimEmailEvent/​CHANGELOG/​ASimEmailEventProofpointPODMessage.md Records the parameterless parser’s initial release.
Parsers/​ASimEmailEvent/​CHANGELOG/​ASimEmailEvent.md Records the parameterless unifier update.
Parsers/​ASimEmailEvent/​ARM/​vimEmailEventProofpointPODMessage/​vimEmailEventProofpointPODMessage.json Adds the filtering parser deployment template.
Parsers/​ASimEmailEvent/​ARM/​vimEmailEventProofpointPODMessage/​README.md Provides filtering parser deployment links.
Parsers/​ASimEmailEvent/​ARM/​imEmailEvent/​imEmailEvent.json Updates the deployed filtering unifier.
Parsers/​ASimEmailEvent/​ARM/​FullDeploymentEmailEvent.json Includes both source parsers in full deployment.
Parsers/​ASimEmailEvent/​ARM/​ASimEmailEventProofpointPODMessage/​README.md Provides parameterless parser deployment links.
Parsers/​ASimEmailEvent/​ARM/​ASimEmailEventProofpointPODMessage/​ASimEmailEventProofpointPODMessage.json Adds the parameterless parser deployment template.
Parsers/​ASimEmailEvent/​ARM/​ASimEmailEvent/​ASimEmailEvent.json Updates the deployed parameterless unifier.

Review effort: Balanced

Aditi Bhoge and others added 4 commits October 23, 2026 11:30
Register the exact CCP custom-table schema and normalize JSON-encoded dynamic columns so CI fixture ingestion produces typed EmailEvent arrays.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Document that checked-in parser ARM templates are generated and committed by the repository GitHub action, rather than manually edited.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86
Keep the parser mappings native to their dynamic connector columns; the hosted recipient-array error comes from missing ArrayValuesType metadata, and the filter validator lacks EmailEvent support.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 723d3767-4cf2-432d-8609-3cff81330f86

@yummyblabla Derrick Lee (yummyblabla) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't

@@ -0,0 +1,216 @@
$ErrorActionPreference = 'Stop'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this file

- [Normalization and the Advanced Security Information Model (ASIM)](https://aka.ms/AboutASIM)
- [Deploy all of ASIM](https://aka.ms/DeployASIM)
- [ASIM Email Event normalization schema reference](https://aka.ms/ASimEmailEventDoc) No newline at end of file
- [ASIM Email Event normalization schema reference](https://aka.ms/ASimEmailEventDoc)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes not needed in this README

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants