ASIM: Add Proofpoint POD MailLog EmailEvent parsers by adbhogeGit · Pull Request #15410 · Azure/Azure-Sentinel

adbhogeGit · 2026-10-23T17:04:33Z

Summary

Adds EmailEvent 1.0.0 ASIM and parameterized vim parsers for ProofpointPODMailLog_CL, unifier registrations, sample fixtures, documentation, and the current CCP table schema in the offline CI catalog. ARM generation is handled by the GitHub Action; no manually generated ARM changes are maintained.

Task: https://dev.azure.com/msdata/Sentinel%20Graph/_workitems/edit/5510370

CI fixes

Commit af9ac0a registers the exact eight-column CCP schema, resolving the offline validator's KS204 unknown-table error. Both parser variants now read the optional source id through column_ifexists because the hosted ASIM ingestion helper strips that reserved column. Missing IDs remain unset; no IDs are fabricated.

Both targeted repository KQL tests pass (2/2), alongside 16 offline compiler cases, 30 template checks, and catalog/fixture parity checks. Hosted ASIM schema/data/filter checks for the updated head are pending; previous SEM0100 failures were from the older head.

Limitations

Live validation against the user workspace remains deferred by user direction: it lacks ProofpointPODMailLog_CL. Existing hosted CI workflows may validate their own ingested fixtures. Fixtures are transformed repository samples, not workspace exports. Envelope addresses are not treated as header From/To; missing header, subject, and location fields remain unmapped. Transport handoff is not inbox placement. IPv4 prefix filtering only. Task acceptance details/vendor documentation remain unverified where access was unavailable. No user-workspace deployment or ingestion performed.

Ready for review; live-validation and mapping limitations remain explicit. Changelog PR links are included.

Add MailLog source and filtering parsers, unifier registrations, generated ARM, fixtures and mapping documentation. Live validation is deferred; repository KQL checks need the protected custom-table catalog updated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 321517f1-a1c5-4242-be85-2d144a9cc295

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 321517f1-a1c5-4242-be85-2d144a9cc295

Leave ARM generation to GitHub Actions; preserve parser YAML, registrations, fixtures and changelogs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 321517f1-a1c5-4242-be85-2d144a9cc295

…roofpoint-mail-log-parser

…ML files.

contentautomationbot · 2026-10-23T18:00:55Z

ASIM parsers have been changed. ARM templates were regenerated from the updated KQL function YAML files.
To find the new ARM templates, pull your branch.

Match the connector schema in CI KQL catalog. ASIM sample ingestion excludes id; preserve actual connector IDs when present and leave optional event IDs empty when absent, without changing validation gates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 321517f1-a1c5-4242-be85-2d144a9cc295

…ML files.

Copilot

🔵 Needs a closer look

Hosted filtering validation lacks EmailEvent support, and pending validation leaves the new filtering behavior insufficiently verified for automated approval.

2 open issues

What changed in this PR

Adds Proofpoint POD MailLog support to ASIM EmailEvent 1.0.0, enabling source-independent queries over SMTP receipt and delivery records.

Changes:

Adds source and filtering parsers, with unifier registration and optional source-ID handling.
Adds generated ARM deployment assets, sample fixtures and the offline table schema.
Documents mappings, source limitations, deferred live validation and version history.

File	Description
Sample Data/ASIM/ProofpointPODMailLog_CL_Schema.csv	Defines the fixture table schema.
Sample Data/ASIM/Proofpoint_POD MailLog_EmailEvent_IngestedLogs.csv	Adds 20 transformed sample records.
Parsers/ASimEmailEvent/README.md	Documents mappings, filtering and validation limitations.
Parsers/ASimEmailEvent/Parsers/vimEmailEventProofpointPODMailLog.yaml	Adds the parameterized source parser.
Parsers/ASimEmailEvent/Parsers/imEmailEvent.yaml	Registers the filtering parser and forwards parameters.
Parsers/ASimEmailEvent/Parsers/ASimEmailEventProofpointPODMailLog.yaml	Adds source-event normalization.
Parsers/ASimEmailEvent/Parsers/ASimEmailEvent.yaml	Registers the source parser.
Parsers/ASimEmailEvent/CHANGELOG/vimEmailEventProofpointPODMailLog.md	Records filtering-parser releases.
Parsers/ASimEmailEvent/CHANGELOG/imEmailEvent.md	Records filtering-unifier registration.
Parsers/ASimEmailEvent/CHANGELOG/ASimEmailEventProofpointPODMailLog.md	Records source-parser releases.
Parsers/ASimEmailEvent/CHANGELOG/ASimEmailEvent.md	Records source-unifier registration.
Parsers/ASimEmailEvent/ARM/vimEmailEventProofpointPODMailLog/vimEmailEventProofpointPODMailLog.json	Adds filtering-parser deployment template.
Parsers/ASimEmailEvent/ARM/vimEmailEventProofpointPODMailLog/README.md	Adds filtering-parser deployment links.
Parsers/ASimEmailEvent/ARM/imEmailEvent/imEmailEvent.json	Updates deployed filtering-unifier query.
Parsers/ASimEmailEvent/ARM/FullDeploymentEmailEvent.json	Includes both new parser deployments.
Parsers/ASimEmailEvent/ARM/ASimEmailEventProofpointPODMailLog/README.md	Adds source-parser deployment links.
Parsers/ASimEmailEvent/ARM/ASimEmailEventProofpointPODMailLog/ASimEmailEventProofpointPODMailLog.json	Adds source-parser deployment template.
Parsers/ASimEmailEvent/ARM/ASimEmailEvent/ASimEmailEvent.json	Updates deployed source-unifier query.
.script/tests/KqlvalidationsTests/CustomTables/ProofpointPODMailLog_CL.json	Registers the eight-column offline KQL schema.

_{Review effort: Balanced}

adbhogeGit · 2026-10-23T21:48:50Z

+Product:
+  Name: Proofpoint POD MailLog
+Normalization:
+  Schema: EmailEvent


https://github.com/Azure/Azure-Sentinel/pull/15412/changes#diff-c179e7dd168d04e52f01168b1ca17fdc4137701ed517be1678b30d66f8cae6c4

+    | extend
+        EmailMailFromAddress = iff(EmailMailFromAddress contains "@", EmailMailFromAddress, ""),
+        RelayHost = trim_end(@"\.", coalesce(RelayHost, Relay)),
+        RelayIp = iff(isnotnull(parse_ipv4(RelayIp)) or isnotempty(parse_ipv6(RelayIp)), RelayIp, ""),
+        EmailProtocolName = case(tostring(sm.proto) =~ "SMTP" or tostring(sm.mailer) in~ ("smtp", "esmtp"), "SMTP", ""),
+        EventSubType = iff(IsDelivery, "EmailSent", "EmailReceived")
+    | extend
+        SrcIpAddr = iff(IsDelivery, "", RelayIp),
+        DstIpAddr = iff(IsDelivery, RelayIp, ""),
+        Src = iff(IsDelivery, Dvc, coalesce(RelayIp, RelayHost)),
+        Dst = iff(IsDelivery, coalesce(RelayIp, RelayHost), Dvc),
+        SrcFQDN = iff(not(IsDelivery) and RelayHost contains ".", RelayHost, ""),
+        SrcHostname = iff(IsDelivery, tostring(split(Dvc, ".")[0]), tostring(split(RelayHost, ".")[0])),
+        SrcDomain = iff(IsDelivery, substring(Dvc, indexof(Dvc, ".") + 1), substring(RelayHost, indexof(RelayHost, ".") + 1)),
+        DvcHostname = tostring(split(Dvc, ".")[0]),
+        DvcDomain = iff(Dvc contains ".", substring(Dvc, indexof(Dvc, ".") + 1), ""),
+        DvcFQDN = iff(Dvc contains ".", Dvc, ""),
+        EmailMailFromDomain = iff(EmailMailFromAddress contains "@", tostring(split(EmailMailFromAddress, "@")[1]), ""),
+        DvcAction = iff(IsDelivery and EventResult == "Success" and DeliveryStatus startswith "Sent", "Delivered", "NA")
+    | where array_length(ipaddr_has_any_prefix) == 0
+        or has_any_ipv4_prefix(SrcIpAddr, ipaddr_has_any_prefix)
+        or has_any_ipv4_prefix(DstIpAddr, ipaddr_has_any_prefix)


Derrick Lee (yummyblabla) · 2026-10-23T21:57:33Z

+  let parser = (disabled:bool=false, pack:bool=false) {
+    ProofpointPODMailLog_CL
+    | where not(disabled)
+    | where isnotempty(tostring(sm.qid)) and (bag_has_key(sm, "from") or array_length(sm["to"]) > 0)


This is a weird filter statement. What is it supposed to filter out?

Aditi Bhoge and others added 5 commits October 21, 2026 23:33

Link MailLog parser changelogs to draft PR 15410

ec3fe2f

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 321517f1-a1c5-4242-be85-2d144a9cc295

Remove generated ARM artifacts from MailLog parser PR

998a1bf

Leave ARM generation to GitHub Actions; preserve parser YAML, registrations, fixtures and changelogs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 321517f1-a1c5-4242-be85-2d144a9cc295

Merge remote-tracking branch 'origin/master' into adbhoge-microsoft-p…

7eddde4

…roofpoint-mail-log-parser

[ASIM Parsers] Generate deployable ARM templates from KQL function YA…

2054ea9

…ML files.

adbhogeGit marked this pull request as ready for review October 23, 2026 18:01

adbhogeGit requested review from a team as code owners October 23, 2026 18:01

Aditi Bhoge and others added 2 commits October 23, 2026 11:21

[ASIM Parsers] Generate deployable ARM templates from KQL function YA…

71ae0b6

…ML files.

adbhogeGit added the ASIM label Oct 23, 2026

adbhogeGit requested a balanced review from Copilot October 23, 2026 18:58

Copilot started reviewing on behalf of adbhogeGit October 23, 2026 18:59

View session

Copilot AI reviewed Oct 23, 2026

View reviewed changes

adbhogeGit mentioned this pull request Oct 23, 2026

Add EmailEvent filtering validator support #15412

Open

Derrick Lee (yummyblabla) reviewed Oct 23, 2026

View reviewed changes

Navigation Menu

Repository navigation

ASIM: Add Proofpoint POD MailLog EmailEvent parsers - #15410

ASIM: Add Proofpoint POD MailLog EmailEvent parsers#15410
adbhogeGit wants to merge 7 commits into
masterfrom
adbhoge-microsoft-proofpoint-mail-log-parser

adbhogeGit commented Oct 23, 2026 •

edited

Loading

Uh oh!

contentautomationbot Bot commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

adbhogeGit Oct 23, 2026

Uh oh!

Derrick Lee (yummyblabla) Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

adbhogeGit commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

CI fixes

Limitations

Uh oh!

contentautomationbot Bot commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

adbhogeGit Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Derrick Lee (yummyblabla) Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adbhogeGit commented Oct 23, 2026 •

edited

Loading