[ASIM] Add Microsoft Defender for Office 365 EmailEvent parser by JamesAde11 · Pull Request #15354 · Azure/Azure-Sentinel

JamesAde11 · 2026-10-19T18:05:36Z

Change(s)

Add regular and filtering ASIM EmailEvent parsers for the native Microsoft Sentinel EmailEvents table, using EmailEvent schema version 1.0.0.

The contribution includes:

Field mappings for sender, recipient, timestamps, delivery results, threat information, email authentication results, and source IP addresses.
Filtering by time, sender, recipient, subject, IP prefix, direction, delivery location, and result.
Integration into the regular and filtering EmailEvent unifiers.
Generated Azure Resource Manager deployment templates and parser changelogs.
Synthetic sample data, focused static tests, a reproducible KQL regression query, and mapping documentation.

The Python fixture-generation and template-check helpers used during development are retained locally and are not included in this PR.

Each source record is normalized independently. The parsers do not join other event tables, expand records, deduplicate messages, or normalize post-delivery remediation events. Delivery success is kept separate from threat detection.

Reason for Change(s)

Provide source-specific ASIM EmailEvent coverage for Microsoft Defender for Office 365 EmailEvents, allowing queries to use normalized email fields instead of product-specific field names.

Version Updated

New source-specific parsers: 0.1.0.
Existing EmailEvent unifiers: 1.0.0 to 1.0.1.
EmailEvent schema version remains 1.0.0.

Testing Completed

Local static validation completed:

Nine focused Kusto.Language syntax, binding, output-schema, and cross-platform source-fingerprint checks passed, most recently on October 19, 2026.
Thirty existing repository-template checks passed, 15 for each parser variant, most recently on October 19, 2026.
Six generated ARM JSON files parsed successfully.
Four generated ARM queries matched their corresponding parser YAML.

Runtime validation remains pending:

Fourteen synthetic mapping cases and 26 control assertions are prepared in a self-contained KQL regression query. The query has passed static binding checks but has not been executed by a KQL engine.
Official ASIM schema and data tester execution remains pending.
Validation against current representative source records and performance evaluation remain pending.

All included fixture records are synthetic. They do not establish production correctness. No deployment, sample ingestion, or validation against customer email records has been performed.

Checked that the validations are passing and have addressed any issues that are present

The targeted local checks listed above passed, including a rerun after removing the Python helpers. GitHub checks must be evaluated on the latest commit and are separate from those local results. Hosted ASIM tests require maintainer SafeToRun approval for each code update; a new push can invalidate the previous approval. Full runtime validation is incomplete.

This draft requests review of the mappings, delivery-result semantics, filtering behavior, source compatibility, and remaining validation requirements before merge.

Known Limitations

Unavailable source fields are not invented. Attachment and URL counts are mapped, but detailed attachment and URL records are outside scope.

Deployment links reference the upstream master branch and will not resolve to the new templates until those files are merged.

This contribution does not itself deploy parsers or change the installed _Im_EmailEvent built-in function.

Add regular and filtering parsers, unifier integration, synthetic fixtures, local static tests, mapping notes, changelogs, and generated deployment templates. Runtime and representative-data validation remain pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4a87d261-1f21-44f1-818d-2dce48c48b95

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4a87d261-1f21-44f1-818d-2dce48c48b95

Hash LF-normalized UTF-8 source text in the generator and C# tests. Exercise LF and CRLF inputs so Windows-generated fixtures bind on Linux CI without weakening stale-source detection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4a87d261-1f21-44f1-818d-2dce48c48b95

+
+        private static Dictionary<string, object> Parser(string prefix)
+        {
+            var path = Path.Combine(Root, "Parsers", "ASimEmailEvent", "Parsers", prefix + Suffix + ".yaml");


+
+        private static GlobalState Globals(bool optionalColumns = true)
+        {
+            var path = Path.Combine(Root, ".script", "tests", "KqlvalidationsTests", "CustomTables", "EmailEvents.json");


+            var code = Analyze(Declaration(parser) + parser["ParserName"] + "()");
+            var output = Assert.IsType<TableSymbol>(code.ResultType).Columns.ToDictionary(c => c.Name, c => c.Type.Name);
+            var specification = new Dictionary<string, (string Type, string Class)>();
+            using (var csv = new TextFieldParser(Path.Combine(Root, "ASIM", "dev", "ASimTester", "ASimTester.csv")))


+            foreach (var prefix in new[] { "ASim", "vim" })
+            {
+                var parser = Parser(prefix);
+                var code = Analyze(Declaration(parser) + parser["ParserName"] + "()");
+                schemas.Add(Assert.IsType<TableSymbol>(code.ResultType).Columns.Select(c => c.Name + ":" + c.Type.Name).ToArray());
+                var sourceReads = code.Syntax.GetDescendants<NameReference>()
+                    .Where(n => n.ReferencedSymbol is TableSymbol table && table.Name == "EmailEvents").ToList();
+                Assert.Single(sourceReads);
+                var prohibited = new[] { "MvExpandOperator", "MvApplyOperator", "SummarizeOperator", "DistinctOperator", "JoinOperator" };
+                Assert.DoesNotContain(code.Syntax.GetDescendants<QueryOperator>(), op => prohibited.Contains(op.GetType().Name));
+            }


+        [InlineData("\r\n")]
+        public void SyntheticRegressionQueryBindsWithoutAnyExternalTables(string lineEnding)
+        {
+            var path = Path.Combine(Root, "Parsers", "ASimEmailEvent", "Tests", "EmailEventsRegression.kql");


+            var query = File.ReadAllText(path);
+            foreach (var name in new[] { "ASim" + Suffix, "vim" + Suffix, "vimEmailEventEmpty", "ASimEmailEvent", "imEmailEvent" })
+            {
+                var source = Path.Combine(Root, "Parsers", "ASimEmailEvent", "Parsers", name + ".yaml");


Retain development helpers outside the repository and remove their README commands. Parser logic and the existing KQL regression assets are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3079d9d2-7890-4fc1-9495-30193ae5209b

JamesAde11 · 2026-10-20T23:15:37Z

Superseded by #15373. Closing this earlier draft in favor of the replacement contribution.

JamesAde11 and others added 2 commits October 19, 2026 11:04

ASIM: Link EmailEvents changelogs to PR 15354

c31c497

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4a87d261-1f21-44f1-818d-2dce48c48b95

JamesAde11 changed the title ~~ASIM: Add Microsoft Defender for Office 365 EmailEvents parsers~~ [ASIM] Add Microsoft Defender for Office 365 EmailEvent parser Oct 19, 2026

JamesAde11 requested review from Ofer Shezaf (oshezaf) and Derrick Lee (yummyblabla) October 19, 2026 18:22

github-advanced-security Bot found potential problems Oct 19, 2026

View reviewed changes

Derrick Lee (yummyblabla) added ASIM SafeToRun This is used only for ASim parsers Fork PR Pipeline run. labels Oct 19, 2026

JamesAde11 mentioned this pull request Oct 20, 2026

[ASIM] Add Microsoft Defender for Office 365 EmailEvent normalization parser #15373

Open

JamesAde11 closed this Oct 20, 2026

v-atulyadav self-assigned this Oct 21, 2026

Navigation Menu

Repository navigation

[ASIM] Add Microsoft Defender for Office 365 EmailEvent parser - #15354

[ASIM] Add Microsoft Defender for Office 365 EmailEvent parser#15354
JamesAde11 wants to merge 4 commits into
Azure:masterfrom
JamesAde11:asim/emailevent-microsoft-defenderoffice365

JamesAde11 commented Oct 19, 2026 •

edited

Loading

Uh oh!

JamesAde11 commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

JamesAde11 commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change(s)

Reason for Change(s)

Version Updated

Testing Completed

Checked that the validations are passing and have addressed any issues that are present

Known Limitations

Uh oh!

JamesAde11 commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JamesAde11 commented Oct 19, 2026 •

edited

Loading