Repository navigation
TrendAI Vision One (CCF): Add multi-tenant support, fix pagination, dedupe OAT records - #15300
Open
V1ManagedServices wants to merge 10 commits into
Open
V1ManagedServices wants to merge 10 commits into
V1ManagedServices wants to merge 10 commits into
Conversation
The Vision One v3 API returns the continuation URL in the response body as `nextLink`, not in an HTTP `Link` header. The connectors declared `pagingType: LinkHeader` with `linkHeaderTokenJsonPath: $.nextLink`, which never matched, so polling stopped after the first page. Measured impact on OAT: - API reports totalCount 433, first page returns 200 - continuation request returns an empty page - clients relying on nextLink retrieve at most 200 of 433 records Switch both pollers to `NextPageUrl`, matching the legacy pre-CCF poller which already used NextPageUrl for the same endpoints. Co-Authored-By: Claude <noreply@anthropic.com>
The OAT transform converted three detail fields to datetime using
unguarded epoch-millisecond arithmetic:
todatetime(datetime(1970-01-01) + tolong(d.<field>) * 1ms)
`rt` is not an epoch-millisecond timestamp. It is a Windows FILETIME
value (100ns intervals since 1601-01-01). Two real records confirm this:
uuid 01a0c961-1c46-7580-a96a-0cfb1769e38f
rt = 134345584181707458
as FILETIME -> 2026-09-22T13:46:58.170746Z
detectedDateTime -> 2026-09-22T13:46:58Z
firstSeen (ms) -> 2026-09-22T13:46:58.170Z
The old expression interpreted that value as ~4.26 million years,
overflowing the datetime range (max 9999-12-31T23:59:59.999 =
253402300799999 ms). The result was a null datetime, which the
ingestion pipeline rejects, dropping the entire record.
Changes:
- firstSeen / lastSeen keep epoch-ms conversion but are now bounded,
so unexpected magnitudes yield null instead of dropping the record.
- rt is stored verbatim as a string (detail_rt_g) with no conversion,
matching the legacy Python connector which kept rt untransformed
and matched what the Vision One UI displays.
- OAT table column detail_rt_t (datetime) replaced by
detail_rt_g (string).
Co-Authored-By: Claude <noreply@anthropic.com>
The connector previously supported a single Vision One tenant. The CCF
model separates the connector UI definition from individual connection
resources, so a single connector can now host multiple independent
connections.
Connector definitions (Workbench and OAT):
- Add a DataConnectorsGrid listing deployed connections, mapping the
Connection Alias, Vision One Business ID and API Domain columns from
properties.addOnAttributes, with a DeleteConnector row action.
- Move configuration inputs into a ContextPane "Add Connection" form
with friendlyName, apiDomain, customerId, apiToken and the existing
TMV1 filter fields. Document delete-and-re-add for token rotation,
since existing connections cannot be edited in place.
Poller configuration:
- Store friendlyName, apiDomain and customerId in addOnAttributes so
each connection is identifiable in the grid.
- Build the dataConnectors resource name from
uniqueString(friendlyName, apiDomain) so a new connection creates a
new resource instead of overwriting an existing one.
Ingestion:
- Add a customerId field to both DCR input streams and map it to
xdrCustomerID_g / xdrCustomerId_g, restoring the per-record tenant
identifier the legacy Python connector wrote from clp_id.
Packaged solution template mainTemplate.json updated accordingly.
Co-Authored-By: Claude <noreply@anthropic.com>
- Switch paging from NextPageUrl to LinkHeader with linkHeaderTokenJsonPath $.nextLink: NextPageUrl re-appends query parameters to the nextLink, which the Vision One API rejects with HTTP 400, dropping the entire window's batch. - Add queryWindowDelayInMin 10 on both pollers: OAT events are ingested in Vision One up to several minutes after detection time; querying by detection time without a delay missed late-arriving rows. - OAT: force the v2 search flow with search_v3=false so pagination returns full pages (the v3 flow returns an empty second page and caps at top). - OAT: top 200 (API page cap). Verified end-to-end on a fresh Sentinel workspace with two Vision One tenants: Workbench 8/8 per tenant, OAT 419/419 per tenant over the same UTC window, zero missing rows. Co-Authored-By: Claude <noreply@anthropic.com>
Regenerated the package with the official packaging tool and fixed the
issues that broke the Sentinel portal "Add Connection" flow.
Connector sources:
- Fix ARM expression escaping in both pollers: use the single trailing
bracket form ([[parameters('x')] / [[parameters('x')[0]]) instead of
the doubled form that produced an extra ']' and failed ARM parsing.
- Index array-typed dropdown parameters ([[parameters('apiDomain')[0]]).
- OAT: wire excludeThirdPartyOat through addOnAttributes so the DCR
transform can read it; replace the ARM expression previously embedded
in transformKql, which could never resolve because the DCR's
contentTemplate declares no parameters.
- Restore the excludeThirdPartyOat dropdown with its original
defaultValue of ["true"].
Packaging:
- Add package.sh to build the deployable zip locally.
- Rebuild mainTemplate.json via package-generator.ps1 (single source of
truth; both DCR copies now derive from the same files, so they cannot
drift).
- Keep 3.0.1.zip and add 3.0.2.zip.
Co-Authored-By: Claude <noreply@anthropic.com>
Add arg_max dedupe by detail_uuid_g to TrendAIOAT_Complete so repeated CCF poller ingests of the same detection collapse to one row. Regenerate the packaged mainTemplate and 3.0.2.zip to match. Co-Authored-By: Claude <noreply@anthropic.com>
The OAT table column detail_rt_t (datetime) was renamed to detail_rt_g (string) in 8a17862. Azure tables are immutable, so upgrades from 3.0.1 failed with 'Invalid output table schema' when adding a connection, because the live table still held the old column. Restore detail_rt_t (datetime) so the 3.0.1 schema remains a strict subset, and add detail_rt_s (string) carrying the raw FILETIME value that the DCR writes. Existing workspaces gain the new column on upgrade; fresh installs declare both. Regenerate the packaged mainTemplate and 3.0.2.zip. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
OAT deduplication is not tenant-scoped, and parser and release metadata are incomplete.
2 open findings
What changed in this PR
Adds multi-tenant TrendAI Vision One ingestion, improves OAT handling, and advances the solution to 3.0.2.
Changes:
- Adds tenant-aware connector configuration and metadata.
- Improves OAT timestamp handling and deduplication.
- Rebuilds deployment artifacts for version 3.0.2.
| File | Description |
|---|---|
Parsers/TrendAIOAT_Complete.yaml |
Adds OAT deduplication. |
Package/testParameters.json |
Reorders workbook parameters; excluded from review. |
Package/package.sh |
Adds packaging helper; excluded from review. |
Data/system_generated_metadata.json |
Adds generated 3.0.2 metadata. |
Data/Solution_TrendAIVisionOne.json |
Bumps solution to 3.0.2. |
...Workbench_PollerConfig.json |
Adds multi-tenant polling; excluded from review. |
...Workbench_DCR.json |
Adds customer attribution; excluded from review. |
...Workbench_ConnectorDefinition.json |
Adds multi-connection UI; excluded from review. |
...OAT_table.json |
Adds string-form OAT timestamp; excluded from review. |
...OAT_PollerConfig.json |
Updates OAT polling; excluded from review. |
...OAT_DCR.json |
Updates OAT transformations; excluded from review. |
...OAT_ConnectorDefinition.json |
Adds multi-connection UI; excluded from review. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
v-atulyadav
added
Solution
Contributor
|
Kindly check the Copilot comments. If the feedback has been addressed, please mark the corresponding comments as resolved. and attach a screenshot showing the CCF connector in a connected state for validation and review purposes. Thanks! |
detail_uuid_g alone collapsed identical UUIDs across different Vision
One tenants now that multiple tenants write into the same table.
Group by xdrCustomerId_g, detail_uuid_g instead, synced into both
mainTemplate.json copies. Also add the missing 3.0.2 ReleaseNotes.md
entry per releasenotes.instructions.md.
Contributor
Author
Hello v-maheshbh (@v-maheshbh) , The requested screenshots have been attached, and all Copilot-related issues have also been addressed. Kindly review and proceed with the necessary actions. Thank you. |
Contributor
Kindly remove the package.sh file from the Package folder as it is not required. Thanks! |
Contributor
|
Please check the above comments. Thanks! |
github-actions
Bot
added
the
ISV Solution
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan