Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) by flanakin · Pull Request #2126 · microsoft/finops-toolkit

flanakin · 2026-05-06T15:07:27Z

Summary

Renames IngestionSetup_v1_3.kql → IngestionSetup_v1_4.kql, consolidating all FOCUS 1.3 and 1.4 column additions into a single GA release
Adds ContractCommitment_raw table definition to IngestionSetup_RawTables.kql
Updates app.bicep and .build.config to reference v1_4 (removes v1_3)
Removes all "preview" designations — FOCUS 1.4 ships as GA alongside the FOCUS 1.4 specification

Test plan

KQL functions compile without errors
IngestionSetup_v1_4.kql contains transform functions, final tables, and update policies for all datasets: Prices, Costs, CommitmentDiscountUsage, ContractCommitment, Recommendations, Transactions
.build.config Fabric KQL includes IngestionSetup_v1_4.kql (not v1_3)

🤖 Generated with Claude Code

Adds IngestionSetup_v1_3.kql with Costs/Prices/CommitmentDiscountUsage/ Recommendations/Transactions transforms and final tables renamed for FOCUS 1.3. Cost and Usage gains 8 new FOCUS 1.3 columns: AllocatedMethodId/Details/ResourceId/ResourceName/Tags (data-generator split cost allocation), ContractApplied (per-row contract commitment application), ServiceProviderName + HostProviderName (replacing the deprecated ProviderName/PublisherName, with empty-fallback for back compat). Costs_raw now carries the new columns so downstream v1_2 transforms keep working and v1_3 transforms can read them directly. The v1_3 file is wired into the Bicep deployment alongside v1_0 and v1_2. The unversioned Costs() function still aliases to v1_2 — phase 2 adds Costs_v1_3() and retargets Latest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Consolidates FOCUS 1.3 and 1.4 column additions into a single v1_4 ingestion setup file. Removes the intermediate v1_3 file, adds ContractCommitment_raw table definition, and updates app.bicep and .build.config to reference v1_4. Co-Authored-By: Claude <noreply@anthropic.com>

flanakin

Changes here need to also be applied to the other 3 FOCUS 1.4 PRs

flanakin · 2026-05-20T21:26:00Z

@@ -359,6 +455,11 @@

 // Costs_raw table -- Redefine all columns
 .alter table Costs_raw (


FOCUS 1.4 has a requirement to include ALL native columns. This will significantly bloat the table size for cross-cloud data. Given we don't have a full set of columns for this yet, we need to have a way to ingest any additional columns either by auto-accepting new columns on ingestion or dumping them into a single x_ column. We need to research options and weigh pros/cons. This can happen in a separate PR, but don't resolve this without a link to that PR.

🤖 [AI][Claude Code] 🤔 Needs discussion

Filed a tracking issue to scope the research: #2159 — Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat. The issue captures the three candidate approaches (auto-accept new columns, single x_NativeColumns JSON, hybrid) and explicitly scopes implementation to a separate PR.

This PR ships the FOCUS 1.4 schema with the columns we know today; native-columns handling will follow #2159's resolution. Leaving this thread unresolved per your request until the follow-up PR is linked.

- Pluralize ContractCommitments_raw / _transform_v1_4 / _final_v1_4 / mapping / update policy - Add BillingPeriods_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add InvoiceDetails_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails + InvoiceDetailId to Costs_raw, Costs_final_v1_4, and Costs_transform_v1_4 - Fix x_Source* annotations on ContractCommitments_raw to Hubs v1_4+ - Update Costs_raw ProviderName/PublisherName annotations to FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) - Update Costs_raw Region annotation: deprecated -> removed - File #2159 for the FOCUS 1.4 native-columns architecture research 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

flanakin · 2026-05-26T07:22:37Z

🤖 [AI][Claude Code] PR Update Summary

Addressed: 7 thread(s)

✅ Implemented: 6
🤔 Needs discussion: 1 (Thread 7 — filed as issue Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159)

Key changes:

Pluralized ContractCommitment_* -> ContractCommitments_* across tables, functions, mapping, and update policies (Thread 1)
Updated x_Source* annotations Hubs v1_3+ -> v1_4+ on ContractCommitments_raw (Thread 2)
Updated ProviderName/PublisherName/Region annotations to mark them as removed (Thread 3)
Added BillingPeriods_raw and InvoiceDetails_raw datasets with transforms/finals/policies (Thread 4)
Added 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails to Costs_raw (Thread 5)
Added InvoiceDetailId to Costs_raw (Thread 6)
Filed Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159 for native-columns research (Thread 7)

Cascading next: Same pluralization + new BillingPeriods/InvoiceDetails datasets will be added to PRs #2128 (hub functions), #2135 (plugin docs), and #2136 (tests + changelog).

- Rename ContractCommitment_v1_4() -> ContractCommitments_v1_4() (and update folder label) - Add BillingPeriods_v1_4() and InvoiceDetails_v1_4() hub functions - Add BillingPeriods(), ContractCommitments() (renamed), and InvoiceDetails() unversioned aliases in HubSetup_Latest.kql - Add CommitmentProgramEligibilityDetails + 12 ContractCommitment* + InvoiceDetailId column defaults to Costs_final_v1_0 and Costs_final_v1_2 union arms (back-compat shims) and add them to the project list 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

- Rename ContractCommitment() -> ContractCommitments() in skill references and agent docs - Add BillingPeriods() and InvoiceDetails() to the function list and dedicated agent doc sections - Mention the new FOCUS 1.4 per-row columns (CommitmentProgramEligibilityDetails, 12 ContractCommitment* columns, InvoiceDetailId) - Update output-style guidance to reference all three supplemental datasets 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

Tests: - Pluralize ContractCommitments_raw / _final_v1_4 / _v1_4 assertions - Assert new BillingPeriods_raw + InvoiceDetails_raw datasets exist with their FOCUS 1.4 columns - Assert new BillingPeriods_v1_4(), InvoiceDetails_v1_4(), ContractCommitments_v1_4() hub functions - Assert HubSetup_Latest aliases for BillingPeriods(), ContractCommitments(), InvoiceDetails() - Assert Costs_raw and Costs_final_v1_4 include the 14 new FOCUS 1.4 columns - Assert NO singular ContractCommitment_raw / ContractCommitment_v1_4 / ContractCommitment() exists - Assert Costs_raw keeps removed ProviderName/PublisherName/Region for back compat Changelog: - Expand v15.0.0 entry with the 14 new FOCUS 1.4 columns and the three new supplemental datasets (ContractCommitments, BillingPeriods, InvoiceDetails) - Update plugin entry to list ContractCommitments() / BillingPeriods() / InvoiceDetails() 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

flanakin · 2026-05-27T06:41:39Z

+    ProviderName:               string,    // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)
+    PublisherName:              string,    // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)
+    Region:                     string,    // FOCUS 0.5-1.0-preview (removed; kept for back compat)


Suggested change

ProviderName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)

PublisherName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)

Region: string, // FOCUS 0.5-1.0-preview (removed; kept for back compat)

ProviderName: string, // FOCUS 0.5-1.3 (removed; keep for back-compat)

PublisherName: string, // FOCUS 0.5-1.3 (removed; keep for back-compat)

Region: string, // FOCUS 0.5-1.0-preview (removed; keep for back-compat)

flanakin · 2026-05-27T07:00:18Z

+        // FOCUS 1.3
+        isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity,
+        // FOCUS 1.0-preview, 1.0
+        isempty(CommitmentDiscountStatus), real(null),
+        CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)),
+        CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio,
+        real(null)
+    )
+    | extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case(
+        // FOCUS 1.3


Suggested change

// FOCUS 1.3

isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity,

// FOCUS 1.0-preview, 1.0

isempty(CommitmentDiscountStatus), real(null),

CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)),

CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio,

real(null)

)

| extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case(

// FOCUS 1.3

// FOCUS 1.2+

isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity,

// FOCUS 1.0-preview, 1.0

isempty(CommitmentDiscountStatus), real(null),

CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)),

CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio,

real(null)

)

| extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case(

// FOCUS 1.2+

flanakin · 2026-05-27T07:00:53Z

+    //
+    // Pricing
+    | extend old_x_AmortizationClass = x_AmortizationClass, x_AmortizationClass = case(
+        // FOCUS 1.3


Suggested change

// FOCUS 1.3

// FOCUS 1.2+

flanakin · 2026-05-27T07:05:05Z

+    )
+    | extend x_ConsumedCoreHours = iff(ConsumedUnit == 'Hours' and isnotempty(x_SkuCoreCount), x_SkuCoreCount * ConsumedQuantity, real(null))
+    | extend SkuPriceDetails = case(
+        // FOCUS 1.3


Suggested change

// FOCUS 1.3

// FOCUS 1.2+

flanakin · 2026-05-27T07:10:12Z

+```
+
+
+//===| BillingPeriods |=================================================================================================


BillingPeriods should be first in this file (alphabetized list)

flanakin · 2026-05-27T07:19:49Z

+        ContractApplied  = parse_json(ContractApplied),  // FOCUS 1.3+
+        ContractCommitmentBenefitCategory          = column_ifexists('ContractCommitmentBenefitCategory', ''),                  // FOCUS 1.4+
+        ContractCommitmentCreated                  = column_ifexists('ContractCommitmentCreated', datetime(null)),              // FOCUS 1.4+
+        ContractCommitmentDiscountPercentage       = column_ifexists('ContractCommitmentDiscountPercentage', real(null)),       // FOCUS 1.4+
+        ContractCommitmentDurationType             = column_ifexists('ContractCommitmentDurationType', ''),                     // FOCUS 1.4+
+        ContractCommitmentFulfillmentInterval      = column_ifexists('ContractCommitmentFulfillmentInterval', ''),              // FOCUS 1.4+
+        ContractCommitmentLastUpdated              = column_ifexists('ContractCommitmentLastUpdated', datetime(null)),          // FOCUS 1.4+
+        ContractCommitmentLifecycleStatus          = column_ifexists('ContractCommitmentLifecycleStatus', ''),                  // FOCUS 1.4+
+        ContractCommitmentModel                    = column_ifexists('ContractCommitmentModel', ''),                            // FOCUS 1.4+
+        ContractCommitmentOfferCategory            = column_ifexists('ContractCommitmentOfferCategory', ''),                    // FOCUS 1.4+
+        ContractCommitmentPaymentInterval          = column_ifexists('ContractCommitmentPaymentInterval', ''),                  // FOCUS 1.4+
+        ContractCommitmentPaymentModel             = column_ifexists('ContractCommitmentPaymentModel', ''),                     // FOCUS 1.4+
+        ContractCommitmentPaymentUpfrontPercentage = column_ifexists('ContractCommitmentPaymentUpfrontPercentage', real(null)), // FOCUS 1.4+


Port from older FOCUS exports

flanakin · 2026-05-27T07:20:09Z

+        ContractedUnitPrice = coalesce(ContractedUnitPrice, x_OnDemandUnitPrice),
+        EffectiveCost,
+        HostProviderName = iff(isempty(HostProviderName), PublisherName, HostProviderName),  // FOCUS 1.3+; falls back to deprecated PublisherName for back compat
+        InvoiceDetailId = column_ifexists('InvoiceDetailId', ''),  // FOCUS 1.4+


Port from older FOCUS exports

flanakin · 2026-05-27T07:26:44Z

+        PricingCurrency = coalesce(PricingCurrency, x_PricingCurrency),
+        PricingQuantity,
+        PricingUnit,
+        // ProviderName and PublisherName removed in FOCUS 1.4. Use ServiceProviderName / HostProviderName.


Suggested change

// ProviderName and PublisherName removed in FOCUS 1.4. Use ServiceProviderName / HostProviderName.

flanakin · 2026-05-27T07:27:26Z

+        ResourceType,
+        ServiceCategory,
+        ServiceName,
+        ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName),  // FOCUS 1.3+; falls back to deprecated ProviderName for back compat


Suggested change

ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated ProviderName for back compat

ServiceProviderName = iff(isempty(ServiceProviderName), PublisherName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated PublisherName for back compat

flanakin · 2026-05-27T07:32:23Z

+    PricingCurrency:            string,
+    PricingQuantity:            real,
+    PricingUnit:                string,
+    // ProviderName and PublisherName removed in FOCUS 1.4


Suggested change

// ProviderName and PublisherName removed in FOCUS 1.4

Copilot AI review requested due to automatic review settings May 6, 2026 15:07

flanakin requested review from MSBrett and RolandKrummenacher as code owners May 6, 2026 15:07

microsoft-github-policy-service Bot requested a review from helderpinto May 6, 2026 15:08

microsoft-github-policy-service Bot added the Needs: Review 👀 PR that is ready to be reviewed label May 6, 2026

microsoft-github-policy-service Bot assigned MSBrett May 6, 2026

microsoft-github-policy-service Bot added the Tool: FinOps hubs Data pipeline solution label May 6, 2026

microsoft-github-policy-service Bot assigned helderpinto and RolandKrummenacher May 6, 2026

Copilot started reviewing on behalf of flanakin May 6, 2026 15:08 View session

This was referenced May 6, 2026

Add FOCUS 1.4 hub functions (HubSetup_v1_4.kql) #2128

Draft

FOCUS 1.3 and 1.4 support across the toolkit #2120

Open

flanakin marked this pull request as draft May 6, 2026 23:16

flanakin added this to the v15 milestone May 15, 2026

flanakin changed the title ~~Add FOCUS 1.3 ingestion (IngestionSetup_v1_3.kql)~~ Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) May 17, 2026

This was referenced May 17, 2026

Add ContractCommitment dataset #2130

Closed

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql) #2132

Closed

flanakin commented May 26, 2026

View reviewed changes

flanakin mentioned this pull request May 26, 2026

Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159

Open

flanakin mentioned this pull request May 26, 2026

Refresh Claude Code plugin skill files for FOCUS 1.4 #2135

Draft

2 tasks

flanakin mentioned this pull request May 27, 2026

Add FOCUS 1.4 schema tests and changelog entries #2136

Draft

3 tasks

flanakin commented May 28, 2026

View reviewed changes

flanakin removed the Needs: Review 👀 PR that is ready to be reviewed label May 28, 2026

flanakin added the Needs: Attention 👋 Issue or PR needs to be reviewed by the author or it will be closed due to no activity label May 28, 2026

		@@ -359,6 +455,11 @@

		// Costs_raw table -- Redefine all columns
		.alter table Costs_raw (

		```


		//===\| BillingPeriods \|=================================================================================================

	ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated ProviderName for back compat
	ServiceProviderName = iff(isempty(ServiceProviderName), PublisherName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated PublisherName for back compat

Conversation

flanakin commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

flanakin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flanakin commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flanakin commented May 6, 2026 •

edited

Loading