[FR] Add non-FOCUS columns to FOCUS datasets as a non-functional requirement

[Matt-Cowsert]

🧠 Problem Statement

FOCUS standardizes cost and usage data with a consistent set of required columns. However, the diversity of provider offerings and the pace of change in services mean that some data necessary for accurate analysis may not map neatly into existing FOCUS columns. Without a clear expectation for providers to include supplemental columns, practitioners risk losing critical context that is available in non-FOCUS datasets.

---

🧪 Use Case

As a FinOps Practitioner migrating from native to FOCUS datasets
I need all information from my native billing dataset available in FOCUS (either as standard or custom columns)
So that I can adopt FOCUS without losing critical billing context needed for accurate analysis and decision-making

As a FinOps Practitioner analyzing provider-specific features
I need custom columns for provider-unique attributes (e.g., capacity type, marketplace metadata, service-specific configurations)
So that I can perform provider-native optimizations and cost attribution that FOCUS standard columns don't yet support

As a FinOps Practitioner validating FOCUS data completeness
I need explicit confirmation that all native dataset columns are mapped to FOCUS standard or custom columns
So that I can trust FOCUS data has no information loss and confidently retire native dataset queries

As a FOCUS Data Generator implementing FOCUS alongside native exports
I need clear requirements for when to include custom columns
So that I can ensure completeness without duplicating data already standardized in FOCUS columns

As a FinOps Tooling Vendor building FOCUS-compatible analytics
I need to know which custom columns exist across providers
So that I can support provider-specific features while maintaining FOCUS-based core functionality

As a FinOps Practitioner correlating FOCUS and proprietary datasets
I need custom columns that enable reliable joining between FOCUS and native datasets (e.g., x_ChargeId or documented correlation keys)
So that I can cross-reference FOCUS data with proprietary details without maintaining duplicate large datasets or building fragile correlation logic

Acceptance Criteria

Completeness Requirements:

• FOCUS datasets include custom columns (x_ prefix) for all information present in native datasets but not covered by FOCUS standard columns
• No native dataset information is lost when practitioners adopt FOCUS datasets
• Custom columns maintain same granularity and accuracy as native dataset equivalents
• Practitioners can achieve parity between FOCUS and native dataset analyses using combination of standard and custom columns

Data Integrity & Consistency:

• Custom columns do NOT duplicate data already captured in FOCUS standard columns
• When native data transforms into FOCUS standard columns, custom columns are NOT added for the native representation
• FOCUS-defined metrics and dimensions (particularly summable values like costs and quantities) maintain integrity when custom columns are added
• Row-level aggregation and splitting to align with FOCUS requirements preserves accuracy across all custom columns
• Providers include custom columns that enable reliable correlation between FOCUS and proprietary datasets:
  • Providers with existing unique charge identifiers (e.g., ChargeId) include them as custom columns (e.g., x_ChargeId)
  • Providers without existing identifiers document correlation guidance and include minimal custom columns required for dataset joins
  • Correlation columns serve as linking mechanisms; uniqueness within FOCUS datasets is not required

Naming & Documentation:

• Custom columns follow x_ prefix naming convention consistently
• Custom column naming avoids conflicts with current or planned FOCUS standard columns
• Data generators provide documentation describing custom columns, their purpose, and relationship to native dataset columns
• Documentation clarifies which native columns map to FOCUS standard vs. custom columns (integration with FR [FR] Add provider column to FOCUS column mappings as a non-functional requirement #1098)

Practitioner Impact:

• Practitioners can migrate from native to FOCUS datasets without analytical capability loss
• Provider-specific optimization opportunities remain accessible via custom columns
• Practitioners understand which capabilities require custom columns vs. FOCUS standard columns
• Cross-provider analysis uses FOCUS standard columns; provider-specific analysis supplements with custom columns

---

Enhancement to Existing Supported Feature: Custom Columns

Current Feature Description:

FOCUS supports the inclusion of custom columns to facilitate reporting capability that is not covered by the columns included in the specification.

Draft Enhancement - Revised Description:

FOCUS supports the inclusion of custom columns to facilitate reporting capability that is not covered by the columns included in the specification. When providers offer FOCUS datasets alongside native billing datasets, custom columns ensure completeness by capturing all native dataset information not represented in FOCUS standard columns. This completeness requirement ensures practitioners can adopt FOCUS without losing critical billing context, while maintaining the integrity of FOCUS-defined metrics and dimensions.

Completeness Principle (1.4): Data generators publishing both FOCUS and native datasets MUST include custom columns (x_ prefix) for all information available in native datasets but not covered by FOCUS standard columns. This ensures no data loss and enables practitioners to confidently migrate from native to FOCUS-based workflows while retaining access to provider-specific features and attributes.

Draft Enhancement - Add to Description (Guidance):

When to Include Custom Columns:

• Native dataset contains information with no FOCUS standard column equivalent
• Provider-specific features or attributes require additional context beyond FOCUS standard columns
• Marketplace metadata or publisher information not captured in FOCUS columns
• Service-specific configuration or capacity details needed for optimization

When NOT to Include Custom Columns:

• Information is already captured in FOCUS standard column (avoid duplication)
• Native column is transformed/aggregated into FOCUS standard column (document in mapping, don't duplicate)
• Data would violate FOCUS metrics integrity (e.g., would break cost summation)

Draft Enhancement - Add Supporting Context:

Relationship to Native Datasets:

• Custom columns bridge the gap between FOCUS standardization and native dataset completeness
• Practitioners can perform cross-provider analysis with FOCUS standard columns, supplemented by provider-specific analysis using custom columns
• Custom columns enable gradual FOCUS adoption (providers can add features to standard columns over time, reducing custom column dependency)

Integration with FR #1098 (Mapping Documentation):

• Provider mapping documentation should identify which native columns become custom columns
• Conformance validation checks that all native columns are accounted for (mapped to FOCUS standard OR included as custom)

No changes to Directly Dependent Columns (already has x_CustomColumn)

No changes to Example SQL Query (already demonstrates custom column usage)

Introduced (Version)

0.5 (baseline), 1.4 (completeness requirement)

---

🎯 Desired Outcome / Practitioner Impact

Practitioners receive a FOCUS dataset that not only adheres to standardized columns but also includes supplemental provider-defined columns when needed. This ensures completeness, continuity with non-FOCUS datasets, and reduces the risk of data loss or misinterpretation.

---

📨 Type of Request

How would you categorize this request? Select the option that best describes the novelty or source of the idea.

• Refinement of an existing FOCUS attribute
• Addition of a provider-supported field not yet in FOCUS
• Net-new concept (not currently supported by providers or FOCUS)
• Supporting Content (e.g., appendices, use cases)

---

🏛️ Organizations Requesting or Supporting This Feature

• Neos
• Twilio
• Microsoft
• FinOps Guys
• Electronic Arts

---

🌐 FinOps Scope Alignment (Optional)

• Public Cloud – e.g., AWS, Azure, GCP, OCI
• Software-as-a-Service (SaaS)
• Data Center
• Licensing
• AI
• Custom

---

📊 Impacted Parties

• FinOps Practitioner
• FOCUS Data Generator
• Vendor Supporting FOCUS
• Other (please explain in comments)

---

🌫️ Level of Ambiguity

How clearly defined is the request? Rate from 1 to 5:

• 1 = very well-defined, low complexity
• 3 = moderately scoped, some ambiguity
• 5 = vague, high complexity or conceptual

4

---

📂 Supporting Documentation

Reference Issue #1018 and related community call notes.

---

🛠️ Proposed Solution / Approach

FOCUS should establish a non-functional requirement that providers and data generators include supplemental columns in their FOCUS datasets where necessary. These columns:
• MUST accurately represent information available in non-FOCUS datasets but not otherwise covered by FOCUS columns.
• MUST avoid duplication of data already standardized in FOCUS.
• MUST preserve the integrity of FOCUS-defined metrics and dimensions, particularly summable values like costs and quantities.
• SHOULD follow consistent naming conventions (column IDs for files/tables, display names for reports/documentation).
• SHOULD be accompanied by conformance documentation describing how the supplemental columns extend the dataset.

Normative Requirements
When publishing a non-FOCUS dataset alongside a FOCUS dataset:
• MUST include provider-defined custom columns for all information present in the non-FOCUS dataset but absent from FOCUS columns.
• SHOULD allow practitioners to select a subset of FOCUS and custom columns.
• SHOULD provide conformance documentation indicating whether the dataset fully, partially, or does not conform to each requirement.
• SHOULD include explanations for partial or non-conformance so practitioners can adjust their workflows accordingly.

Impact on FOCUS Specification
• Adds clarity that provider-defined columns are expected, not optional, to ensure comprehensive datasets.
• Creates a balance: providers retain flexibility to enrich datasets, while practitioners gain predictability in how supplemental data appears.

---

💬 Community Support (Add Your Voice)

If your organization supports this request or has a similar use case, add a comment with:

• Organization name
• Short rationale or use case

FOCUS Staff & Maintainers will aggregate supporting orgs over time.

[shawnalpay]

Attention @flanakin! This Feature Request is related to your efforts with WI #838 in 1.2. Go forth and evangelize. 🗣

@Matt-Cowsert I think the use case is more around allowing the user to use FOCUS datasets, and only FOCUS datasets, for their FinOps analytics purposes. (Whether one agrees with that philosophy is another thing -- but that's the heart of the feature request.)