Addressed comments

Author: jatinsu

enhancements/okd/okd-featureset.md

@@ -83,10 +83,12 @@ This proposal introduces a new "OKD" feature set to the OpenShift API configurat
 #### Scenario 1: Installing a new OKD cluster
 
 1. OKD cluster administrator initiates a cluster installation using `openshift-install` built for OKD
-2. The installer automatically sets the feature set to "OKD" in the cluster configuration
-3. The cluster installs with the OKD feature set enabled by default
-4. The cluster has access to TechPreview features while maintaining upgrade capability
-5. The administrator can upgrade the cluster to newer OKD versions without changing the feature set
+2. During cluster bootstrap, the Cluster Version Operator (CVO) starts up
+3. The CVO detects it is running on an OKD build (via build metadata)
+4. If no FeatureGate resource exists, the CVO assumes the OKD feature set
+5. If a FeatureGate resource exists with Default ("") feature set, the CVO automatically migrates it to "OKD" by patching the resource
+6. The cluster operates with the OKD feature set, providing access to select TechPreview features while maintaining upgrade capability
+7. The administrator can upgrade the cluster to newer OKD versions without changing the feature set
 
 #### Scenario 2: Attempting to enable OKD feature set on OpenShift
 
@@ -99,9 +101,9 @@ This proposal introduces a new "OKD" feature set to the OpenShift API configurat
 
 1. OKD cluster administrator has a running cluster with `featureSet: OKD`
 2. Administrator attempts to change the feature set to "Default" 
-3. The API validation rejects the change with error: "OKD may not be changed"
+3. The API validation rejects the change with error: "OKD cannot transition to Default"
 4. The cluster continues operating with the OKD feature set
-5. The administrator must reinstall the cluster if a different feature set is required
+5. The administrator must reinstall the cluster if the Default featureset is required
 
 #### Scenario 4: Attempting to Change OKD to TechPreviewNoUpgrade or DevPreviewNoUpgrade
 1. OKD cluster administrator has a running cluster with `featureSet: OKD`
@@ -156,7 +158,7 @@ And various operator-specific CRDs across multiple API groups.
 
 #### Standalone Clusters
 
-The OKD feature set is specifically designed for standalone OKD clusters and is the primary use case for this enhancement. The feature set will be enabled by default during installation and cannot be changed afterwards.
+The OKD feature set is specifically designed for standalone OKD clusters and is the primary use case for this enhancement. The feature set will be enabled by default during installation and can be changed to TechPreviewNoUpgrade or DevPreviewNoUpgrade afterwards.
 
 #### Single-node Deployments or MicroShift
 
@@ -196,6 +198,24 @@ var AllFixedFeatureSets = []FeatureSet{
 }
 ```
 
+There is also a test to ensure that all featuregates enabled in the Default featureset are enabled in the OKD featureset
+```
+```
+	// Check that all Default featuregates are in OKD
+		missingInOKD := defaultEnabled.Difference(okdEnabled)
+
+		if missingInOKD.Len() > 0 {
+			missingList := missingInOKD.List()
+			sort.Strings(missingList)
+
+			t.Errorf("ClusterProfile %q: OKD featureset is missing %d featuregate(s) that are enabled in Default:\n  - %s\n\nAll featuregates enabled in Default must also be enabled in OKD.",
+				clusterProfile,
+				missingInOKD.Len(),
+				strings.Join(missingList, "\n  - "),
+```
+
+
+```
 #### Validation Rules
 
 The FeatureGate spec includes Kubernetes validation rules:
@@ -211,11 +231,12 @@ This ensures:
 
 #### Platform Detection
 
-The installer and cluster operators must be able to detect whether they are running on OKD vs OpenShift to:
-- Automatically enable the OKD feature set during installation on OKD clusters
+The CVO must be able to detect whether it is running on OKD vs OpenShift to:
+- Automatically enable the OKD feature set on OKD clusters
+- Automatically migrate existing OKD clusters from Default to OKD feature set
 - Prevent the OKD feature set from being enabled on OpenShift
 
-This detection is typically done through:
+This detection is done through the `version.IsOKD()` function in the CVO, which typically checks:
 - Build tags during compilation (`scos` for OKD, `ocp` for OpenShift)
 - Cluster version metadata
 - Installation metadata persisted during cluster creation
@@ -277,6 +298,7 @@ Instead of creating a new OKD feature set, we could modify the TechPreviewNoUpgr
 - Documentation and user expectations would be confusing
 - API contracts would be violated (TechPreviewNoUpgrade explicitly blocks upgrades)
 - Difficult to maintain and reason about platform-specific behavior
+- Would require a reinstallation for an upgrade
 
 ### Alternative 2: Use CustomNoUpgrade for OKD
 
@@ -288,6 +310,7 @@ Instead of creating a dedicated OKD feature set, use the existing CustomNoUpgrad
 - No clear differentiation between OKD and custom configurations
 - Does not provide a default, curated experience for OKD users
 - Makes it harder to manage and communicate what features are enabled on OKD
+- Would require a reinstallation for an upgrade
 
 ### Alternative 3: Create an OKDTechPreview Feature Set
 
@@ -298,6 +321,16 @@ Create a separate OKDTechPreview feature set in addition to the OKD feature set
 - The OKD feature set can already include appropriate TechPreview features
 - OKD's role as a community distribution means users expect to adopt new features
 - Can be reconsidered in the future if the use case becomes clearer
+- May not support upgrading, which would force users to potentially reinstall if they want upgrades
+
+### Alternative 4: Create a OKD clusterprofile
+
+Instead of creating a OKD feature set, create a new Clusterprofile that would enable certain featuregates in addition to the default featuregates
+
+**Rejected because:**
+- A Clusterprofile is outside the scope of our goal
+- Implementing a cluster profile is difficult and affects more components then is required
+- OKD featureset is a "featuregate profile" rather than a cluster profile
 
 ## Open Questions [optional]
 
@@ -319,54 +352,25 @@ Create a separate OKDTechPreview feature set in addition to the OKD feature set
 
 **Integration Tests:**
 - API server correctly rejects attempts to enable OKD on OpenShift clusters
-- API server correctly rejects attempts to change from OKD to other feature sets
+- API server correctly rejects attempts to change from OKD to default
 - FeatureGate custom resource can be created with OKD feature set on OKD clusters
 - Feature gates are correctly applied when OKD feature set is enabled
 
-**E2E Tests:**
-- OKD cluster installs successfully with OKD feature set enabled by default
-- OKD cluster can be upgraded with OKD feature set enabled
-- Features enabled by the OKD feature set function correctly
-- OpenShift cluster installation/configuration rejects OKD feature set
-
 **Upgrade Tests:**
 - OKD clusters with OKD feature set can upgrade from version N to N+1
 - Feature set remains OKD after upgrade
 - Feature gates are correctly maintained across upgrades
 
 ## Graduation Criteria
 
-The OKD feature set will be introduced as a stable feature, not following the typical Dev Preview -> Tech Preview -> GA progression, because:
-
-1. It is part of the OpenShift API contract
-2. The feature set mechanism is already GA
-3. This is adding a new value to an existing, stable enum
-4. OKD is already a mature distribution
-
-### Initial Release
-
-The OKD feature set will be considered stable when:
-
-- [ ] All API changes are merged to openshift/api
-- [ ] CRD manifests are generated for all relevant API resources
-- [ ] Validation logic is implemented and tested
-- [ ] Installer changes to enable OKD feature set by default are implemented
-- [ ] CI jobs for OKD with the new feature set are passing
-- [ ] Documentation is updated to describe the OKD feature set
-
-### Ongoing Requirements
-
-- Maintain compatibility with feature gate framework changes
-- Keep CRD manifests in sync as new API versions are added
-- Document any new features added to the OKD feature set
-- Ensure CI continues to test OKD with the feature set enabled
+**N/A as this is OKD**
 
 ## Upgrade / Downgrade Strategy
 
 ### Upgrade Strategy
 
 **OKD Clusters:**
-- Existing OKD clusters without the OKD feature set: During the upgrade to the first version supporting the OKD feature set, the feature set should be automatically enabled if the cluster is detected as OKD. This should be handled by the cluster-version-operator or similar component.
+- Existing OKD clusters without the OKD feature set: During the upgrade to the first version supporting the OKD feature set, the CVO automatically migrates the Default feature set to OKD during startup. The CVO detects the OKD build via `version.IsOKD()` and patches the FeatureGate resource accordingly.
 - OKD clusters with OKD feature set already enabled: No changes needed; the feature set persists across upgrades.
 - Upgrades are explicitly supported with the OKD feature set enabled.
 
@@ -375,12 +379,7 @@ The OKD feature set will be considered stable when:
 
 ### Downgrade Strategy
 
-**Downgrading from a version with OKD feature set to a version without:**
-- If an OKD cluster with the OKD feature set is downgraded to a version that does not recognize the OKD feature set:
-  - The cluster-version-operator should handle the unknown feature set gracefully
-  - Ideally, the cluster should continue operating but may log warnings about the unknown feature set
-  - This scenario should be tested to ensure it does not break the cluster
-  - If necessary, clusters may need to be reinstalled rather than downgraded
+**Downgrading from a version with OKD feature set to a version without is not supported**
 
 **General downgrade considerations:**
 - Downgrades are generally not supported in OpenShift/OKD
@@ -389,51 +388,20 @@ The OKD feature set will be considered stable when:
 
 ### Migration Path for Existing OKD Clusters
 
-OKD clusters deployed before the introduction of the OKD feature set will need a migration strategy:
-
-**Option 1: Automatic migration during upgrade**
-- Detect OKD clusters using build metadata or cluster version
-- Automatically enable the OKD feature set during CVO upgrade
-- Log the change clearly for administrator awareness
+OKD clusters deployed before the introduction of the OKD feature set will be automatically migrated:
 
-**Option 2: Manual migration**
-- Require administrators to manually set the OKD feature set
-- Provide clear documentation and tooling
-- May be more transparent but requires user action
+**Automatic migration during upgrade:**
+- The CVO detects OKD clusters using build metadata (`version.IsOKD()`)
+- During CVO startup, if the FeatureGate resource has Default ("") feature set, the CVO automatically patches it to "OKD"
+- The migration is logged clearly for administrator awareness
+- If the patch fails, the CVO logs a warning and continues with Default, retrying on next restart
+- After successful migration, the CVO restarts to cleanly apply the new feature set
 
-**Recommendation:** Implement Option 1 (automatic migration) with clear logging and documentation. This provides the smoothest upgrade experience for OKD users.
+This automatic migration provides the smoothest upgrade experience for OKD users without requiring manual intervention.
 
 ## Version Skew Strategy
 
-The OKD feature set introduces version skew considerations:
-
-**Control Plane and Node Version Skew:**
-- Feature gates are primarily evaluated at the control plane (API server) level
-- Nodes respect feature gates propagated through the kubelet configuration
-- Standard OpenShift version skew policies apply (e.g., nodes can be N-2 versions behind control plane)
-- The OKD feature set does not introduce new version skew constraints beyond existing feature gate behavior
-
-**Component Version Skew:**
-- All components must be aware of the OKD feature set enum value
-- Components from older versions that do not recognize "OKD" as a valid value may fail validation
-- This is mitigated by:
-  - Synchronizing API changes across all components
-  - Using CI to test component compatibility
-  - Following standard OpenShift component versioning
-
-**Operator Version Skew:**
-- Operators must handle clusters with the OKD feature set enabled
-- Operators should either:
-  - Explicitly support the OKD feature set
-  - Treat it equivalently to an appropriate existing feature set (likely Default + TechPreview)
-  - Gracefully handle unknown feature sets
-
-**API Client Version Skew:**
-- Clients using older API definitions may not recognize the OKD feature set value
-- This is acceptable as long as:
-  - Clients do not actively reject unknown enum values
-  - The API server continues to accept and persist the OKD value
-  - Clients can read the raw value even if they don't understand it
+**We plan to deliver this as part of a single release so there will be no version skew.**
 
 ## Operational Aspects of API Extensions
 
@@ -476,19 +444,13 @@ The OKD feature set itself does not introduce new SLIs, but relies on existing i
 
 ### Failure Modes
 
-**Failure Mode 1: Invalid feature set value**
-- **Symptom:** API server rejects FeatureGate resource with validation error
-- **Impact:** Cluster administrators cannot modify the FeatureGate configuration
-- **Detection:** API server logs show validation errors; CLI commands return error messages
-- **Recovery:** Correct the feature set value to a valid option (Default, TechPreviewNoUpgrade, CustomNoUpgrade, OKD, or empty string)
-
-**Failure Mode 2: Attempt to change OKD feature set to Default**
+**Failure Mode 1: Attempt to change OKD feature set to Default**
 - **Symptom:** API server rejects update with error "OKD cannot be transitioned to Default"
 - **Impact:** Cluster administrators cannot change the feature set from OKD to default 
 - **Detection:** API server logs show validation errors; CLI commands return error messages
 - **Recovery:** This is expected behavior; cluster must be reinstalled if a the default featureset is required
 
-**Failure Mode 3: Version skew in component awareness of OKD feature set**
+**Failure Mode 2: Version skew in component awareness of OKD feature set**
 - **Symptom:** Components fail to start or report degraded status due to unknown feature set value
 - **Impact:** Specific operators or components may not function correctly
 - **Detection:** Operator logs show errors about unknown feature set; operator conditions show Degraded=True
@@ -539,11 +501,6 @@ oc adm node-logs --role=master --path=kube-apiserver/audit.log | grep -i feature
 
 **Important:** The OKD feature set cannot be disabled once enabled. This is by design to ensure cluster consistency.
 
-**If OKD feature set must be removed:**
-1. Back up all critical cluster data and configurations
-2. Plan for cluster downtime
-3. Reinstall the cluster with the desired feature set
-4. Restore applications and data to the new cluster
 
 ### Graceful Degradation
 
@@ -556,40 +513,10 @@ The OKD feature set validation is enforced at the API level:
 
 ### Impact on Cluster Health
 
-**When OKD feature set is functioning correctly:**
-- No impact on cluster health indicators
-- Cluster operates normally with features enabled according to the OKD feature set definition
-
-**When OKD feature set is configured incorrectly:**
-- API server prevents invalid configurations through validation
-- Cluster continues operating with last known good configuration
-- No automatic reconciliation or rollback occurs
-- Manual intervention required to correct configuration issues
+Stability depends on the features that are added outside the Default featureset. We need to be careful when we choose to not ruin the stability of the cluster.
 
 ## Infrastructure Needed [optional]
-
-**CI Infrastructure:**
-- OKD build and test jobs must be updated to expect the OKD feature set by default
-- E2E test suites should include scenarios with the OKD feature set enabled
-- Upgrade test jobs for OKD should verify feature set persistence
-
-**Build Infrastructure:**
-- No changes needed; existing OKD build infrastructure can accommodate this change
-- The `scos` build tag will continue to differentiate OKD from OpenShift builds
-
-**Documentation:**
-- Update OKD installation documentation to explain the OKD feature set
-- Add troubleshooting guides for feature set related issues
-- Document the differences between OKD and OpenShift feature sets
-- Provide guidance on which features are enabled in the OKD feature set
-
-**Repository Infrastructure:**
-- No new repositories required
-- Changes are made to existing openshift/api repository and will be vendored to other repos
-- The OpenShift Kubernetes repo will need changes
-- The Cluster Config Operator repo will need changes to allow the OKD featureset to allow upgrades
-- Generated CRD manifests will be committed to the repository
-
+**N/A**
 ## Implementation History
 
 - 2025-08-12: Initial PR opened (https://github.com/openshift/api/pull/2451)