Release Notes for CAO - Couchbase Operator 2-9-0 (#45)

* draft 1 of release notes for cao 2-9-0

* Updated preview file

* Updated preview file-a

* testing preview

* RN regenerated

* testing preview 2

* Regenerated the updated release notes

* Updated RN and generated whats new

* minor edit

* minor fix in RN main page

* Minor edit in whatsnew

* Another minor fix in whatsnew

* Formatting fixes

* Added n edited whats new from PM

* generated the edited n rewritten fixed issues

* Minor formatting fixes

* replace the term operator with kubernetes

* Added Known Issues

* minor grammar fix

* Additional sections

* Added link to Server docs

* changed 2-9-0 to 2-9 for consistency

* Listed and rewrote 8 more known issues

* Added 5 more Known Issues

* Updated preview file

* Added 11 more Known Issues

* Added known issue for K8S 4485

Author: rao-shwe

modules/ROOT/pages/release-notes.adoc

@@ -1,68 +1,350 @@
+= Release Notes for Couchbase Kubernetes Operator 2.9
+:page-toclevels: 2
+
+This page summarizes the fixes and known issues in Couchbase Kubernetes Operator 2.9, and links to the associated issues.
+
+== New Features
+
+For information about new features and major improvements made in Couchbase Kubernetes Operator 2.9, see xref:whats-new.adoc[What's New].
+
 [#release-290]
-== Release 2.9.0 (November 2025)
+== Release 2.9 (December 2025)
 
-Couchbase Operator 2.9.0 was released in November 2025.
-This maintenance release contains fixes to issues.
+Couchbase Kubernetes Operator 2.9 was released in December 2025.
+This release contains fixes to issues and known issues.
 
 [#fixed-issues-v290]
-== Fixed Issues
+=== Fixed Issues in 2.9
+
+For Couchbase Kubernetes Operator 2.9 released in December 2025, these are the fixed issues.
+
+*https://jira.issues.couchbase.com/browse/K8S-1537/[K8S-1537]*::
+
+The cluster UUID is no longer required when creating remote cluster connections.
+
+*https://jira.issues.couchbase.com/browse/K8S-2829/[K8S-2829]*::
+
+You can now specify the `cao.couchbase.com/additionalArgs` annotation on CouchbaseBackup and CouchbaseRestore resources to pass additional `cbbackupmgr` arguments to the container.
 
+*https://jira.issues.couchbase.com/browse/K8S-3016/[K8S-3016]*::
+
+You can now specify the Couchbase Server password policy using the CouchbaseCluster resource.
+
+*https://jira.issues.couchbase.com/browse/K8S-3121/[K8S-3121]*::
+
+You can now specify to preserve the CouchbaseBackupRestore resource after the restore completes.
+
+*https://jira.issues.couchbase.com/browse/K8S-3153/[K8S-3153]*::
+
+New TCP tunables (`tcpKeepAliveIdle`, `tcpKeepAliveInterval`, `tcpKeepAliveProbes`, `tcpUserTimeout`) are now available through the CouchbaseCluster resource when using Couchbase Server 8.0.
 
 *https://jira.issues.couchbase.com/browse/K8S-3258/[K8S-3258]*::
 
-Added a new `logging.configNameReleasePrefix` boolean to the helm chart.
-This defaults to false, but setting it to true will prefix the fluent-bit config with the release name.
-Setting this to true for existing clusters will trigger recreation of all pods so should only really be used for new clusters.
+Added the `logging.configNameReleasePrefix` boolean to the Helm chart.
+The default value is `false`.
+When set to `true`, the Operator prefixes the Fluent Bit configuration with the release name.
++
+Couchbase recommends enabling this setting only on new clusters because enabling it on existing clusters triggers recreation of all pods.
 
-*https://jira.issues.couchbase.com/browse/K8S-4091/[K8S-4091]*::
+*https://jira.issues.couchbase.com/browse/K8S-3371/[K8S-3371]*::
+
+You can now specify environment variables for the CouchbaseBackup and CouchbaseBackupRestore pods to allow `cbbackupmgr` tuning.
+
+*https://jira.issues.couchbase.com/browse/K8S-3434/[K8S-3434]*::
+
+`spec.monitoring` is deprecated and no longer attaches an exporter sidecar to the Couchbase Server pod.
+
+*https://jira.issues.couchbase.com/browse/K8S-3535/[K8S-3535]*::
+
+If `couchbasecluster.spec.buckets.managed` is set to `false`, restoring from backup automatically creates buckets.
+
+*https://jira.issues.couchbase.com/browse/K8S-3616/[K8S-3616]*::
+
+New REST API bucket settings for the Data Service are now available in Couchbase Server 8.0.
+
+*https://jira.issues.couchbase.com/browse/K8S-3638/[K8S-3638]*::
+
+You can now specify a merge schedule on the CouchbaseBackup resource.
+
+*https://jira.issues.couchbase.com/browse/K8S-3646/[K8S-3646]*::
+
+You can now set the Query Service `CompletedStreamSize` using the CouchbaseCluster resource.
+
+*https://jira.issues.couchbase.com/browse/K8S-3650/[K8S-3650]*::
+
+When using Couchbase Server 8.0, you can no longer create Memcached buckets.
+
+*https://jira.issues.couchbase.com/browse/K8S-3715/[K8S-3715]*::
+
+Added RBAC roles for users to match new roles added in Couchbase Server 8.0.
+
+*https://jira.issues.couchbase.com/browse/K8S-3786/[K8S-3786]*::
+
+You can now specify `default` and `disk_io_optimized` for the Data Service reader threads.
+
+*https://jira.issues.couchbase.com/browse/K8S-3917/[K8S-3917]*::
+
+You can now set `overheadMemory` for `autoResourceAllocation` to specify a static overhead amount.
 
-Updated the `spec.networking.addressFamily` field to accept `IPv4Only`, `IPv4Priority`, `IPv6Only` and `IPv6Priority`.
-The current `IPv4/IPv6` values will have the `Ipv4/6Only` functionality.
-I.e.
-customers that have set the fields will not see any change.
+*https://jira.issues.couchbase.com/browse/K8S-3951/[K8S-3951]*::
 
+`cao.couchbase.com/autoCompaction.magmaFragmentationPercentage` has been replaced by a field in the CouchbaseCluster CRD.
+
+*https://jira.issues.couchbase.com/browse/K8S-4013/[K8S-4013]*::
+
+You can now disable DNS resolution verification when creating pods before activating them in the cluster.
+
+*https://jira.issues.couchbase.com/browse/K8S-4016/[K8S-4016]*::
+
+Fixed a bug that caused a panic when a member pod became unresponsive.
+
+*https://jira.issues.couchbase.com/browse/K8S-4028/[K8S-4028]*::
+
+Added an upgrade stanza to the CouchbaseCluster resource to give users more control over upgrades.
+
+*https://jira.issues.couchbase.com/browse/K8S-4091/[K8S-4091]*::
+
+Updated `spec.networking.addressFamily` to accept `IPv4Only`, `IPv4Priority`, `IPv6Only`, and `IPv6Priority`.
+The existing `IPv4` and `IPv6` values retain the `IPv4Only` and `IPv6Only` behavior, so no change is required for existing configurations.
 +
-These should be considered deprecated and will be removed in a future release.
- +
-The priority/only choice determines whether `addressFamilyOnly` is true or false.
+These values are deprecated and will be removed in a future release.
++
+The priority or only option determines whether `addressFamilyOnly` is set to `true` or `false`.
 
 *https://jira.issues.couchbase.com/browse/K8S-4097/[K8S-4097]*::
 
-The MirWatchdog is an out-of-band check that allows for additional alerting to be in place in the unlikely scenario that an Operator is unable to reconcile a cluster due to reasons outside of its controls/capabilities and which therefore require manual intervention by a user to resolve.
-Scenarios include but are not limited to, tls expiration, couchbase authentication errors and loss of quorum.
-By default this is disabled, but can be enabled and configured using the `mirWatchdog` field in the couchbase cluster CRD.
+The MirWatchdog is an out-of-band check that provides additional alerting.
+It is used when the Operator cannot reconcile a cluster due to reasons outside its control and requires manual user intervention.
+Scenarios include, but are not limited to, TLS expiration, Couchbase authentication errors, and loss of quorum.
+This feature is disabled by default but can be enabled and configured by using the `mirWatchdog` field in the CouchbaseCluster CRD.
 If the cluster enters this condition, it will:
 +
-* Set the cluster_manual_intervention gauge metric to 1
-* Add (where possible) the `ManualInterventionRequired` condition to the cluster, with a message detailing the reason for entering the MIR state.
-* Raise a `ManualInterventionRequired` Kubernetes event, with the event message set to the reason for entering manual intervention
-* Optionally, reconciliation will be skipped until the manual intervention required state has been resolved, i.e.
-the issue that put the cluster into that condition has been fixed.
+. Set the `cluster_manual_intervention` gauge metric to `1`.
+. Add the `ManualInterventionRequired` condition to the cluster, where possible, with a message describing the reason for entering the MIR state.
+. Raise a `ManualInterventionRequired` Kubernetes event, with the message describing the reason for entering manual intervention.
+. Optionally, skips reconciliation until the manual intervention required state is resolved, that is, until the issue that caused the condition is fixed.
+
+*https://jira.issues.couchbase.com/browse/K8S-4101/[K8S-4101]*::
+
+Added support for the Encryption at Rest feature of Couchbase Server 8.0.
+
+*https://jira.issues.couchbase.com/browse/K8S-4108/[K8S-4108]*::
+
+The CouchbaseUser resource now includes an `enabled` flag to allow administrators to enable or disable user accounts.
+
+*https://jira.issues.couchbase.com/browse/K8S-4109/[K8S-4109]*::
+
+The CouchbaseUser resource now allows the administrators to enforce password change on a user’s first login using the `couchbaseuser.spec.userPassword.requireInitialChange` field.
+
+*https://jira.issues.couchbase.com/browse/K8S-4111/[K8S-4111]*::
+
+CouchbaseBucket resources now support `durabilityImpossibleFallback` with values `disabled` and `fallbackToActiveAck`.
+
+*https://jira.issues.couchbase.com/browse/K8S-4112/[K8S-4112]*::
+
+Added multiple settings to CouchbaseBucket resources to configure XDCR Conflict Logging.
+
+*https://jira.issues.couchbase.com/browse/K8S-4114/[K8S-4114]*::
+
+Added a CouchbaseCluster resource setting that enables auto‑failover of Ephemeral Buckets with no replicas in Couchbase Server 8.0 and later versions.
+
+*https://jira.issues.couchbase.com/browse/K8S-4117/[K8S-4117]*::
+
+Added `data.diskUsageLimit` to the CouchbaseCluster resource to enable Disk Usage Guardrails.
+
+*https://jira.issues.couchbase.com/browse/K8S-4118/[K8S-4118]*::
+
+Added support for SDK Telemetry settings in Couchbase Server 8.0 and later versions.
+
+*https://jira.issues.couchbase.com/browse/K8S-4120/[K8S-4120]*::
+
+For CouchbaseBuckets, now the default storage engine is `magma` and the vBucketCount is `128`.
 
 *https://jira.issues.couchbase.com/browse/K8S-4144/[K8S-4144]*::
 
-In prior versions of Couchbase Operator, the metrics port annotation (`prometheus.io/port`) was set to 8091, even if TLS was enabled.
- It will now correctly set to 18091.
+In the earlier versions of Couchbase Kubernetes Operator, the metrics port annotation `prometheus.io/port` was set to `8091`, even when TLS was enabled.
+It now correctly sets to `18091`.
+
+*https://jira.issues.couchbase.com/browse/K8S-4158/[K8S-4158]*::
+
+EvictionPolicy changes can now be applied to an online bucket during a swap rebalance.
 
 *https://jira.issues.couchbase.com/browse/K8S-4161/[K8S-4161]*::
 
-Operator 2.9.0 now allows you to set `spec.cluster.analytics.numReplicas`.
-This feature is only supported for couchbase server versions 7.6+.
+Operator 2.9.0 allows you to set `spec.cluster.analytics.numReplicas`.
+This feature is supported only on Couchbase Server 7.6 and later versions.
+
+*https://jira.issues.couchbase.com/browse/K8S-4203/[K8S-4203]*::
+
+Fixed an issue where metrics scrapes became too large when the Operator managed many clusters.
+
+*https://jira.issues.couchbase.com/browse/K8S-4209/[K8S-4209]*::
+
+Full backups can now be resumed.
+
+*https://jira.issues.couchbase.com/browse/K8S-4273/[K8S-4273]*::
+
+Fixed an issue where the Operator failed to remove pods from the cluster.
+
+*https://jira.issues.couchbase.com/browse/K8S-4279/[K8S-4279]*::
+
+Fixed an issue where log message tags were inconsistent.
+
+*https://jira.issues.couchbase.com/browse/K8S-4404/[K8S-4404]*::
+
+Fixed an issue that caused upgrades to fail when image definitions used SHA256 digests.
+
+[#known-issues-29]
+=== Known Issues in 2.9
+
+For Couchbase Kubernetes Operator 2.9 released in December 2025, these are the known issues that aren’t yet resolved.
+
+*https://jira.issues.couchbase.com/browse/K8S-3839/[K8S-3839]*::
+
+Removing a server group from a cluster that uses InPlaceUpgrades can cause the Operator to fail to reconcile the cluster.
+
+*https://jira.issues.couchbase.com/browse/K8S-4349/[K8S-4349]*::
+
+The CouchbaseCluster CRD now exceeds the size limit for client-side apply.
+Use the `--server-side` option with `kubectl apply` to apply the resource.
+
+*https://jira.issues.couchbase.com/browse/K8S-4433/[K8S-4433]*::
+
+When running in mixed mode and before the upgrade to 8.0 completes, creating a Memcached bucket can cause the Operator to fail reconciliation.
+
+*https://jira.issues.couchbase.com/browse/K8S-4436/[K8S-4436]*::
+
+Upgrading the Couchbase Kubernetes Operator while the cluster is not fully upgraded causes the Operator to complete the cluster upgrade immediately.
+
+*https://jira.issues.couchbase.com/browse/K8S-4445/[K8S-4445]*::
+
+The CouchbaseRestore resource status may fail to update from Couchbase Backup containers.
+This does not prevent the restore from occurring.
+
+*https://jira.issues.couchbase.com/browse/K8S-4448/[K8S-4448]*::
+
+Changing the server groups applied to a server class while the Operator is recovering a pod can block reconciliation.
+
+*https://jira.issues.couchbase.com/browse/K8S-4456/[K8S-4456]*::
+
+Operator must be paused when using `cao create pod`.
+Otherwise, the Operator identifies the pod as foreign and removes it.
+
+*https://jira.issues.couchbase.com/browse/K8S-4469/[K8S-4469]*::
+
+Editing a Couchbase bucket storage backend while enabling or disabling `BucketMigrationRoutines` on a CouchbaseCluster can trigger a race condition that leads to an unreconcilable state.
+
+*https://jira.issues.couchbase.com/browse/K8S-4471/[K8S-4471]*::
+
+When `spec.networking.addressFamily` is set to `IPv6Priority` or `IPv6Only`, the Operator creates the cluster Service as IPv4 SingleStack, which causes pod launch failures.
+Patching the Service to `PreferDualStack` (IPv4, IPv6) allows the cluster to be created.
+
+*https://jira.issues.couchbase.com/browse/K8S-4474/[K8S-4474]*::
+
+Attempting to change bucket settings during a storage backend migration causes the Operator to fail to reconcile the cluster.
+
+*https://jira.issues.couchbase.com/browse/K8S-4477/[K8S-4477]*::
+
+The admission controller allows you to set the `collectionHistoryDefault` value on Couchstore buckets.
+This setting has no effect on Couchstore buckets.
+
+*https://jira.issues.couchbase.com/browse/K8S-4482/[K8S-4482]*::
+
+Attempting to change the `evictionPolicy` setting during a storage backend migration while `OnlineEvictionPolicyChange` is `true` can lead to an unreconcilable state.
+
+*https://jira.issues.couchbase.com/browse/K8S-4485/[K8S-4485]*::
+
+Manually editing a bucket’s storage backend with BucketMigrationRoutines disabled can prevent the Operator from reconciling the cluster.
+
+*https://jira.issues.couchbase.com/browse/K8S-4486/[K8S-4486]*::
+
+You can start a Couchbase Server upgrade while a Bucket Storage Backend Migration is in progress.
+This can result in an unreconcilable state.
+
+*https://jira.issues.couchbase.com/browse/K8S-4487/[K8S-4487]*::
+
+Manual Intervention Watchdog does not clear the rebalancing condition when entering the Manual Intervention Required condition.
+
+*https://jira.issues.couchbase.com/browse/K8S-4488/[K8S-4488]*::
+
+Reconciliation can begin before the Manual Intervention Watchdog is fully disabled.
+
+*https://jira.issues.couchbase.com/browse/K8S-4490/[K8S-4490]*::
+
+During stabilization, the Operator reconciles some, but not all CouchbaseCluster settings.
+
+*https://jira.issues.couchbase.com/browse/K8S-4496/[K8S-4496]*::
+
+It is possible to set replicas lower than the minimum required for a CouchbaseCluster when using an Ephemeral bucket.
+
+*https://jira.issues.couchbase.com/browse/K8S-4497/[K8S-4497]*::
+
+The dynamic admission controller treats the default bucket storage backend as Couchstore, even on 8.0 clusters.
+
+*https://jira.issues.couchbase.com/browse/K8S-4498/[K8S-4498]*::
+
+The order of password policy updates and user password changes can affect whether the Operator can make these changes successfully.
+
+*https://jira.issues.couchbase.com/browse/K8S-4499/[K8S-4499]*::
+
+It's possible to rotate admin credentials to a password that does not meet the cluster password policy, which can prevent the Operator from reconciling the cluster.
+
+*https://jira.issues.couchbase.com/browse/K8S-4504/[K8S-4504]*::
+
+Setting reader and writer threads to `balanced` during an upgrade from Couchbase Server 7.6 to 8.0 can cause memcached to crash.
+Removing the reader and writer thread settings from the CRD resolves the issue.
+
+*https://jira.issues.couchbase.com/browse/K8S-4507/[K8S-4507]*::
+
+Enabling shard affinity in mixed mode, before the cluster fully supports the setting, can cause the Operator to fail reconciliation.
+
+*https://jira.issues.couchbase.com/browse/K8S-4508/[K8S-4508]*::
+
+The Operator may not clear the unreconcilable condition after the resource is fixed.
+
+*https://jira.issues.couchbase.com/browse/K8S-4509/[K8S-4509]*::
+
+Holding a cluster in mixed mode between 7.2.x and 7.6.x can prevent the Operator from reconciling some cluster settings.
+
+*https://jira.issues.couchbase.com/browse/K8S-4510/[K8S-4510]*::
+
+Remaining in mixed mode on older Couchbase Server versions can cause the Operator to log errors due to incorrect version checks.
+
+*https://jira.issues.couchbase.com/browse/K8S-4511/[K8S-4511]*::
+
+You cannot currently specify Arbiter nodes in the services order during an upgrade.
+
+*https://jira.issues.couchbase.com/browse/K8S-4512/[K8S-4512]*::
+
+The Operator can update Index Storage settings twice accidentally.
+
+*https://jira.issues.couchbase.com/browse/K8S-4514/[K8S-4514]*::
+
+The Operator can update the Fluent Bit configuration accidentally.
+
+*https://jira.issues.couchbase.com/browse/K8S-4515/[K8S-4515]*::
+
+While processing a bucket migration, the Operator does not send the correct online change flag for modifications to Couchbase Server when changing the Eviction Policy.
+
+*https://jira.issues.couchbase.com/browse/K8S-4520/[K8S-4520]*::
+
+The admission controller allows encryption at rest to be enabled in mixed mode.
+
+*https://jira.issues.couchbase.com/browse/K8S-4521/[K8S-4521]*::
 
-*https://jira.issues.couchbase.com/browse/K8S-4270/[K8S-4270]*::
+The admission controller allows use of the Couchbase User `spec.user` field for users that reference clusters not running version 8.0.0.
 
-Potentially where we use `kubectl apply` for CRDS, we add a note that this error is possible in 2.9+, and to add `--server-side` to the `kubectl apply` command.
+== Feedback
 
-*https://jira.issues.couchbase.com/browse/K8S-4286/[K8S-4286]*::
+You can have a big impact on future versions of the Operator (and its documentation) by providing Couchbase with your direct feedback and observations.
+Please feel free to post your questions and comments to the https://forums.couchbase.com/c/couchbase-server/Kubernetes[Couchbase Forums].
 
-In the latest build,
-the mirWatchdog feature is now set to off by default.
-The sequence has been adjusted to move the skip function after the validationRunner,
-and changes have been included in the CRD.
-Additionally,
-the system now skips the DAC during status changes.
-These updates aim to streamline operations and improve efficiency.
-// Generated by [chatgpt:gpt-4o]
+== Licenses for Third-Party Components
 
+The complete list of licenses for Couchbase products is available on the https://www.couchbase.com/legal/agreements[Legal Agreements] page.
+Couchbase is thankful to all of the individuals that have created these third-party components.
 
+== More Information
 
+* xref:server:release-notes:relnotes.adoc[Couchbase Server Release Notes Version 8.0]
+* xref:server:introduction:whats-new.adoc[What's New in Couchbase Server Version 8.0]