From e0441ceb46a204b2a631548eb665ce2fff4dffe3 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 18:05:57 -0500 Subject: [PATCH 01/10] WIP Signed-off-by: John Matthews --- docs/Overview.md | 319 +++++++++++++++++------------------------------ 1 file changed, 115 insertions(+), 204 deletions(-) diff --git a/docs/Overview.md b/docs/Overview.md index 4ca644a..9cfcc55 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -1,223 +1,134 @@ -# Workload Availability Operator overview +# Overview of Workload Availability Operator The Workload Availability Operator (WAO) packages six medik8s operators as one -installation. This proof of concept tests a simpler way to distribute and manage -those operators. It preserves their existing behavior and upstream development. - -This guide explains the design, then shows how to install WAO and enable one -component on a disposable OpenShift cluster. - -## What WAO manages - -An **operand** is a component that another operator manages. WAO treats each -medik8s operator as an operand, with its own controller process. - -The components cover three tasks: - -- **Detect unhealthy nodes:** Node Health Check (NHC) creates remediation requests - according to a configured policy. -- **Remediate nodes:** Self Node Remediation (SNR), Storage Based Remediation - (SBR), Fence Agents Remediation (FAR), and Machine Deletion Remediation (MDR) - provide different remediation methods. -- **Maintain nodes:** Node Maintenance (NMO) cordons and drains a node when an - administrator creates a maintenance request. - -The [component table](../README.md) -links to each upstream project. - -## How the parts work together - -WAO separates package installation, component activation, and node operations. - -| Stage | Responsible controller | Result | -| --- | --- | --- | -| Install WAO | Operator Lifecycle Manager (OLM) | Installs WAO, all component API definitions, and component permissions. | -| Enable a component | WAO | Creates that component's Deployments, Services, configuration, and webhooks. | -| Process a policy or request | The upstream component | Performs health checks, remediation, or maintenance through its existing API. | - -OLM uses one WAO package instead of six independent component packages. Its -bundle contains the ClusterServiceVersion (CSV), which describes the installation, -and the CustomResourceDefinitions (CRDs), which define the APIs. - -WAO creates one cluster-scoped `WorkloadAvailabilityConfig` named `cluster`. -Every component starts `Disabled`. An administrator changes a component's -`state` to `Enabled` to start its controller in WAO's installation namespace. - -The WAO controller repeatedly compares the requested state with the cluster. -It applies the component resources and reports deployment and webhook readiness. -The component continues to process its own API objects independently. - -**Enabling a component can have immediate effects.** For example, SNR starts -privileged node agents when enabled. NMO requires a separate `NodeMaintenance` -request before it drains a node. The trial below uses NMO. - -## Why the design uses these boundaries - -- **Keep the upstream operators unchanged.** WAO consumes published component - images. Each project retains its controllers and APIs, while WAO provides one - installation and upgrade lifecycle. -- **Let OLM install permissions.** OLM creates component service accounts and - role-based access control (RBAC) resources during installation. WAO manages - component workloads. Component permissions remain present when a component - is disabled. -- **Start with every component disabled.** Installing WAO creates no component - controller, remediation policy, remediation request, native configuration, or - node agent. Administrators choose which components to activate. -- **Preserve resource ownership.** WAO reports a conflict if a required object - already exists without WAO ownership. It does not adopt that object. -- **Make disabling explicit.** WAO removes the resources it manages, including - webhooks. It preserves webhook failure policies and never changes `Fail` to - `Ignore`. Objects created by an upstream controller can remain. - -Disabling a component stops its manager, but existing agents or native API -objects can remain active or await that manager. WAO reports detected retained -objects in `status.components[].retainedResources` with phase `DisableIncomplete`. -Complete active maintenance or remediation before disabling a component. -See [Enabled and Disabled behavior](../README.md#what-enabled-and-disabled-mean) -for the component-specific consequences. - -### How upstream configuration becomes a WAO release - -WAO copies each upstream `config/` directory into this repository. Kustomize -overlays apply WAO's adjustments without editing the copied files. - -```text -Copied upstream config/ - ├─ CRDs and RBAC overlay → OLM bundle - └─ Runtime overlay → manifests embedded in the WAO binary +installation. It intends to simplify publishing and installation while preserving +the existing behavior. + +This document explains the design and how to work with it. + +## Goal +Would like to see Medik8s 5.9 published as a single operator. +Need to work over next month to learn if there are any blockers that would prevent this. + + +## High Level Understanding +![WAO High Level](images/WAO_High_Level.svg) + +### WAO is a single operator that manages operands + +WAO treats the existing medik8s operators as **operands** which are managed by the WAO operator. + +WAO is responsible for: +- **CRDs** - Collecting all of the operand CRDs and packaging so OLM will deploy. + - This means that all of the operand CRDs are installed into the cluster at time of WAO installtion. + - CRDs are NOT gated on the WAO operator's `enabled` state. +- **RBAC** - Collecting all of the operand RBAC related files and packaging in the WAO bundle so OLM will deploy on install of WAO. +- **Runtime manifests** - Understanding the runtime manifests for each operand and deploying them when the operand is `enabled`. + - and...cleaning them up when the operand is `disabled`. + +WAO is **not** responsible for the business logic of each operand. +- Each operand will continue to be developed in it's own repository. +- Operand changes are intended to be minimal to none + - As of today we have not made any changes to any of the operands, yet it's likely will want to refactor the webhooks soon to allow them to be brought into the main WAO manager process. + +### Recap of responsbilitlities: OLM - WAO - Operand +- **OLM** + - On install of WAO + - Deploys all CRDs + - Deploys all RBAC + - Deploys the WAO operator +- **WAO** + - On enable + - Deploys the operand's runtime manifests + - On disable + - Removes the operand's runtime manifests +- **Operand** + - Responsible for reconciling it's CRs + - Currently runs their own webhook server, we likely will want to move this to run in the main WAO manager process. + +### What does install of WAO look like? + +1. Install the WAO operator via OLM +2. The WAO operator creates the `WorkloadAvailabilityConfig` CR called `cluster` on bootstrap + - Only one instance of a `WorkloadAvailabilityConfig` is allowed as per validation in the `WorkloadAvailabilityConfig` CRD +3. All operands are set to be disabled by default +4. User needs to enable operands +5. After enabling operands, the WAO operator will reconcile the operand's runtime manifests + 6. It will create the Deployment, Service, ConfigMaps, and Webhooks of the operand. + +#### How does a user enable/disable an operand? + +Edit the `WorkloadAvailabilityConfig` CR called `cluster` to enable or disable an operand. + +See [/config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig.yaml](/config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig.yaml) +```yaml +apiVersion: workloadavailability.medik8s.io/v1alpha1 +kind: WorkloadAvailabilityConfig +metadata: + name: cluster +spec: + components: + nodeHealthCheck: + enabled: false + selfNodeRemediation: + enabled: false + storageBasedRemediation: + enabled: false + nodeMaintenance: + enabled: false + fenceAgentsRemediation: + enabled: false + machineDeletionRemediation: + enabled: false ``` +* For testing there is a script [hack/enable_all.sh](../hack/enable_all.sh) +* You can also tweak the memory and cpu limits and requests for each operand, [config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml](../config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml) -The overlays separate installation resources from resources needed when a -component starts. They also provide namespace settings, component selectors, -and OpenShift service-ca integration for webhook certificates. + -Developers select image repositories and tags in [operands.yaml](../operands.yaml). -WAO substitutes those image references into its embedded manifests at runtime. -The public configuration API has no image override. The current development -workflow trusts the selected tags and source branches without resolving image -digests or source revisions. -See [operand configuration overlays](operand-config-overlays.md) for the design. -See [updating operand manifests](updating-operand-manifests.md) for the commands. +## Goal -## Try the component lifecycle +Goal -### 1. Prepare +WAO branches and links -Use a disposable OpenShift cluster that you are authorized to modify. It must -have classic OLM and the OpenShift service-ca controller. Use a cluster without -existing installations of the six component operators. -The cluster must be able to pull the images selected in [operands.yaml](../operands.yaml). +Gists of example of commands -On your workstation, you need: +Walk through manual update of an operand -- Git, Bash, Make, curl, and jq. -- Go matching [go.mod](../go.mod). -- The OpenShift `oc` client and an explicit kubeconfig with cluster-admin access. -- A running Podman engine. For Docker, add `CONTAINER_TOOL=docker` to the Make commands. +Walk through helper with operands.yaml -Run the commands from the WAO repository root in the same shell. Set the path -to your disposable cluster's kubeconfig: +Walk through bundle formation with overlays +Walk through resource changes for limits +Walk through objects yaml -```sh -WAO_KUBECONFIG=/path/to/disposable.kubeconfig -``` - -The build uses the copied manifests already in this repository. You do not need -the six upstream checkouts unless you want to refresh those copies. - -### 2. Install WAO - -```sh -KUBECONFIG="$WAO_KUBECONFIG" make dev-olm-deploy DEV_CLUSTER_TYPE=external - -oc --kubeconfig="$WAO_KUBECONFIG" wait --for=create \ - workloadavailabilityconfig/cluster --timeout=120s - -oc --kubeconfig="$WAO_KUBECONFIG" get workloadavailabilityconfig cluster -o yaml -``` - -The target builds your working tree, including uncommitted changes. It publishes -development images to ttl.sh and installs the bundle through OLM. These images -expire after two hours by default. Repeat the deployment if they expire. - -The Makefile uses shared [medik8s/tools targets](https://github.com/medik8s/tools/blob/main/dev/dev.mk). -It uses `../tools` when available, or downloads the tools into `.tools/`. -Make installs the required generation and packaging tools when needed. - -Expect all six component states and phases to become `Disabled`. -The `Available=True` condition means WAO completed reconciliation. It does not -mean that workloads have protection. - -### 3. Enable Node Maintenance - -This step starts the NMO controller. **Do not create a `NodeMaintenance` request -for this trial.** Such a request would cordon and drain a real node. - -```sh -oc --kubeconfig="$WAO_KUBECONFIG" patch workloadavailabilityconfig cluster \ - --type=merge -p '{"spec":{"components":{"nodeMaintenance":{"state":"Enabled"}}}}' - -oc --kubeconfig="$WAO_KUBECONFIG" wait workloadavailabilityconfig/cluster \ - --for='jsonpath={.status.components[?(@.name=="nodeMaintenance")].phase}=Available' \ - --timeout=300s -oc --kubeconfig="$WAO_KUBECONFIG" get deployments \ - -n openshift-workload-availability \ - -l workloadavailability.medik8s.io/component=nodeMaintenance -``` - -Expect the NMO Deployment to become ready. Its component phase becomes -`Available` after the deployment is ready and its webhook CA bundle is present. -The other five components remain disabled. - -If the wait fails, inspect the configuration status: - -```sh -oc --kubeconfig="$WAO_KUBECONFIG" get workloadavailabilityconfig cluster -o yaml -``` - -Check the component's `message`, `deployments`, and `admissionReady` fields. -`Progressing` means WAO is waiting for readiness. `Degraded` reports a -reconciliation error, such as an ownership conflict. - -### 4. Disable NMO and remove the trial installation +Walk through next steps +Test +Upgrade scenario. -```sh -oc --kubeconfig="$WAO_KUBECONFIG" patch workloadavailabilityconfig cluster \ - --type=merge -p '{"spec":{"components":{"nodeMaintenance":{"state":"Disabled"}}}}' +Clean up +Gap no webhooks until enable +Objects yaml in golang -oc --kubeconfig="$WAO_KUBECONFIG" wait workloadavailabilityconfig/cluster \ - --for='jsonpath={.status.components[?(@.name=="nodeMaintenance")].phase}=Disabled' \ - --timeout=300s -``` - -Expect WAO to remove the NMO Deployment and webhook configuration. -If the phase becomes `DisableIncomplete`, inspect `retainedResources` before -continuing. Follow the [uninstall guidance](../README.md#uninstall) if you enabled -other components or created native API objects. - -After the wait succeeds, remove the configuration while WAO still runs: - -```sh -oc --kubeconfig="$WAO_KUBECONFIG" delete workloadavailabilityconfig cluster - -KUBECONFIG="$WAO_KUBECONFIG" make dev-olm-undeploy DEV_CLUSTER_TYPE=external - -make bundle-reset -``` +Integration with Virt auto pilot -The development cleanup leaves CRDs installed. `bundle-reset` restores normalized -local bundle content after the development build changes it. Run it before tests -or a commit. +## Current Status +### Testing +- Testing is early, we've started to capture notes here: + - https://github.com/medik8s/.github/wiki/wao-testing +- Beginning to setup automated tests starting with: + - https://github.com/openshift/rhwa-system-tests/pull/112 +### 'wao' branch +WAO is setup to track the `wao` branch of each operand's repo. +The `wao` branch started from each operand's`medik8s5.8` branch as of 11/2/2026 -## Continue from here -- **Change an operand version or configuration:** use the - [update guide](updating-operand-manifests.md). -- **Read the controller:** start with the - [configuration API](../api/v1alpha1/workloadavailabilityconfig_types.go), then the - [lifecycle controller](../internal/controller/workloadavailabilityconfig_controller.go) - and [manifest renderer](../internal/operands/render.go). -- **Contribute a change:** see [development and tests](../README.md#development) - and the repository [contributor instructions](../AGENTS.md). +## Next Steps +1. Testing: Work to understand if there are blockers that prevent the goal of WAO for medik8s 5.9 +2. Iron out the upgrade scenario +2. Additional cleanup in WAO code: + - Implement `bundle-k8s`, we currently only create a bundle for `ocp` + - Consider: Bring webhooks to run in the WAO process so they are availabile and running even when a component is disabled. + - Consider: Rewrite the internal/operands/manifests//objects.yaml in Go and stop trying to sync from the operand repo. From 5ec822e80ead3540d29212591d62f9c00a073db2 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 18:08:01 -0500 Subject: [PATCH 02/10] Add high level svg Signed-off-by: John Matthews --- docs/images/WAO_High_Level.svg | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 docs/images/WAO_High_Level.svg diff --git a/docs/images/WAO_High_Level.svg b/docs/images/WAO_High_Level.svg new file mode 100644 index 0000000..48e1f6a --- /dev/null +++ b/docs/images/WAO_High_Level.svg @@ -0,0 +1,4 @@ + + +eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO19V3NcIkuT9v37K4jZ26W3vNk7kIQsyIDsdzZcdTAwMTSYRkJCwGDkNt7/vploRjR0NWqghSA+mDgnZrDVXZnPk5mV5n//lUr96r91/F//nfrlv1bLzUatW3759Z/4/LPf7TXaLXiJXHL/3WtcdTAwMGa61eE77/v9Tu+//+u/yp2ON/qUV20/fXzSb/pPfqvfg/f+P/h3KvW/w//DK41cdTAwMWF+Pps+zPJivtx9yD4+m07//v2431x1MDAxOX50+Ka/XHUwMDBi6vrVfrl11/RHL73iarjxXGKXVComP194g1x1MDAxN4SRnuGcXGKqrVx1MDAxMsyoz1dfXHUwMDFhtf49vIMy41FcIoUyevTRe79xd9+HV1x1MDAxNfN44PmPn/7vXHUwMDE0+Xym1++2XHUwMDFm/a12s93F9f1cdTAwMDf18c9odZVy9fGu21x1MDAxZbRqn+/pd8utXqfchdsxel+90WxcdTAwMTb7b8Nvh9tcbrfv18RvXFz+WTCbeD7qU/Cjd/ctv4e3nH4+2+6Uq40+3lx1MDAxYUpGV4Er7OzXhrvzP6M1dctP/j5uT2vQbH4+3WjVfLzpv8pk7NdatT+/9ndrR/vG/zzz79HafVx1MDAxZr/YSmGtIXR0USNcdTAwMTGjSkw+W2i3huJGmeDKWNjv0ap62yBj/eG31svNnj/aXHUwMDAyXFzazqT8XHUwMDA1ZXBMxPr+62hfXHUwMDAyXHUwMDEy+vrS2H1o2H67/dq/eDlriFx1MDAwN//49Nfn+/7952+j2zfo1MpcdTAwMWbrXHUwMDAx4WPKcEElXHUwMDE39vP1atdcdTAwMWZ7nTGBb/p8vdloPU7e+2a7+ji6xH9cdTAwMDXu6YQ+uVdcdTAwMWLSp7GL/VAlQz3OtCVcXFx1MDAwYi1lQGNQaCRcdTAwMTGeZKAukfokqCeptVx1MDAxYdRGXHUwMDE4XHUwMDAz31x1MDAxNlYqtlGoXGKFom6FXHUwMDFhe/dcdTAwMWbNUVZYJjkxLs1hNkpzXHUwMDE0IYxYoedSnIVk29jAWr+W7ZGooojCzTk+ylx1MDAwN3a33epcdTAwMTdcdTAwMWLveEWMXHUwMDA0ZKPn50avjH07flwiV35qNIdiPPbtmWbjXHUwMDBlb9uvKlxcqN/9XHUwMDE1vHf9XHUwMDA2cNnnXHUwMDFinlx1MDAxYbVakHuq8KXlRsvv7schsXa3cddolZsl11x1MDAxNZVcdTAwMDf99pnf+1h5vzvwg/fK3/urO9RcdTAwMGJoT7NcXPGbJ+1eo/+x73jBUzAhk0/nuvSm9fu895vdlNPNJ2XzsTlWXHUwMDEx+G3FlaZBpnz7XHUwMDAzXHUwMDE5Ulx1MDAxMm1oYG1cdTAwMWJ+XVx1MDAxY1x1MDAwZdjC/KpcdTAwMDRcdTAwMTGGKlx1MDAxMmLS4YZG0qtkXHUwMDEyWNmokcJ+O732TmpcdTAwMTecNWjl8fdLd++9Re/pVn1cdTAwMTZ6ZUxKzUa4XHUwMDE2hiApuLVBtZqfXt2rjUGviinPKFx1MDAwNSsx1HIu7YQmaU9zZE2HJmn4pLWWXHUwMDAz61KOb1x0q5MkXHUwMDFiVXKrXHUwMDEyj8+sYN7AXHUwMDFkJmH1XHUwMDE46lOUzlxiXHL2KJeczaMy06RaKW3kVKlcdTAwMTaB2zNcdTAwMDexXHUwMDE29rb+aT2VW+W7IPetJsN+QWGTXGZcdTAwMGKXllxuXdlcdTAwMTKY9qVsW++8VOu/VWq3tcvyy8XT4e/YTFx1MDAwYtjrXHSj+DhcZuBdXHUwMDA19vU42TBtKml4XHUwMDEwXHUwMDBiMy1cdTAwMDBcdTAwMDZcdTAwMDA6XHUwMDAxo9xlkFM6+exf3LBGcK1cdTAwMTmdXHUwMDBiN+aj2qPe9plK72yfbVdcdTAwMDfb2TPZu3h7OJ6JaqWikkyx9qW0UidEte7VxqJa4Vx1MDAxMfA6rFwiKky1XFxz78NB3VBtsrok41Mtt4JZXHUwMDFkiOJcdTAwMDRVJuTZ/lVcdTAwMTmmXHUwMDE1I8wmzbRSXHQ2zX6cWahDTFvMnq1cctN+QWGTTFx1MDAwYpf2I0z71mxmSrJTem9cdTAwMWTfZVp7d+81c5OZgWmlJ5jWXFzwXHSfVkjgYKI2NJsoNKiFaZZRXHUwMDBltrlcdTAwMTXOsJeNZFmm4CPK0iXGi3O3TJezL75Pcnst/Vp6L5xtv8zGslx1MDAxYcxcdD5cdTAwMDWQlNZEJ8Oy7tXGYlnqXHUwMDExTiRh3OHQXG4lPGmDXHUwMDFhsaHYXHUwMDA09EjP4s1qYrUy3EWyjERqXGankkvCbcKBYlx1MDAxMGrONU1QqMMsW1hcdTAwMWaW/YK+Qixb+Fx1MDAxOZZVVT24f7k8Olx1MDAxN4V8u3JOXHUwMDFmSOm0Olx1MDAwM8tSz1xuxIZJlpVCelxmjC65IdpkXHUwMDAxwizuz1rg2eHhrFx1MDAwYjfC/PtcdTAwMTk7VlxcwabpwOe+nWp/N8T58+/zTo6W2/0+z2b3XHUwMDFlMnezUa0hMlx1MDAxMPBcdTAwMGKjktZKJnQ0615tXHUwMDFjqqXaXHUwMDAzdaBMu6hWSupcdEqoU5c2bJuaX5lsfLalkigqXHUwMDA0k64oXHUwMDEwU5FsSylcdTAwMDdcdTAwMTHUcr6EhqmCLYhJUrBDdJvLrFx1MDAwZt1+wWOTdFx1MDAwYpf2M3Tb8k9cdTAwMDePW9f8uFxcOb1cdTAwMTf19EupJmegW+ZpNPv4ZPhYcerRyVx1MDAxM9xcct0ujlx1MDAxMJmF6Vx1MDAxNlx1MDAwM1rg7ynmcmyZXHUwMDBlXHUwMDE575/HTtZSopfKtnv2+fm0VM3vNU+Pbvavu0+N26PnXHUwMDE52dbowIWGQVx0XlazJItM0SX3amOxrfE440JTV/hYcVxyr1xu5lSlXHLbpubXpews4WNOXHUwMDA108yV3cDCXHUwMDA3MX91xmjYUCnYXFw6M02s8Vx1MDAwMHjaqcisYlx1MDAxZOLaQv54fbh2OomFjmrzxz/Ctfa0fJRcdTAwMWJsXHUwMDFmXHUwMDFlNlx1MDAxZc/3XHUwMDFmy8WL/aPn3FxmXFwrPKLBdjOcg1x1MDAwYmHGQUJcdTAwMTPhKWFcdTAwMTgjRlx1MDAxOG3JhnSTXHUwMDA0iq3Fs6MspYZx7jLWhYrkXFzKhWJcdTAwMWHYeonpUbeHZn+XXHUwMDE1XG5v7drr1UDXqrnWcWUm0tUgnnRcdTAwMTRYXHUwMDBio5PVxCbk4rpXXHUwMDFii3Stp7Ti4MTCxlgxoU+UeIZLXHUwMDFkrU9cdTAwMWLmTc2vUNszMC9cdTAwMDW1kcxprYpAutLkMVxm5aBtiieeJZW4cIeoN7+9Pm7uXHUwMDE3nDZJvXBpP0K9pTPVL+Wf35rZ16avr/LF7YOH+NSbpsKTWjBcdGDAjbKCj5cqMEWAmlx1MDAxObNcdTAwMDbIWcNcdTAwMWJG4rFh38XBYmfxXGIzN8CkgjrNd2pcIulXgatsXHUwMDA0nVx1MDAwZkPmZN9M5eoxt3XGstnz5iMp+T2Vbs1U+8OtsmpKXHUwMDFhp1wilqik2Ne52lx1MDAxOOybZp5hVMC2aK6kkFx1MDAwMVx1MDAxN2qoUdp4XHUwMDFhkHSaRmF9XHUwMDEwXHUwMDEwMOxcdTAwMTDBeFx1MDAwNt+U/8ygU7n4XHUwMDA0XGZ7Q8FcdTAwMGVyh4tIZPmPJFx1MDAxOLNcYuY2J1P+k7h4h/hcdTAwMTdTeSuwoCBccqwm+35Ba65cdTAwMWPlyVx1MDAwYltcdTAwMDL5XHUwMDBl9rv77Waa5mq94n3Wv1x1MDAxMPXsnT9cdTAwMDP5alx1MDAwZuttLYCEi3y5kFx1MDAwMCVcdTAwMWLy/Sag2F2cfCnsXHUwMDFiIcZcciBcXE8+O7LgXHUwMDAxc4SxXHUwMDAxZf529rWckJ3a81Hz5viu3qxUsseN2tvKsq97tXHY13hCXHUwMDE4XHUwMDAzSlx1MDAwNSs2lMrJ3H+gVjKdfcE/ZsC+lFxua1x0XHUwMDAz42rDvvGVam+GY16mrFRCKaf2yFDKxOiYl1x1MDAxOVx1MDAwMt6KmEt7fpR/McF3Pfj3XHUwMDBiZnNlLv9cdTAwMDD/tk5Z/6S4975XLz5f9N+yp/ePhVJ8/mVkmOxcdTAwMDFQ4HR+4d+eZVpjLG3Dv6mkoWJ/Yf7FyLOy7vJ9TiPpV1x1MDAxOKWYXHUwMDEwcom+r1SD0uHBLiht3pb2y2YrrXu3K8u+7tXGYV/kTqpcdTAwMDVVgmO3XHUwMDAyLsadXyGlp6hk0zSKaE9TcH5cdTAwMDVcXJaWetP7Ylx1MDAxNp06mCWn2cBcdTAwMWRmzqp2znmU8lBMS1x1MDAwNPNIJ1x1MDAxY31eXHUwMDA2+1x1MDAxNtaFfb/gNVdG81x1MDAwZrDvXHUwMDEzqejq4+lZ6aRa0PqQPeU0XHUwMDEzM7Av91xmuL6AXHUwMDA0TvaV8LpcdTAwMTaCaj401TfcmyROXHUwMDFjLu77XHUwMDAyilx1MDAwYlx0e+PEXHUwMDBmXHUwMDFk3XXKKEKVUfNZ7/Oxb/awrC7J/nsnzbbo1k7tqlFpXHUwMDE1V5Z93auNxb7c42D3XHUwMDE4o4xUSlx1MDAwNeLGXHUwMDFmdVx1MDAwMsxcdTAwMTOcqmiFXCJg72rwfMN1XHUwMDA2XHUwMDFi6v1SpY5m8HyJXHUwMDFjVlx1MDAwNrh1x0ZcdTAwMDaeqZDGSmLnS1T8Ue7FLOD14N4vWM2V3vxcdTAwMDPc+7p3UHmgO+etZnr7rlm+OK1UqZqJeynVjFp35Fx1MDAxOfbbU9IwrviGe5NcdTAwMDaK/OLcayjsmuLhRjroY5HInE1pJFx1MDAwM+w3S0y5Kj7cZfzadWPPlFx1MDAwNv3yabtfqG7N1vBxmdTrXm0s6lx1MDAxNZ5cdTAwMDJrVUpOKfZ01HxcXKFAX7iQOlqhKKBcdTAwMDG4vYJQI1GvTFitNtxcdTAwMWKlUoX43IvhXCLK3bXvXCKcRTGqyLOWgKQl3TBjXHUwMDE5h7754zWh3i9IzZXt/Fx1MDAwM9R7xU7PL+9K212b3amVSjl+q7KNXHUwMDE5qJd6Womh01x1MDAwYpJcYpbgOFBcdTAwMThgZmXBRlx1MDAxN8O6sk22c5I4cbww9VxubIRccirrynaW0dVcdTAwMTJaXHUwMDAy+EuzzFx1MDAwNlWX5PZYP/X5201ja79zf1LQtcFMrTOWyrzu1cZkXoFNrIlcdTAwMWGmwmk2WT5APSM1m6JQ4DZLaS2jTFx0JczmwHdcdTAwMTaVOpnlwFx1MDAxN2DLaOM88FXR5zWMWOzSLNcw5IxZwevBvV+wmivd+Vx1MDAwN7j3vdyqllx1MDAwZlx1MDAwZs9cdTAwMWbPXlxiLylVL5ozXHUwMDEym3tcdTAwMDElPMlcdTAwMTjnhlmlXHTh48kh2Fx1MDAxMsnjSiopXHJcdTAwMDCJ2LTTSFx1MDAxNClOXHUwMDE3Jl+MXVx1MDAxMmacfVx1MDAwMTSPXHUwMDBlm8GWXG7wqshcXGGz+dg3d/dU289cdTAwMTTUjb1p79rMc3WQK15cdTAwMDfY9z/dX/vx4a1C9+Cluvso1Vx1MDAxM2f3uYw883dcdTAwMGXGf+Xv75e73fZsrK4w+j6lykNh0nFCdcPuu1x1MDAxMIPVwVx1MDAxM/ZcZpVcdTAwMDSY24JcdTAwMWVcdTAwMWE7oanI2Vx1MDAwNHztaE3djFBIza+qZ/FJnSshlHWnQGpcdTAwMTJ5jMxcdTAwMDBFrVx1MDAwNFlLOolLUTBcdTAwMTSmtMaaWbxDnL5cdTAwMDYzXHUwMDE0vqDJn56hUL6/f3spnFx1MDAxZFx1MDAxMf3UvT+/b1x1MDAxZd93dlx1MDAxY2G2XGJcdTAwMTZcdTAwMDdD3ZNcXFtUfzBcdTAwMGLVuFx1MDAwM1xyxFxmim/wjJFcdTAwMTBJXHUwMDAyiVxmXHUwMDFiXHUwMDBlX1x1MDAxY1x1MDAxOIqLc7i1ipmx4SnBXHUwMDAwXFykXHUwMDEzgOfGYJnZZU4r2s73mrWDekmmXHUwMDA31Ozk5e3Db1wiZlwiW3Qq9ZQmXHUwMDFkSZKte7UxyFx1MDAxNjbDo4JowGRcdTAwMGJKwSZcdTAwMTTKUE9cdTAwMTMpo1x1MDAxNWpTL5yaX6NK8alWU7CFjHVX2vPotljg63Cq52vbukzZXHUwMDBlMe06XHJV+ILTVmWowoNcdTAwMTWdi/716fYze+zXXo8ubnpbjk4+kdTLPGOIYWB/g2Fux0NtVmH+XHQjmIKise3phnpcdTAwMTNcdTAwMDSK8yRStlx1MDAxOKC5jlx1MDAwMJDobpRcdTAwMTTAXHUwMDFmK1mXSL1isNV9OLohb5bQt/PK7WWGtporS73u1caiXpxSwiXOdYPNXHSs50OhLDqxMlqhNtSbml+jLmZcYl1cdTAwMGLKXHUwMDAx+GzgR1x1MDAwM6ojXCJ7dShihFx1MDAxNVTMlXHxo9y7TmNcdTAwMTa+ILVVXHUwMDE5s1C8KKjDrVx1MDAxYzl+uXw5vKqcta5cdTAwMDZsPz73UjusXHUwMDA2ZpqbMPdcIqt6OIBcdTAwMDFsdL4h36Sh4nJh8kXnylxu5zw0XHUwMDE5PaXFMPxcdTAwMTBfZpOsI/t60r0/6ORqjVx1MDAwM0WL15n66WB1nV73amMxL04tXHUwMDAxNZJgsoaYXHUwMDE3uzt4ioE2RarThnpT8+vT1Vxm1KuItlx1MDAxNLuZuVQnXHUwMDFjd/60WjVcdTAwMTdcdTAwMDT7uq6f37tOw1x1MDAxN74gtVVcdTAwMTm+QG23c1PM0Vx1MDAxZSuq3W3VfdyiucJcZtyrPCxYlMzJvWBcdTAwMWTC0rglauP4fux6klhxvTj3XHUwMDAyiDNcdTAwMWIxvkXqSOtdK0K10MvMl95Jly8ytpy+3O6LXHUwMDFlK1x1MDAxNNv7md+zXHUwMDE1XG4vk33dq43FvjjNhChGnOxLhfZcdTAwMDS4TdFcbrVh39T8XHUwMDFhdTNcdTAwMTP7MqGVos6ER1x1MDAxM82+nFx0zlTypUrfz77rNIvhXHUwMDBiWluVWVxmN42Li+1bv2LLpvR2fbZvjnb5LOyrPYxBWuWKOlPwiD2qNCblbtg3lTRWlFx1MDAxN0+Z5lpHzU5T0Vx1MDAxOVwi1Fxu0GgxZ4f5+dj3+PU19/77uH60l99/L+X3XHUwMDFlhD48W1n2da82XHUwMDE2+zJPw4KUcEWdsSuxx7FcYilSoTbsm5pfoyrx2VdZajQhzFx1MDAxNTVSNDLhUVDD0XZaP893nWYzfEFqqzKb4UTky+k3q48u863+oP586Fx1MDAwZopsXHUwMDA27uVcdTAwMWVsLOVcXLm4l1x1MDAwZlx1MDAxZGMsXCImm6hzKmmkqC7MvVxmy0mtJE7zXcnIbCuL9DBnh/n5qLf+Kvduzyuvlzune6X9i/5D97KWXVnqda82XHUwMDBl9XLrYVx1MDAxMlx1MDAwZqHSRb3gXGZ7klxya1x1MDAwMzdh5+GziSpULT71Yq9ccs0tc1x1MDAxNytFz1x1MDAxMqPopjBcdTAwMWHsarku5LtO01x1MDAxOb5gtVWZzlDZ71x1MDAxNK5ZWr6d+jvPtJh7eqWX/TBO4O+N40RaXHUwMDEwXHUwMDAySKCxnlFZIyb6XHRcdTAwMTgqPcBcbuBkJqRgMlxc/0BcdTAwMDXH0VxynFD4m6DBXHUwMDEyifvRlXFcbl+NXHUwMDFk+SjD+snRj6w0cNTKvXt/ucjxOy5cdTAwMTWzKCpcdTAwMDaoXHUwMDE2zEZcZjWlkdmb2MhJmeCQmITqJLhccvZcdTAwMGZcdMOJZUTMXHUwMDA1J512Y9JcdTAwMTJcdTAwMTj9LTWSqOE/Pv/+P//pfHe0XHUwMDE4XHUwMDBmX1xyXHTw6PtCXHUwMDA2Q69f7vazsKWN1t3kpfitWsQrw09lsMrq3i+HXHUwMDA0XHUwMDA0Plx1MDAxN/lap918u1x1MDAxYm7vV9bE9Vx1MDAwM906rZ3fZVxuu/VOl976XHUwMDBm268ztJHnSnrUXHUwMDEyI4hV0lx1MDAwNvu/fFxiPyZwoydpXGI+Ni1cdTAwMDW+XHUwMDA2hpFWXHUwMDBiK3LqrL5dO+i/+K18X2ZcbupzaMks+JHAMHFjpZYymMg9Qlx1MDAxMEuiKycslstcdTAwMTG6zCSSPb/3+Fxcr+5k386zXHUwMDE5xdpXlcZRY7YyRSa0njJzOUlr3r3aXHUwMDE41nyaS+6hsa6pgM3BXCL1cd1cdTAwMDNcdTAwMDHzXHUwMDA0QPhcdTAwMTTd21xme0l9v/ZFzHpwXHUwMDFh/sAolClcdTAwMWQ+21x1MDAxYbpv0W3pXHUwMDE17Fx1MDAxZjF2vtPiZWpCyPBfn7EwXzDlioyFean7uXTreiv/VN/qkmz/oPh60IvP5+D/e1x1MDAwMCeMSimVhL1cdTAwMGZcdTAwMTE68bCFtpBcXGlNXHUwMDAzk+c3hL48SFm8l5BcIoBcdTAwMTewVa4ufjZcdTAwMWN3XHUwMDE4ZaVgQ5SlXHUwMDBljzk9lVx1MDAwN7unLz0utiqydfN2cNmFXHUwMDFmW1U6d682XHUwMDBlnTMmgYyZ4MpyUDNcdTAwMTNoY/ynQYjxrJR0iuZtpsekvl/3ZqjcXHUwMDEwXFyDX8Sd/f6siXS7gcbB80y+O8G3c/n6jJj5giVXZMRM7VKSonzunIluYbDV3M/0eod78blcdTAwMWNY2GOMcTK077HzUJjLhSZCXG6iZHBk2IbKl1x1MDAwNydcdTAwMTHHXHUwMDAyM3U10ESzsda6gTGsJHpcdTAwMGUrXHUwMDE11Fx1MDAwMusstbbyKD24vN0+6l3L18f662W5/Na6L6wsm7tXXHUwMDFih82t9EDtJDA50jnTk6pnrUelttGqt1x1MDAxOUWT+n7la8bncovly0I4R9aA4Vx1MDAxNdl9XHUwMDEzbWxcdTAwMDJcdTAwMWVV0lx1MDAxM9OXQedrM7PmXHUwMDBiolxclZk12Wvy/pzVoib98uX10UXlQO/Gp3OcXHUwMDFmiU1QJMHe+CpcXFxiojxGXHUwMDE1cFx1MDAwMWfMXG6yXHS1/1x1MDAwNKL0XHUwMDE3pnOGQ6FcdTAwMTV1OlxylLLoVivgMlx1MDAxMlx1MDAwNXu7RDbvnVx1MDAxNqvpk1x1MDAwMePNgU2fn5jdl+r26vrm7tXGYfNhqFx1MDAxZLRcdTAwMDaPQMxYfsWH6oFmWlwi2Fx1MDAxNNXbjLdJfbvydeOzOehcdTAwMTFcdTAwMGVhNFx1MDAxMXQuXCJtZiGp1nbO+Y0/yuZrNFx1MDAwNWc6T67IXHUwMDE0nNY52Sv1q6+NQm03U3ovVnW9M8v8V848qTR4XHUwMDA32JRamIloXHUwMDFmsrkmXHUwMDA00Fxco9lcdTAwMTlw3TdsvjxASWBMO7Yuosxdw0ZBuSPpnHCixVK79lcqubPr7XSeXHUwMDE1XHUwMDFmq8+3hUPzbP3SyrK5e7Vx2Jwx4VF0rK0kQjFq+ITuKelccufUReveZmJO6vu1b4ZIO1hWYDS7h9pRRqKNZkGpJIDB68fn6zNa51x1MDAwYqZcXJHROu9Fc1Z7MPZgL3+ZKZZzpzu7pzOM1qGEeTjRXHUwMDAxvFx1MDAwM02sJjRE6NJcdTAwMTPU4vRcdTAwMWSqXHRjm2j7T0BKRMBvXHUwMDA2QpdEgm3mbDlcdTAwMGWcXHUwMDEymaHDXHIh2PJniXTe5OWTi9fH7c47PfM7ue5ho3KhV5bO3auNQ+dcdTAwMTabKXGNXHTOhoXGSVx1MDAwZZ1zikly0Yq3XHUwMDE5wpP6dtXrxSdzXG74OdxcbreSXHUwMDA1puaE2FxccZzGlPigvO9n8/VcdTAwMTnW81x1MDAwNU+uyLCeNC1cdTAwMWNlM+cvXHUwMDBmz++5O917+X1zd96Nz+ZcXONcYlx1MDAwZoNcdTAwMTUwlmlNJ1x1MDAxYlx1MDAxZTLmXHUwMDExpcF1YFxcKVx1MDAxN6ZcYlwiPGFcYsZyidBWXHUwMDA1UGmEKVR5XHUwMDEya+2kJEpcblx1MDAxYbBeN1x1MDAxODNcdTAwMTfGJOCvK4blkUFcdTAwMGVcYibGRZfKYFx1MDAxMVxip8s8Sj/1XHUwMDFmOp2Xhl9Vu6W3/Ht2r1li3UT5XYPkXHUwMDA2pt8sllx1MDAxOOdcXG1cdTAwMWN+Z0xcct1xLob9XHUwMDEwmZrURf2FLoLtPfTWN2fp36l7M3U01tqOp6tcdTAwMDfT3CPP0lx1MDAxOeVcdTAwMDLWXHUwMDFiVM7l8PvMilx1MDAxMOL3y8xxKrtcdTAwMWH83m93XCLJ/Vx1MDAwYtqcJHfXVS2B3I8y9UNYZ7srffb74ea5sFXaik/uXFxzT3FthNJCWlx1MDAwMPzQJD6gZYtHqSBsRG/aSiRZy7r4+bjlw8KE8LhsvPcy2lx1MDAwMVx1MDAwN2uMULHM0/HCSYtXt663zMPtXHUwMDE1z1xc3O5cdTAwMTXuOjPOy1x1MDAxYlx1MDAwZcWJxiWQ0GBd/kJcdTAwMDTtXm1cZoLmOLlcdTAwMTIsWKJcdTAwMTW4bUHn609cdTAwMTlcdTAwMWHzNJ6aR6mTUZ7S2GraWDvsu1x1MDAxONapTVeJKH1cdTAwMWHEZ11gOMs5dZZwKlx1MDAxZe1Uc0WxiUvyR95JXHUwMDBit4t0/2lcdTAwMWR3/G653175rlx1MDAxMl+Qmot6w5e2XHUwMDA08u2Zy252/+61d5++ad5dse7vfHYnXHUwMDBlTmDfVUBcdCBOTlx1MDAwNeN2PIlcdTAwMDbse1x1MDAwZkescob5XHUwMDE11lGtaoxnMEpcdTAwMDdcdTAwMWZX2L51jVNofqCJxPNcZuY5x75N7u4z0V1XXHUwMDE1aCtcdTAwMTmbXHUwMDEwtpzeM1x1MDAwNqw4YoNyPvNQTVSjVq333Vxi0fTr/fns8rGfXG5N0FxmLX9cdChQb29f9p/rW3fPOc1cdTAwMTTvXHUwMDE2L/LPjtKUMFxuKGo8MFx1MDAwNNCRM9g8Y1xmXHUwMDA0XHUwMDE4WNZSXHUwMDAw4yhknWDyw1x1MDAwNlx1MDAwNJJcdTAwMDCBl/ggIMFww/HG7qKSyM6pmlx1MDAxYnSr5puUXHUwMDE5jVx1MDAwMti1XGbszCkogCVzNCjnXHUwMDFiXHUwMDE0+H5cdTAwMTTos6ve/f3DneyR3bt0vrZ3l804erB/XGZcblx1MDAxZoNcdTAwMDGw8z2mlVWaMEMxejdcdTAwMDZcdTAwMDSSMniZKtB0QlRcdTAwMTDf/1x1MDAwMoHEbHdOucRe7fCHOVx1MDAwZe3w0ozmSqD1qjDPbj2QYfluxHtcXLc8ssWUXHUwMDA1XHUwMDE2xmwoZ4upKSVoXFwyZoO7l4zVXHUwMDAwW26mOFx1MDAxNzieRZi58Fwi0Vx1MDAxNlPRUoyPsPyOvu9fXHUwMDEz37uMXHUwMDE2U5OK7Dcr7Zd4yXp327X3bdP/fXHr7zxcdTAwMWY9X1VPJFxyXHUwMDAzRbhcdTAwMTdcdTAwMWT2l1wi0lIhlFByrP7qXHUwMDAzJ7jHrDRcdTAwMDD/2LBcIlxcXHUwMDEzj/0ssIud5lZcdTAwMDOHXHUwMDEwh71ArcbRoSAyllx1MDAxYms43aCEXHUwMDFiJSqELFx1MDAwZVx1MDAxM1x1MDAxMkxcblwitVx1MDAwYiZUdOBfclx1MDAwNtufeHtcdTAwMWLBmFx1MDAxMiNuXHSjXHUwMDA0I1rPXHUwMDE1909cdTAwMTQlXCKFXHUwMDE4XHUwMDFm6bD8/ixKzNmIrn3N6uVOZvu1cV57OFx1MDAwNnPl6qzgiD+6IEJ5WHiDRziCXHUwMDFhM9FcdTAwMDdcdTAwMGKMXHUwMDA3eNlIuDXWXGJcdTAwMTZcdTAwMWUhPpzhXHUwMDAwtojA6lx1MDAxY0mMM5mXMU9g5IJcdTAwMTOjjVHy/6+j+lkwgi6MXHUwMDExXHUwMDE4RzY6om90tCmhpWLA6YlcdTAwMDcgXHUwMDA00ywwed5cdTAwMDFcdTAwMTJ4XHUwMDFj8fMgXHUwMDExKcZcdTAwMWavTlxu8FqCxHXtvvv6sH+fvyX1garlXHUwMDFltUyreCCBKCBcdTAwMDRcdTAwMThcZowqwdV4zj+CXHUwMDA0jp2wXG5cdTAwMTBfOXrr4N2VhFmGdoiyRLncXHKKM8hcdTAwMDF/lVx1MDAwNLThYk062v5cdTAwMDBGsIUxgmtcblRcdTAwMTi0+EZcdTAwMTihI1x1MDAwZlx1MDAwMSmxmG+VfJBSXHUwMDAw/9Jp7lx1MDAwNoffXHUwMDFkQddPYoRbivGRXHUwMDBlXHTwWmJcdTAwMDS76Fx1MDAxM/qYPa1cdTAwMGbuzk+Y/3BCeDtcdTAwMTPb14DbosB1XHUwMDA0nKBcdTAwMTPBSYnttcCOUEpa7JA32s6Rs0HAWcFcdTAwMWF8wTBcdTAwMGI14ICOLFx0uMVEUWVcdTAwMTkzmM8ysj43IDFcdTAwMGVcdTAwMTJ8YZBcdTAwMTBSY4GPM83IRDpcdTAwMWKKgoktTOJN9Fx1MDAwNdfgxE/DXGKp6M+HJKKFXHUwMDE4XHUwMDFm6ZD8riVG3Fa6XHUwMDBm5uj1rV1672+J/vmWKD3F6o3PXHUwMDE1XHUwMDFlQFx1MDAxMFx1MDAwYjyiKMNObVx1MDAxM3aE9qzRnIClgdlcdTAwMGXhU0wsPuSAXHUwMDFlWoCxQVxmcVxy0Vx1MDAxOPbUVsxSqeFccmZdXHUwMDBlNH5cdTAwMDAjxOJcdTAwMDFcdIGFIcqZ71x1MDAxYt1cdTAwMDbAKlxuNjafr1x1MDAwZuY0jFx1MDAxMFRcdTAwMDVuS1x1MDAxOCOwX9To9Z+zI6KEXHUwMDE4XHUwMDFmIfFdS4jgN0fl9E2ttGd8dZxhz1x1MDAwNdWuO2ZXueNcdTAwMTFcdTAwMTQ7dKIzXHUwMDA21tZcYvL/XCKEhJtCXHTlVpnAmdVcYiFcZpZcdTAwMDWAlUHB2lx1MDAxMEw44lx1MDAxMfDlYIvAeyi11lx1MDAxYWI2XHUwMDEwXHUwMDExXHUwMDA1XHUwMDExcvF4XHUwMDA0mPZUcGfMUkaXXHUwMDE241x1MDAwMTW2V0xcdTAwMWUjZHBailx1MDAwMyPgXHUwMDFkcyVEJIxcdTAwMTFRYoyPsFx1MDAwMK8lSNS650dH1+VzYs3FQfOy/v5C2zxcZlx1MDAxMrVG+andqo3jhJQ4O1x1MDAxNlx1MDAwZrZcdTAwMTmxOCN8XGYnwMLwhkNIOew2tcKRhmyFJ1x1MDAxNJhcdTAwMWGGay2JUS5vQ61J5mSt3e8vN1x1MDAxOaJC1OLIILU2xlpXSiXlU3IqkVx1MDAxZmBrxVxcPsbn+mbKRr5921Hb9uHkbecxe9l6PXg6V9vvs1xyuYObXHUwMDFlqIJcYlx1MDAwM4/EcrlcdTAwMTmAZ5qB7lxcbUixXFz5RWiBXHUwMDEzXHUwMDA1XHUwMDFho1x1MDAxNKxrnH0pUV+plWaeXCJcdTAwMWFcdTAwMDfEa27cQ+4oW1x1MDAxM636XHUwMDAxutVupXKlXHUwMDE4YaSFS1x1MDAxYa73XHUwMDE5LiVcdTAwMWP+XHUwMDFmJVxyMIqJhokz65dcdTAwMDKuXGLni2ckb7Vb9cbdP62dVrnS9FPt7j+t7UZcdTAwMGb//k/LL1fv/2m1P5J5nKlIgbjTT6csf8F+rpTlj2tPfV566s+Vp/DCU6HrXkJcdTAwMGXTVqF78FLdfZTqibP7XFxGnvk7XHUwMDA3sXKYXHUwMDE4ULRi4OOAkac0IaFcdTAwMDE82NGfcqPwxCVQXHUwMDFl/lx1MDAxN2lcdTAwMTTDskQwjP6WJTrOXHUwMDE0XHUwMDEwrVx1MDAxODFcdTAwMWOLJyzH6PVcYtBWXHUwMDAxeujE8z9cdD07XHUwMDBi8zl4ZeCwSff8nfBcdTAwMTD5z6aAwjBsypp4WVwi1cGch1x1MDAxMFx1MDAxZSkwXHUwMDFm1c+fKkTLMT7SXHUwMDBlXHUwMDExjm/rXHUwMDA3Ns//uGl/ulx1MDAxMJRb1fLh4fnj2Vx1MDAwYuElpepFc0ZcdTAwMDJmz6+ndm0oh+1upVx1MDAxMbSHQLZf/dpcdF7/2OWnRmI5vGhcdTAwMGbT2ULrXHUwMDFjScyPp1TtXFyX+NH5SbrZ3slXybGRul5qxcIt8DuwhSXuhlx1MDAwMGlcdTAwMGa0lP2ALWo8JVx1MDAwNfatXHUwMDA2OSPKXHUwMDAxXFxcbnBccnaaXHUwMDFizMSm1tUvXHUwMDA1flx1MDAwMuRcdTAwMDFcdTAwMGK6MJy+XHUwMDAxrSjQyi1cZlpgXHUwMDBlMVAuXHUwMDEzxifc60jQXHUwMDAyz4Ur/lx1MDAxZLXU1ujR3Vxug1x1MDAxNrj9K3DMXHUwMDExKcP4SE+Kb3zAWq28yzdbXHUwMDFkPJ2+5reqRbM9ODvOvl7ukjhuVJpcdTAwMWLp4fBcYqVcYubIkVx020ZcdTAwMWHPMqybZ1ipQZjjoENcdTAwMGKPWGrBVFx1MDAxN5JTV6VcdTAwMDZfrajlKsHCLENcdTAwMDM5XHUwMDA3frXSXHUwMDE5hYieXHUwMDE5iFap5TTRnErswiUtXHUwMDE4vlPCk0qY4PCqOZyorUFcdTAwMTe3OJX3a41H00tJzzg9pUCTtlx1MDAxNS3amHolS/B9MtUnnXtSXHUwMDBmd1f6Kn3ev3h4a5w7bFxiXHUwMDE3PFjmWc2JMZhFwrVcdTAwMWSPs1x1MDAxOFx0Vlx1MDAxZtGaKzxKtjacUEVcdTAwMTWYWNhVQ0V2UtjAQyQ87MWHXHUwMDA3sLilXHUwMDFha3xcdTAwMTBsplx1MDAxNlnOSYU0WKKXaCXXXHUwMDEwXHUwMDFmwFx1MDAxNZ5qXHUwMDFmKKBcdTAwMWJcdTAwMWSU9Zl7qX1qk11TXFxwXsFcdTAwMTLwQFxc3Jw8+1x1MDAwZlfng+d+7fbh9f13QznqucJnnlx1MDAxMrufwtZcctGAhruZM4WD1IdxV1xyRkW4rlx1MDAxM/BcdTAwMDJYXGapXHUwMDAzfFrOmXDgXHUwMDAx8axQXHUwMDA0XHUwMDFjXHUwMDEyZTHRW1x1MDAwNVxc31xyPozjw/7CXlx1MDAwNXa+1mD7uVwiIVwiums5+vjgjfCE6z+5xfKoKaFZZahcbiRp/1g9V6RcdTAwMTjjI1x1MDAxZJbgn3Us5jz11LX3+87zVlmfXHUwMDE2iuL5Vu63no5itV7HXHUwMDE5p4xrraSWQlxiSsetXHUwMDA2QVxm2Fx1MDAwNFx1MDAxNO+RgbvDwrlcdTAwMTHgTniCWzxrskIzJVx1MDAxZM1iVqz50irBwkF8s0EpbrUlTrOBR7dZkkKDzUCSTp6E7Vx1MDAwNo1cdTAwMWHRRkj/NWZcdTAwMWPNVYTx12o47/ndVM2vNmp+L/Vy36jep/5pne1dZj7brMC/++1Uo1x1MDAwNSrWbLoti9Vo4zbVtpj7OpfRW6bVubFHV1x1MDAwZq1ms/PYI1cl2ns/i1x1MDAwMyzUaM9cYk2JXHUwMDAxd4NxO3nqS8E8UWB/MsmlpVx1MDAwZXeE47mwtJrCu9zeyFx1MDAwNlZcImHlMD6sMJxNjY1iXHUwMDFjsFwiaWRbXHRcdTAwMDZmpbRKzpcwMc2u4LjjU3BFXG4yX1x1MDAwNehcdTAwMTiu/FGmXlxuz0DLrVpIXHUwMDAx/5x8pkDx/NbHUfB3I8w3+S5cdFxc71x1MDAxMpDmrn4js9v1SrdZub4l+9nDs9rBflx1MDAxY09cdTAwMDd4yMNcdTAwMWFcIiFcdTAwMTlcYkiwMHCYtVx1MDAwNX6Q5ZiCZ1x1MDAxOSOUOJrRKutJXHUwMDBlXHUwMDA0KSWWXGZcdTAwMDVcdTAwMGLaXHUwMDAzjaGtx1x1MDAwMYQss1xiWcquViXZKkFPYWFHR1xuLqk7hcuGksJHhWRSKk5s4nNjLDdkmp1jOcjNzyeAp6PFXHUwMDE4XHUwMDFmYVx1MDAwMV5LP+ekUt99vLT1wVnmvfSuTq6fXHUwMDA2t0exQFx1MDAwMuxcckuwTelHP/GJcFxieC9cdTAwMWVcdTAwMWHJ4F5Lg72rXHUwMDFkKEE9cGmpXHUwMDA2X1x1MDAwN6dWuLrHM/AmLeGcXHUwMDBla1H0XHUwMDA2I6Iw4njxPE8wXHQ1aKZcdTAwMDMjpoxX11x1MDAwMiRAJT8r0lxublx1MDAwMqUpYYzAQ7mVwIhcYiFcdTAwMWW+XHUwMDFhkt+1hFxibp+ah6XyoNW6V+TxdXe/SToyrlx1MDAxZCFxXGIpNqJcdTAwMTbBtK6P7G+cXHUwMDBlXHQ3hVx1MDAxMXBrjGtcdTAwMDSdwlx1MDAxM1iD91x1MDAxN+hLSenKU+VcdTAwMDRMJKs0XHUwMDA3NVx1MDAwMsbaXHUwMDFjqERixElcdTAwMDK5Y5iXinmALtcmOlx1MDAxNdxaMFx1MDAxNsfswKRQXHUwMDAy/ptcdTAwMTIxtUJcdTAwMGKxXHUwMDEyKFx1MDAxMSXI+Fxii/BawsTTw5HU3dZjd+/Znt9cdTAwMWNcdTAwMTdfSTfrqFx1MDAxM3FZXHUwMDEygJJEUmJcYlx1MDAwNrkmemszKTymqMLuXHUwMDAzVErhOmZlXHUwMDFlIK3CbGlcdTAwMTBPZ3dtyqmH3fA13F54XHUwMDFmZZtsrSiYOF08W1x1MDAwYm6vUMowXHUwMDE3TOhIh0MrsFx1MDAxYYlcdTAwMGUobEIoIcFOn1x1MDAxMv+wkq1Cn7x0tFx1MDAxY3+8XHUwMDFjXHUwMDEy4bWEiZN3s3dy2Wv2t85cdTAwMWYvuzm/S84qsdrkWax6IcOuuJhxNVx1MDAxOf403Fx1MDAxM8QqJeBcdTAwMTZRqVxcUVx0TFVcdTAwMDdcbsOEXHUwMDBljolvYZjggNNcdTAwMTT/YNKxNcFcdJdcdTAwMWKYXHUwMDE4h4mzhWGCcy6pdqOEij5+NVpQab4hLIFTrKckbdihq7tcbjBcdTAwMTElx/hcYkvwWqLExTl9q7ZcdTAwMGZcdTAwMDeFI59cdTAwMTdN9f76Mf/uOH51oYTwNDNcZtP0maGT5SqaeFxubFirh4VzjiSN4WBsjv1acfC1sa5umlx1MDAwMqt+sasrdiFD5NmARFx1MDAwNEhcdTAwMTRcdTAwMTNwOVx1MDAxOJdcdTAwMWPMYldcdTAwMDVd+OR2XHUwMDE0mMDPWZL4WYrC3lJTQELx4MCPXHUwMDFmXHUwMDA1XHS3XHUwMDE4XHUwMDBmX1x1MDAwZUnwqoPEv/6s6Fe50yn24Z5/Slx1MDAxMchmozZxcvTxXFzf7/z6PFx1MDAxOVx1MDAxYT6Vb9f8j7K8ifv/67nhv2TDqvhcdTAwMWb14eOPXHUwMDE0/9m6/KDZb1x1MDAxNEGYqnhcdTAwMGUzlGdcXN9QdFFP/T9P/fv/XHUwMDAwVc9AWyJ9OLMNHCmanagerSBRmanagerSNRmanagerFARmanagerNMOmanagerMDRmanagerNHC bundleSBR bundleSNR bundleFAR bundleNMO bundleMDR bundleOLMNHCmanagerSBRmanagerSNRmanagerFARmanagerNMOmanagerMDRmanagerNHC bundleSBR bundleSNR bundleFAR bundleNMO bundleMDR bundleWAO BundleWAOOperatorOperandsOperandsWAOConfigEnable orDisableeachoperandCurrent Medik8s 5.8Medik8s 5.9User decides which RHWA Operator to installUser installs WAO and decides which operand to enable \ No newline at end of file From 8644ae739a36fc6f5a33850e887df67c1ca4cdb4 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 19:11:46 -0500 Subject: [PATCH 03/10] Rough draft --- docs/Overview.md | 131 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 96 insertions(+), 35 deletions(-) diff --git a/docs/Overview.md b/docs/Overview.md index 9cfcc55..1214d7a 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -7,10 +7,9 @@ the existing behavior. This document explains the design and how to work with it. ## Goal -Would like to see Medik8s 5.9 published as a single operator. +Medik8s 5.9 published as a single operator. Need to work over next month to learn if there are any blockers that would prevent this. - ## High Level Understanding ![WAO High Level](images/WAO_High_Level.svg) @@ -81,38 +80,101 @@ spec: machineDeletionRemediation: enabled: false ``` -* For testing there is a script [hack/enable_all.sh](../hack/enable_all.sh) -* You can also tweak the memory and cpu limits and requests for each operand, [config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml](../config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml) - - - - -## Goal - -Goal - -WAO branches and links - -Gists of example of commands - -Walk through manual update of an operand - -Walk through helper with operands.yaml - -Walk through bundle formation with overlays -Walk through resource changes for limits -Walk through objects yaml - - -Walk through next steps -Test -Upgrade scenario. - -Clean up -Gap no webhooks until enable -Objects yaml in golang - -Integration with Virt auto pilot +- For testing there is a script [hack/enable_all.sh](../hack/enable_all.sh) +- You can also tweak the memory and cpu limits and requests for each operand, [config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml](../config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml) + - See [docs/configuring_resources.md](docs/configuring_resources.md) for more info + + +## Concerns for an operand developer ... +### How do we sync operand changes to WAO? +When we talk about WAO managing an operand we can think of 4 concerns for the operand: +- Operand Image(s): WAO needs to know what images to use for the operand +- CRDs: WAO needs to have the CRD definitions which match the operand's image +- RBAC: WAO needs to have the RBAC resources which match the operand's image +- Runtime manifests: WAO needs to know what runtime manifests to use for the operand's image + +The above is addressed by: +1. Copying the operand's `config` directory into `config/operands//config` + - Each operand has it's own directory at `config/operands//` + - Under this directory are 2 subdirectories + - `config/` - contains the operand's configuration files, a direct copy from the operand repo + - `overlays/` - contains kustomization.yaml files that assemble a manifest from the above config files + - `overlays/rbac` - + - `overlays/runtime` + - The files under `config` are referenced for building: + - CRDs: [config/operands/crd/kustomization.yaml](config/operands/crd/kustomization.yaml) + - RBAC + Runtime manifests + - Example of NHC, similar pattern for each operand: + - RBAC: [config/operands/nhc/overlays/rbac/kustomization.yaml](config/operands/nhc/overlays/rbac/kustomization.yaml) + - Runtime manifests: [config/operands/nhc/overlays/runtime/kustomization.yaml](config/operands/nhc/overlays/runtime/kustomization.yaml) + - which generates the runtime manifests to create when the operand is enabled + - [internal/operands/manifests/nhc/objects.yaml](internal/operands/manifests/nhc/objects.yaml) + - The above objects.yaml is embedded in the WAO manager process and contains the Deployment, Services, ConfigMaps, and Webhooks of the Operand to manage. + +2. Specify the operand image(s) by editing [`config/manager/manager.yaml`](config/manager/manager.yaml) + ```yaml + env: + - name: OPERATOR_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: RELATED_IMAGE_NHC_MANAGER + value: quay.io/medik8s/node-healthcheck-operator:wao + - name: RELATED_IMAGE_NHC_CONSOLE + value: quay.io/medik8s/node-remediation-console:wao + - name: RELATED_IMAGE_SNR_MANAGER + value: quay.io/medik8s/self-node-remediation-operator:wao + - name: RELATED_IMAGE_SBR_MANAGER + value: quay.io/medik8s/storage-based-remediation-operator:wao + - name: RELATED_IMAGE_SBR_AGENT + value: quay.io/medik8s/storage-based-remediation-agent:wao + - name: RELATED_IMAGE_FAR_MANAGER + value: quay.io/medik8s/fence-agents-remediation-operator:wao + - name: RELATED_IMAGE_MDR_MANAGER + value: quay.io/medik8s/machine-deletion-remediation-operator:wao + - name: RELATED_IMAGE_NMO_MANAGER + value: quay.io/medik8s/node-maintenance-operator:wao + ``` +#### Scripted way to sync operands into WAO - make update-operand-manifests +We have a make target that can help to sync the operand manifests and set it's images. +1. Edit [operands.yaml](operands.yaml) + - Ensure you have a local checkout of the operand repo at the branch you want +2. Run `make update-operand-manifests` + - This will copy the operand's config files and update the images to the ones specified in [operands.yaml](operands.yaml) + - You can also run `make update--manifests` to update only a specific operand's manifests + +See [docs/updating-operand-manifests.md](docs/updating-operand-manifests.md) for more information. + +### How do I deploy today? +You can deploy using the `dev-olm-deploy` target as the other operators +- ```make dev-olm-deploy``` + +Or you can walk through a: `make docker-build docker-push bundle-build bundle-push catalog-build catalog-push` +- See [hack/wao-make_and_deploy.sh](hack/wao-make_and_deploy.sh) as a script to walk through above. +- and for cleanup see [hack/wao-purge.sh](hack/wao-purge.sh) +- There are also OLMv1 variants if you are interested + - [hack/olmv1-make-and-deploy-wao.sh](hack/olmv1-make-and-deploy-wao.sh) + - [hack/olmv1-purge-wao.sh](hack/olmv1-purge-wao.sh) + +Before you deploy WAO to your cluster you need to uninstall any existing medik8s operators. +After you have WAO running on your cluster you need to enable the components you want to use. +- See [hack/enable_all.sh](hack/enable_all.sh) + +## Upgrade Scenario +The upgrade scenario is still being tested, the expected workflow is: +1. Put Cluster into maintenance window +2. Ensure nothing is in flight for remediations +3. Pause medik8s operations +4. Backup CRs (kept as a fallback for manual recovery in case of an unexpected issue) +5. Uninstall existing medik8s operators (keep CRDs/CRs) +6. Install WAO +7. Enable components that match was installed/enabled on cluster prior +8. Cluster exists maintenance window + + +- First phase of work is walking through the manual upgrade flow and then working towards this in a CLI. +- Would like to have a declarative way to run this flow for shops that have fully adopted GitOps. +- Need to learn more of how Virt AutoPilot is integrating medik8s, this is likely where we can smooth the upgrade experience for many virt users. ## Current Status ### Testing @@ -124,7 +186,6 @@ Integration with Virt auto pilot WAO is setup to track the `wao` branch of each operand's repo. The `wao` branch started from each operand's`medik8s5.8` branch as of 11/2/2026 - ## Next Steps 1. Testing: Work to understand if there are blockers that prevent the goal of WAO for medik8s 5.9 2. Iron out the upgrade scenario From 7d5037f195feb6100f013dc009f766264121608a Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 19:19:47 -0500 Subject: [PATCH 04/10] Updating links Signed-off-by: John Matthews --- docs/Overview.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/Overview.md b/docs/Overview.md index 1214d7a..65ee12b 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -33,8 +33,8 @@ WAO is **not** responsible for the business logic of each operand. ### Recap of responsbilitlities: OLM - WAO - Operand - **OLM** - On install of WAO - - Deploys all CRDs - - Deploys all RBAC + - Deploys all CRDs (including all operand CRDs and WAO's CRDs) + - Deploys all RBAC (including all operand RBAC and WAO's RBAC)ß - Deploys the WAO operator - **WAO** - On enable @@ -82,7 +82,7 @@ spec: ``` - For testing there is a script [hack/enable_all.sh](../hack/enable_all.sh) - You can also tweak the memory and cpu limits and requests for each operand, [config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml](../config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml) - - See [docs/configuring_resources.md](docs/configuring_resources.md) for more info + - See [docs/configuring-operand-resources.md](docs/configuring-operand-resources.md) for more info ## Concerns for an operand developer ... @@ -91,7 +91,7 @@ When we talk about WAO managing an operand we can think of 4 concerns for the op - Operand Image(s): WAO needs to know what images to use for the operand - CRDs: WAO needs to have the CRD definitions which match the operand's image - RBAC: WAO needs to have the RBAC resources which match the operand's image -- Runtime manifests: WAO needs to know what runtime manifests to use for the operand's image +- Runtime manifests: WAO needs to know what runtime manifests to deploy for the operand The above is addressed by: 1. Copying the operand's `config` directory into `config/operands//config` @@ -102,16 +102,16 @@ The above is addressed by: - `overlays/rbac` - - `overlays/runtime` - The files under `config` are referenced for building: - - CRDs: [config/operands/crd/kustomization.yaml](config/operands/crd/kustomization.yaml) + - CRDs: [config/operands/crd/kustomization.yaml](/config/operands/crd/kustomization.yaml) - RBAC + Runtime manifests - Example of NHC, similar pattern for each operand: - - RBAC: [config/operands/nhc/overlays/rbac/kustomization.yaml](config/operands/nhc/overlays/rbac/kustomization.yaml) - - Runtime manifests: [config/operands/nhc/overlays/runtime/kustomization.yaml](config/operands/nhc/overlays/runtime/kustomization.yaml) + - RBAC: [config/operands/nhc/overlays/rbac/kustomization.yaml](/config/operands/nhc/overlays/rbac/kustomization.yaml) + - Runtime manifests: [config/operands/nhc/overlays/runtime/kustomization.yaml](/config/operands/nhc/overlays/runtime/kustomization.yaml) - which generates the runtime manifests to create when the operand is enabled - - [internal/operands/manifests/nhc/objects.yaml](internal/operands/manifests/nhc/objects.yaml) + - [internal/operands/manifests/nhc/objects.yaml](/internal/operands/manifests/nhc/objects.yaml) - The above objects.yaml is embedded in the WAO manager process and contains the Deployment, Services, ConfigMaps, and Webhooks of the Operand to manage. -2. Specify the operand image(s) by editing [`config/manager/manager.yaml`](config/manager/manager.yaml) +2. Specify the operand image(s) by editing [`config/manager/manager.yaml`](/config/manager/manager.yaml) ```yaml env: - name: OPERATOR_NAMESPACE @@ -140,25 +140,25 @@ We have a make target that can help to sync the operand manifests and set it's i 1. Edit [operands.yaml](operands.yaml) - Ensure you have a local checkout of the operand repo at the branch you want 2. Run `make update-operand-manifests` - - This will copy the operand's config files and update the images to the ones specified in [operands.yaml](operands.yaml) + - This will copy the operand's config files and update the images to the ones specified in [operands.yaml](/operands.yaml) - You can also run `make update--manifests` to update only a specific operand's manifests -See [docs/updating-operand-manifests.md](docs/updating-operand-manifests.md) for more information. +See [docs/updating-operand-manifests.md](/docs/updating-operand-manifests.md) for more information. ### How do I deploy today? You can deploy using the `dev-olm-deploy` target as the other operators - ```make dev-olm-deploy``` Or you can walk through a: `make docker-build docker-push bundle-build bundle-push catalog-build catalog-push` -- See [hack/wao-make_and_deploy.sh](hack/wao-make_and_deploy.sh) as a script to walk through above. -- and for cleanup see [hack/wao-purge.sh](hack/wao-purge.sh) +- See [hack/wao-make_and_deploy.sh](/hack/wao-make_and_deploy.sh) as a script to walk through above. +- and for cleanup see [hack/wao-purge.sh](/hack/wao-purge.sh) - There are also OLMv1 variants if you are interested - - [hack/olmv1-make-and-deploy-wao.sh](hack/olmv1-make-and-deploy-wao.sh) - - [hack/olmv1-purge-wao.sh](hack/olmv1-purge-wao.sh) + - [hack/olmv1-make-and-deploy-wao.sh](/hack/olmv1-make-and-deploy-wao.sh) + - [hack/olmv1-purge-wao.sh](/hack/olmv1-purge-wao.sh) Before you deploy WAO to your cluster you need to uninstall any existing medik8s operators. After you have WAO running on your cluster you need to enable the components you want to use. -- See [hack/enable_all.sh](hack/enable_all.sh) +- See [hack/enable_all.sh](/hack/enable_all.sh) ## Upgrade Scenario The upgrade scenario is still being tested, the expected workflow is: From 33b5103d00a4c54ddfc3692502bc922a57a3e1e1 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 19:23:05 -0500 Subject: [PATCH 05/10] Fix link Signed-off-by: John Matthews --- docs/Overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Overview.md b/docs/Overview.md index 65ee12b..0451251 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -82,7 +82,7 @@ spec: ``` - For testing there is a script [hack/enable_all.sh](../hack/enable_all.sh) - You can also tweak the memory and cpu limits and requests for each operand, [config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml](../config/samples/workloadavailability_v1alpha1_workloadavailabilityconfig_resources.yaml) - - See [docs/configuring-operand-resources.md](docs/configuring-operand-resources.md) for more info + - See [docs/configuring-operand-resources.md](/docs/configuring-operand-resources.md) for more info ## Concerns for an operand developer ... From 4a4f8b330edd6f0fd1c7cc00034b3a2984e1175d Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 19:25:54 -0500 Subject: [PATCH 06/10] Tweak Signed-off-by: John Matthews --- docs/Overview.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Overview.md b/docs/Overview.md index 0451251..f6f842c 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -138,10 +138,10 @@ The above is addressed by: #### Scripted way to sync operands into WAO - make update-operand-manifests We have a make target that can help to sync the operand manifests and set it's images. 1. Edit [operands.yaml](operands.yaml) - - Ensure you have a local checkout of the operand repo at the branch you want + - Ensure you have a local checkout of the operand repo at the branch you want 2. Run `make update-operand-manifests` - - This will copy the operand's config files and update the images to the ones specified in [operands.yaml](/operands.yaml) - - You can also run `make update--manifests` to update only a specific operand's manifests + - This will copy the operand's config files and update the images to the ones specified in [operands.yaml](/operands.yaml) + - You can also run `make update--manifests` to update only a specific operand's manifests See [docs/updating-operand-manifests.md](/docs/updating-operand-manifests.md) for more information. From 6989595f913160f0e737d41305be0d410e401bf7 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 19:27:49 -0500 Subject: [PATCH 07/10] Fixed broken link Signed-off-by: John Matthews --- docs/Overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Overview.md b/docs/Overview.md index f6f842c..811d959 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -137,7 +137,7 @@ The above is addressed by: ``` #### Scripted way to sync operands into WAO - make update-operand-manifests We have a make target that can help to sync the operand manifests and set it's images. -1. Edit [operands.yaml](operands.yaml) +1. Edit [operands.yaml](/operands.yaml) - Ensure you have a local checkout of the operand repo at the branch you want 2. Run `make update-operand-manifests` - This will copy the operand's config files and update the images to the ones specified in [operands.yaml](/operands.yaml) From 88f6862e552a660c56383d14f3bbdbf28e3395ea Mon Sep 17 00:00:00 2001 From: John Matthews Date: Tue, 3 Nov 2026 19:37:21 -0500 Subject: [PATCH 08/10] Addressing coderabbit feedback Signed-off-by: John Matthews --- docs/Overview.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/Overview.md b/docs/Overview.md index 811d959..51d32e5 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -19,22 +19,22 @@ WAO treats the existing medik8s operators as **operands** which are managed by t WAO is responsible for: - **CRDs** - Collecting all of the operand CRDs and packaging so OLM will deploy. - - This means that all of the operand CRDs are installed into the cluster at time of WAO installtion. + - This means that all of the operand CRDs are installed into the cluster at time of WAO installation. - CRDs are NOT gated on the WAO operator's `enabled` state. - **RBAC** - Collecting all of the operand RBAC related files and packaging in the WAO bundle so OLM will deploy on install of WAO. - **Runtime manifests** - Understanding the runtime manifests for each operand and deploying them when the operand is `enabled`. - and...cleaning them up when the operand is `disabled`. WAO is **not** responsible for the business logic of each operand. -- Each operand will continue to be developed in it's own repository. +- Each operand will continue to be developed in its own repository. - Operand changes are intended to be minimal to none - - As of today we have not made any changes to any of the operands, yet it's likely will want to refactor the webhooks soon to allow them to be brought into the main WAO manager process. + - As of today we have not made any changes to any of the operands, yet its likely will want to refactor the webhooks soon to allow them to be brought into the main WAO manager process. -### Recap of responsbilitlities: OLM - WAO - Operand +### Recap of responsibilities: OLM - WAO - Operand - **OLM** - On install of WAO - Deploys all CRDs (including all operand CRDs and WAO's CRDs) - - Deploys all RBAC (including all operand RBAC and WAO's RBAC)ß + - Deploys all RBAC (including all operand RBAC and WAO's RBAC) - Deploys the WAO operator - **WAO** - On enable @@ -42,7 +42,7 @@ WAO is **not** responsible for the business logic of each operand. - On disable - Removes the operand's runtime manifests - **Operand** - - Responsible for reconciling it's CRs + - Responsible for reconciling its CRs - Currently runs their own webhook server, we likely will want to move this to run in the main WAO manager process. ### What does install of WAO look like? @@ -136,7 +136,7 @@ The above is addressed by: value: quay.io/medik8s/node-maintenance-operator:wao ``` #### Scripted way to sync operands into WAO - make update-operand-manifests -We have a make target that can help to sync the operand manifests and set it's images. +We have a make target that can help to sync the operand manifests and set its images. 1. Edit [operands.yaml](/operands.yaml) - Ensure you have a local checkout of the operand repo at the branch you want 2. Run `make update-operand-manifests` @@ -189,7 +189,7 @@ The `wao` branch started from each operand's`medik8s5.8` branch as of 11/2/2026 ## Next Steps 1. Testing: Work to understand if there are blockers that prevent the goal of WAO for medik8s 5.9 2. Iron out the upgrade scenario -2. Additional cleanup in WAO code: +3. Additional cleanup in WAO code: - Implement `bundle-k8s`, we currently only create a bundle for `ocp` - - Consider: Bring webhooks to run in the WAO process so they are availabile and running even when a component is disabled. - - Consider: Rewrite the internal/operands/manifests//objects.yaml in Go and stop trying to sync from the operand repo. + - Consider: Bring webhooks to run in the WAO process so they are available and running even when a component is disabled. + - Consider: Rewrite the internal/operands/manifests/``/objects.yaml in Go and stop trying to sync from the operand repo. From 16e1866a9ea406b13533964046b9d645316865c9 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Wed, 4 Nov 2026 17:37:06 -0500 Subject: [PATCH 09/10] Apply suggestion from @weshayutin Co-authored-by: Wesley Hayutin <138787+weshayutin@users.noreply.github.com> --- docs/Overview.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/Overview.md b/docs/Overview.md index 51d32e5..da45045 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -193,3 +193,5 @@ The `wao` branch started from each operand's`medik8s5.8` branch as of 11/2/2026 - Implement `bundle-k8s`, we currently only create a bundle for `ocp` - Consider: Bring webhooks to run in the WAO process so they are available and running even when a component is disabled. - Consider: Rewrite the internal/operands/manifests/``/objects.yaml in Go and stop trying to sync from the operand repo. + - Condier: In future releases refactoring the code layout to better conform to kubebuilder https://book.kubebuilder.io/reference/project-config.html + From 4cb5039a7eb7884f984a772c9dd5001d15325323 Mon Sep 17 00:00:00 2001 From: John Matthews Date: Wed, 4 Nov 2026 17:44:49 -0500 Subject: [PATCH 10/10] Tweaks from review feedback Signed-off-by: John Matthews --- docs/Overview.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Overview.md b/docs/Overview.md index da45045..d9c6a5b 100644 --- a/docs/Overview.md +++ b/docs/Overview.md @@ -171,9 +171,10 @@ The upgrade scenario is still being tested, the expected workflow is: 7. Enable components that match was installed/enabled on cluster prior 8. Cluster exists maintenance window - - First phase of work is walking through the manual upgrade flow and then working towards this in a CLI. - Would like to have a declarative way to run this flow for shops that have fully adopted GitOps. + - Assume Plan to publish an Argo and pipelines workflow as a reference + - Note upgrade tooling is targetted for a 5.8.z cycle - Need to learn more of how Virt AutoPilot is integrating medik8s, this is likely where we can smooth the upgrade experience for many virt users. ## Current Status @@ -193,5 +194,4 @@ The `wao` branch started from each operand's`medik8s5.8` branch as of 11/2/2026 - Implement `bundle-k8s`, we currently only create a bundle for `ocp` - Consider: Bring webhooks to run in the WAO process so they are available and running even when a component is disabled. - Consider: Rewrite the internal/operands/manifests/``/objects.yaml in Go and stop trying to sync from the operand repo. - - Condier: In future releases refactoring the code layout to better conform to kubebuilder https://book.kubebuilder.io/reference/project-config.html - + - Consider: In future releases refactoring the code layout to better conform to kubebuilder https://book.kubebuilder.io/reference/project-config.html