|
| 1 | +# Create a Self-Managed Azure HostedCluster Without External DNS |
| 2 | + |
| 3 | +!!! note "Developer Preview in OCP 4.21" |
| 4 | + |
| 5 | + Self-managed Azure HostedClusters are available as a Developer Preview feature in OpenShift Container Platform 4.21. |
| 6 | + |
| 7 | +This document describes how to create a self-managed Azure HostedCluster without using External DNS for automatic DNS record management. |
| 8 | + |
| 9 | +## Prerequisites |
| 10 | + |
| 11 | +Before creating a hosted cluster, ensure you have completed: |
| 12 | + |
| 13 | +1. **[Azure Workload Identity Setup](azure-workload-identity-setup.md)** - Created workload identities and OIDC issuer |
| 14 | +2. **[Management Cluster Setup Without External DNS](setup-management-cluster-without-external-dns.md)** - Installed HyperShift operator |
| 15 | + |
| 16 | +Additionally, you need: |
| 17 | + |
| 18 | +- Azure CLI (`az`) configured with appropriate permissions |
| 19 | +- HyperShift CLI binary |
| 20 | +- OpenShift pull secret |
| 21 | +- SSH key (or use `--generate-ssh`) |
| 22 | +- `workload-identities.json` file from Phase 1 |
| 23 | + |
| 24 | +## Environment Setup |
| 25 | + |
| 26 | +Set the required environment variables: |
| 27 | + |
| 28 | +```bash |
| 29 | +# Cluster configuration |
| 30 | +CLUSTER_NAME="my-hosted-cluster" |
| 31 | +LOCATION="eastus" |
| 32 | +BASE_DOMAIN="example.com" |
| 33 | +PULL_SECRET="/path/to/pull-secret.json" |
| 34 | + |
| 35 | +# Azure subscription and resource groups |
| 36 | +SUBSCRIPTION_ID=$(az account show --query id -o tsv) |
| 37 | +TENANT_ID=$(az account show --query tenantId -o tsv) |
| 38 | +PERSISTENT_RG_NAME="os4-common" # From Phase 1 |
| 39 | +MANAGED_RG_NAME="${CLUSTER_NAME}-rg" |
| 40 | + |
| 41 | +# Workload Identity configuration (from Phase 1) |
| 42 | +WORKLOAD_IDENTITIES_FILE="/path/to/workload-identities.json" |
| 43 | +OIDC_ISSUER_URL="https://yourstorageaccount.blob.core.windows.net/oidc-issuer" |
| 44 | + |
| 45 | +# Networking (optional - will be created if not specified) |
| 46 | +VNET_ID="" # Leave empty to create new VNet |
| 47 | +SUBNET_ID="" # Leave empty to create new subnet |
| 48 | +NSG_ID="" # Leave empty to create new NSG |
| 49 | + |
| 50 | +# DNS Zone (optional - for custom ingress domain) |
| 51 | +DNS_ZONE_RG_NAME="" # Leave empty if not using custom DNS zone |
| 52 | + |
| 53 | +# Release configuration |
| 54 | +RELEASE_IMAGE="quay.io/openshift-release-dev/ocp-release:4.21.0-x86_64" |
| 55 | +``` |
| 56 | + |
| 57 | +## Create the HostedCluster |
| 58 | + |
| 59 | +### Basic Cluster Creation (Without Custom DNS) |
| 60 | + |
| 61 | +Create a hosted cluster using default DNS provided by Azure LoadBalancers: |
| 62 | + |
| 63 | +```bash |
| 64 | +hypershift create cluster azure \ |
| 65 | + --name "${CLUSTER_NAME}" \ |
| 66 | + --location "${LOCATION}" \ |
| 67 | + --base-domain "${BASE_DOMAIN}" \ |
| 68 | + --pull-secret "${PULL_SECRET}" \ |
| 69 | + --release-image "${RELEASE_IMAGE}" \ |
| 70 | + --node-pool-replicas 2 \ |
| 71 | + --generate-ssh \ |
| 72 | + --workload-identities-file "${WORKLOAD_IDENTITIES_FILE}" \ |
| 73 | + --oidc-issuer-url "${OIDC_ISSUER_URL}" \ |
| 74 | + --resource-group-name "${MANAGED_RG_NAME}" |
| 75 | +``` |
| 76 | + |
| 77 | +!!! note "No External DNS Domain Flag" |
| 78 | + |
| 79 | + Notice that we **do not** specify the `--external-dns-domain` flag. This tells HyperShift to: |
| 80 | + |
| 81 | + - Use a LoadBalancer for the API server (gets Azure-provided DNS automatically) |
| 82 | + - Use Routes for OAuth, Konnectivity, and Ignition services |
| 83 | + - Not create any custom hostname annotations for External DNS |
| 84 | + |
| 85 | +### With Custom Networking |
| 86 | + |
| 87 | +If you have pre-created VNet, Subnet, and NSG resources: |
| 88 | + |
| 89 | +```bash |
| 90 | +hypershift create cluster azure \ |
| 91 | + --name "${CLUSTER_NAME}" \ |
| 92 | + --location "${LOCATION}" \ |
| 93 | + --base-domain "${BASE_DOMAIN}" \ |
| 94 | + --pull-secret "${PULL_SECRET}" \ |
| 95 | + --release-image "${RELEASE_IMAGE}" \ |
| 96 | + --node-pool-replicas 2 \ |
| 97 | + --generate-ssh \ |
| 98 | + --workload-identities-file "${WORKLOAD_IDENTITIES_FILE}" \ |
| 99 | + --oidc-issuer-url "${OIDC_ISSUER_URL}" \ |
| 100 | + --resource-group-name "${MANAGED_RG_NAME}" \ |
| 101 | + --vnet-id "${VNET_ID}" \ |
| 102 | + --subnet-id "${SUBNET_ID}" \ |
| 103 | + --network-security-group-id "${NSG_ID}" |
| 104 | +``` |
| 105 | + |
| 106 | +### With Custom DNS Zone for Ingress |
| 107 | + |
| 108 | +If you want to use a custom DNS zone for the ingress domain (*.apps): |
| 109 | + |
| 110 | +```bash |
| 111 | +hypershift create cluster azure \ |
| 112 | + --name "${CLUSTER_NAME}" \ |
| 113 | + --location "${LOCATION}" \ |
| 114 | + --base-domain "${BASE_DOMAIN}" \ |
| 115 | + --pull-secret "${PULL_SECRET}" \ |
| 116 | + --release-image "${RELEASE_IMAGE}" \ |
| 117 | + --node-pool-replicas 2 \ |
| 118 | + --generate-ssh \ |
| 119 | + --workload-identities-file "${WORKLOAD_IDENTITIES_FILE}" \ |
| 120 | + --oidc-issuer-url "${OIDC_ISSUER_URL}" \ |
| 121 | + --resource-group-name "${MANAGED_RG_NAME}" \ |
| 122 | + --dns-zone-rg-name "${DNS_ZONE_RG_NAME}" |
| 123 | +``` |
| 124 | + |
| 125 | +!!! info "DNS Zone Resource Group" |
| 126 | + |
| 127 | + The `--dns-zone-rg-name` flag specifies where your Azure DNS zone for the ingress domain resides. This is required by the ingress controller to manage DNS records for `*.apps.${CLUSTER_NAME}.${BASE_DOMAIN}`. |
| 128 | + |
| 129 | + This is **different** from External DNS - it's used by the hosted cluster's own ingress controller. |
| 130 | + |
| 131 | +## Verify Cluster Creation |
| 132 | + |
| 133 | +Monitor the hosted cluster deployment: |
| 134 | + |
| 135 | +```bash |
| 136 | +# Watch the HostedCluster resource |
| 137 | +oc get hostedcluster -n clusters -w |
| 138 | + |
| 139 | +# Check the hosted control plane pods |
| 140 | +oc get pods -n clusters-${CLUSTER_NAME} |
| 141 | + |
| 142 | +# Get the cluster status |
| 143 | +oc get hostedcluster ${CLUSTER_NAME} -n clusters -o jsonpath='{.status.conditions[?(@.type=="Available")].status}' |
| 144 | +``` |
| 145 | + |
| 146 | +## Get Cluster Access Information |
| 147 | + |
| 148 | +### API Server Endpoint |
| 149 | + |
| 150 | +Without External DNS, the API server uses an Azure LoadBalancer with an Azure-provided DNS name: |
| 151 | + |
| 152 | +```bash |
| 153 | +# Get the API server LoadBalancer service |
| 154 | +API_LB_SERVICE=$(oc get svc -n clusters-${CLUSTER_NAME} kube-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') |
| 155 | +echo "API Server: https://${API_LB_SERVICE}:6443" |
| 156 | + |
| 157 | +# Or get the IP address |
| 158 | +API_LB_IP=$(oc get svc -n clusters-${CLUSTER_NAME} kube-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}') |
| 159 | +echo "API Server IP: ${API_LB_IP}" |
| 160 | +``` |
| 161 | + |
| 162 | +### Get Kubeconfig |
| 163 | + |
| 164 | +Retrieve the kubeconfig for the hosted cluster: |
| 165 | + |
| 166 | +```bash |
| 167 | +hypershift create kubeconfig --name ${CLUSTER_NAME} > ${CLUSTER_NAME}-kubeconfig |
| 168 | +``` |
| 169 | + |
| 170 | +The kubeconfig will use the Azure LoadBalancer DNS name for the API server endpoint. |
| 171 | + |
| 172 | +### Get Admin Password |
| 173 | + |
| 174 | +```bash |
| 175 | +oc get secret -n clusters ${CLUSTER_NAME}-kubeadmin-password \ |
| 176 | + -o jsonpath='{.data.password}' | base64 -d |
| 177 | +``` |
| 178 | + |
| 179 | +## Optional: Create Custom DNS Records |
| 180 | + |
| 181 | +If you want to use a custom domain name for the API server instead of the Azure-provided name: |
| 182 | + |
| 183 | +### Create DNS A Record for API Server |
| 184 | + |
| 185 | +```bash |
| 186 | +# Get the LoadBalancer public IP |
| 187 | +API_LB_IP=$(oc get svc -n clusters-${CLUSTER_NAME} kube-apiserver \ |
| 188 | + -o jsonpath='{.status.loadBalancer.ingress[0].ip}') |
| 189 | + |
| 190 | +# Create DNS A record (example using Azure DNS) |
| 191 | +az network dns record-set a add-record \ |
| 192 | + --resource-group YOUR_DNS_ZONE_RG \ |
| 193 | + --zone-name ${BASE_DOMAIN} \ |
| 194 | + --record-set-name api.${CLUSTER_NAME} \ |
| 195 | + --ipv4-address ${API_LB_IP} |
| 196 | + |
| 197 | +echo "Custom API endpoint: https://api.${CLUSTER_NAME}.${BASE_DOMAIN}:6443" |
| 198 | +``` |
| 199 | + |
| 200 | +### Update Kubeconfig with Custom DNS |
| 201 | + |
| 202 | +If you created a custom DNS record, you can manually edit the kubeconfig to use your custom domain: |
| 203 | + |
| 204 | +```bash |
| 205 | +# Edit the kubeconfig server field |
| 206 | +sed -i "s|https://.*:6443|https://api.${CLUSTER_NAME}.${BASE_DOMAIN}:6443|" ${CLUSTER_NAME}-kubeconfig |
| 207 | +``` |
| 208 | + |
| 209 | +## DNS Architecture Without External DNS |
| 210 | + |
| 211 | +Understanding how DNS works in this setup: |
| 212 | + |
| 213 | +| Service | Publishing Type | DNS Handling | |
| 214 | +|---------|----------------|--------------| |
| 215 | +| **API Server** | LoadBalancer | Azure-provided DNS (e.g., `abc123.eastus.cloudapp.azure.com`)<br>Or manually created DNS A record | |
| 216 | +| **OAuth** | Route | Management cluster ingress (e.g., `oauth-clustername.apps.mgmt-cluster.com`) | |
| 217 | +| **Konnectivity** | Route | Management cluster ingress | |
| 218 | +| **Ignition** | Route | Management cluster ingress | |
| 219 | +| **Ingress (*.apps)** | Managed by hosted cluster | Uses DNS zone specified in `--dns-zone-rg-name`<br>Or manual DNS configuration | |
| 220 | + |
| 221 | +## Comparison: With vs Without External DNS |
| 222 | + |
| 223 | +| Aspect | With External DNS | Without External DNS | |
| 224 | +|--------|------------------|---------------------| |
| 225 | +| **API Server** | Route with custom hostname<br>`api-cluster.external-domain.com` | LoadBalancer with Azure DNS<br>`abc123.region.cloudapp.azure.com` | |
| 226 | +| **Setup Complexity** | Higher (service principal, DNS zones, external-dns operator) | Lower (no additional operator) | |
| 227 | +| **DNS Management** | Automatic via external-dns | Manual or Azure-provided | |
| 228 | +| **Custom Domains** | Built-in | Manual DNS record creation | |
| 229 | +| **Best For** | Production, multi-cluster, custom branding | Development, testing, simpler setups | |
| 230 | + |
| 231 | +## Troubleshooting |
| 232 | + |
| 233 | +### API Server Not Accessible |
| 234 | + |
| 235 | +```bash |
| 236 | +# Check LoadBalancer service status |
| 237 | +oc get svc -n clusters-${CLUSTER_NAME} kube-apiserver |
| 238 | + |
| 239 | +# Check if LoadBalancer has external IP/hostname |
| 240 | +oc describe svc -n clusters-${CLUSTER_NAME} kube-apiserver |
| 241 | + |
| 242 | +# Verify network security group allows traffic on port 6443 |
| 243 | +az network nsg rule list \ |
| 244 | + --resource-group ${MANAGED_RG_NAME} \ |
| 245 | + --nsg-name ${CLUSTER_NAME}-nsg \ |
| 246 | + --query "[?destinationPortRange=='6443']" |
| 247 | +``` |
| 248 | + |
| 249 | +### Ingress Controller Issues |
| 250 | + |
| 251 | +```bash |
| 252 | +# Check ingress controller pods in hosted cluster |
| 253 | +oc --kubeconfig ${CLUSTER_NAME}-kubeconfig get pods -n openshift-ingress |
| 254 | + |
| 255 | +# Check ingress controller configuration |
| 256 | +oc --kubeconfig ${CLUSTER_NAME}-kubeconfig get ingresscontroller -n openshift-ingress-operator |
| 257 | +``` |
| 258 | + |
| 259 | +## Next Steps |
| 260 | + |
| 261 | +- [Configure Additional NodePools](../automated-machine-management/nodepool-lifecycle.md) |
| 262 | +- [Upgrade Your Hosted Cluster](../upgrades.md) |
| 263 | +- [Monitor Your Hosted Cluster](../per-hostedcluster-dashboard.md) |
| 264 | + |
| 265 | +## Related Documentation |
| 266 | + |
| 267 | +- [Azure Workload Identity Setup](azure-workload-identity-setup.md) - Phase 1 prerequisites |
| 268 | +- [Management Cluster Setup Without External DNS](setup-management-cluster-without-external-dns.md) - Phase 2 |
| 269 | +- [Self-Managed Azure Overview](self-managed-azure-index.md) - Architecture and prerequisites |
| 270 | +- [With External DNS](create-azure-cluster-on-aks.md) - Alternative setup with automatic DNS |
0 commit comments