Trying to change AvailabilityZones of a private v2 rack

[exalted]

Issue Description

Re-posting https://community.convox.com/t/trying-to-change-availabilityzones-of-a-private-rack/863

Our rack is Private=Yes and we're using InstanceType=t3.medium, which is a problem with AvailabilityZones=us-east-1a,us-east-1b,us-east-1c, because t3.medium isn't available in us-east-1a.

CloudFormation error message:

CIDR Block must change if Availability Zone is changed and VPC ID is not changed

So, we've tried to set AvailabilityZones to:

• us-east-1b,us-east-1c,us-east-1d
• OR us-east-1d,us-east-1e,us-east-1f

However, later we've noticed, in order to change the AvailabilityZones we also needed to change CIDR blocks for all the subnets (including private ones.) So this is what we tried in the end:

Before:

AvailabilityZones="us-east-1b,us-east-1c,us-east-1d"
Subnet0CIDR="10.2.1.0/24"
Subnet1CIDR="10.2.1.0/24"
Subnet2CIDR="10.2.1.0/24"
SubnetPrivate0CIDR="10.2.1.0/24"
SubnetPrivate1CIDR="10.2.1.0/24"
SubnetPrivate2CIDR="10.2.1.0/24"
VPCCIDR="10.2.0.0/1"

Attempt no. 1:

(Uses us-east-1b,us-east-1c,us-east-1d AZs and 10.2.1*.0 IPs.)

convox rack params set AvailabilityZones="us-east-1b,us-east-1c,us-east-1d" Subnet0CIDR="10.2.11.0/24" Subnet1CIDR="10.2.12.0/24" Subnet2CIDR="10.2.13.0/24" SubnetPrivate0CIDR="10.2.14.0/24" SubnetPrivate1CIDR="10.2.15.0/24" SubnetPrivate2CIDR="10.2.16.0/24" --wait

Attempt no. 2:

(Uses us-east-1d,us-east-1e,us-east-1f AZs and 10.2.2*.0 IPs.)

convox rack params set AvailabilityZones="us-east-1d,us-east-1e,us-east-1f" Subnet0CIDR="10.2.21.0/24" Subnet1CIDR="10.2.22.0/24" Subnet2CIDR="10.2.23.0/24" SubnetPrivate0CIDR="10.2.24.0/24" SubnetPrivate1CIDR="10.2.25.0/24" SubnetPrivate2CIDR="10.2.26.0/24" --wait

So, previously here @ddollar suggests this should work, however in both of our attempts CloudFormation failed with the following error message:

NatGateway nat-0d5e9c2f963ab7d1c is in state failed and hence failed to stabilize. Detailed failure message: Elastic IP address [eipalloc-025cc22bfe58bce88] is already associated

I am not sure at this point how to proceed.

Can anyone please jump in and help us out? Thanks!

Latest Update

Dated: 9 Mar 2022