Skip to content

Fix the Terraform deploy and test both container APIs in CI - #14

Merged
HarshCasper merged 10 commits into
mainfrom
ci/assert-container-apis
Oct 19, 2026
Merged

HarshCasper merged 10 commits into
mainfrom
ci/assert-container-apis

Conversation

@HarshCasper

@HarshCasper HarshCasper commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

CI deployed the stack but only listed the API and user pool. The Terraform job also failed on every run since March, after hanging for about 50 minutes.

CI runs on the LocalStack dev image until a release includes the API Gateway fixes below. The deploy works with both Terraform and CloudFormation, and on a LocalStack build that has the fixes both jobs pass make test.

Terraform fixes:

  • aws_service_discovery_private_dns_namespace got aws_vpc.vpc.arn. Cloud Map needs the VPC ID, so namespace creation failed with InvalidVPCId (returned as a 500, which the provider retried for 50 minutes). It now uses aws_vpc.vpc.id, matching the CloudFormation template's !Ref VPC. The apply now takes about 2 minutes.
  • The JWT authorizer audience was the client secret, which is empty because the client has no secret. It now uses the client ID, like the CloudFormation template. The web client sends the Cognito access token, whose client_id must match the audience.
  • The api_test_page output passed the pool ARN and client secret where the web client expects the pool ID and client ID.

Test:

  • New test.sh (make test) waits for both ECS services to run tasks, signs up, confirms and logs in a Cognito user, then for the foodstore and petstore APIs:
    • an anonymous GET on an unknown item returns {}
    • a PUT without a token returns 401
    • a PUT with the access token stores the item
    • a GET and the DynamoDB table return the stored item
  • It finds the cluster, services and user pool by listing them, and reads the DynamoDB table names from the DynamoDBTable variable in the task definitions, since the CloudFormation template does not name them.
  • make cloudformation-setup and the CloudFormation job now wait for CREATE_COMPLETE.
  • Both jobs run make test, time out after 30 minutes, and on failure print the Cloud Map instances, ECS tasks and LocalStack logs.
  • README documents make test.

LocalStack bugs that blocked the API calls:

  1. API Gateway HTTP API private integrations with Cloud Map ignored AWS_INSTANCE_PORT and sent requests to port 80, and ECS service registries registered tasks without the port (and not all of them).
  2. Once the port was used, the private integration dropped the request path: requests went to http://<task ip>:8080 instead of http://<task ip>:8080/foodstore/foods/<id>, and the containers returned 404.

Both are fixed in LocalStack, but the path fix is not in latest (2026.10.0) yet. Run locally against localstack/localstack-pro:dev, the Terraform deploy passed every make test check on 2026.11.0.dev116, and the CloudFormation deploy passed with this test.sh on 2026.11.0.dev119. Both jobs pass on 2026.11.0.dev156 as well, so until a release includes the fix both CI jobs run on localstack/localstack-pro:dev. Switch them back to the default image (lstk start without --image) once it ships.

The private DNS namespace got the VPC ARN instead of its ID, so the Terraform job hung for 50 minutes and failed. Fix that and the JWT audience, wait for the CloudFormation stack, and replace the resource listing with a test that calls both APIs with and without a Cognito token.

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
The CloudFormation template does not name its user pool. Temporarily enable debug logs to see why the ECS tasks are missing from Cloud Map.

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
CloudFormation generates the cluster name. Temporary experiment: register a foodstore container with AWS_INSTANCE_PORT to see whether API Gateway then reaches it.

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
The experiment showed API Gateway ignores AWS_INSTANCE_PORT, which is a LocalStack bug the sample cannot work around.

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
The CloudFormation template leaves the table names to CloudFormation
(stack1-DynamoDBTableFoodstoreFo-<hex>), so the fixed FoodStoreFoods and
PetStorePets names only matched the Terraform deploy. Both deploys pass
the table to the containers as the DynamoDBTable environment variable,
so take it from there.

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
The API Gateway private integration fix (request path forwarded to the
Cloud Map instance) is in localstack/localstack-pro:dev but not yet in
latest (2026.10.0), so both jobs fail at the first API call on latest.
Both jobs pass on dev (2026.11.0.dev156). Switch back to the default
image once a release includes the fix.

Claude-Session: https://claude.ai/code/session_01DZfDmzUBPqNAu3yFKyfCNM
@HarshCasper
HarshCasper merged commit 98f30b3 into main Oct 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant