"Plan not found on PR"

[PeterBocan]

I have a GH workflow setup in such a way that it runs tf plan in the branch workflow on the PR. It is executed in every changed subdirectory which generates multiple plans (one for every subdirectory). When the PR lands, the main workflow runs on the main branch. But the tf apply announces that the plan was not found on the PR. Anybody got the experience with this issue?

[dflook]

Hi, can you post the workflows here?

[PeterBocan]

name: Branch workflow 

on: 
  pull_request:
    branches:
      - main

permissions:
  contents: read
  pull-requests: write

jobs:
  changes:
    name: Detect changes 
    runs-on: ubuntu-latest
    permissions:
      pull-requests: read
    outputs:
      proxy: ${{ steps.filter.outputs.proxy }}
      ireland: ${{ steps.filter.outputs.ireland }}
      london: ${{ steps.filter.outputs.london }}
      dev: ${{ steps.filter.outputs.dev }}
      prod: ${{ steps.filter.outputs.prod }}
    steps:
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            proxy:
              - 'api-proxy/**'
            ireland:
              - 'corp/ireland/**'
            london:
              - 'corp/london/**'
            dev:
              - 'env/dev/**'
            prod:
              - 'env/prod/**'

  build-proxy:
    name: Test build the Docker image
    needs: [changes]
    if: ${{ needs.changes.outputs.proxy == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 
      - name: Build Docker image 
        id: build
        env:
          ECR_REPOSITORY: api_proxy
          ECR_REGISTRY: xxxx.dkr.ecr.eu-west-2.amazonaws.com
          PROD_ECR_REGISTRY: xxxx.dkr.ecr.eu-west-1.amazonaws.com
          IMAGE_TAG: 0.0.0 
        run: |
          cd api-proxy
          docker build .
    outputs:
      IMAGE_PATH: ${{ steps.build.outputs.IMAGE_PATH }}
  
  terraform-dev:
    needs: [changes]
    if: ${{ needs.changes.outputs.dev == 'true' }}
    runs-on: ubuntu-latest
    environment:
      name: development
    env: 
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 
      
      - name: Terraform Plan for Dev
        uses: dflook/terraform-plan@v1
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          TERRAFORM_HTTP_CREDENTIALS: |
            github.com/foo/infrastructure.git=foo:${{ secrets.GH_ACCESS_TOKEN }}
        with:
          path: env/dev
          
  terraform-prod:
    needs: [changes]
    if: ${{ needs.changes.outputs.prod == 'true' }}
    runs-on: ubuntu-latest
    environment:
      name: development
    env: 
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 

      - name: Generate the Prod images tfvars
        env: 
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          ./grab-current-images.sh eu-west-1 prod ./images.tfvars
      
      - name: Terraform Plan for Prod
        uses: dflook/terraform-plan@v1
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          TERRAFORM_HTTP_CREDENTIALS: |
            github.com/foo/infrastructure.git=foo:${{ secrets.GH_ACCESS_TOKEN }}
        with:
          path: env/prod
          var_file: |
            images.tfvars      
    
  terraform-corp-ireland:
    needs: [changes]
    if: ${{ needs.changes.outputs.ireland == 'true' }}
    runs-on: ubuntu-latest
    environment:
      name: development
    env: 
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 
      
      - name: Terraform Plan for Corp Ireland
        uses: dflook/terraform-plan@v1
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          TERRAFORM_HTTP_CREDENTIALS: |
            github.com/foo/infrastructure.git=foo:${{ secrets.GH_ACCESS_TOKEN }}
        with:
          path: corp/ireland

  terraform-corp-london:
    needs: [changes]
    if: ${{ needs.changes.outputs.london == 'true' }}
    runs-on: ubuntu-latest
    environment:
      name: development
    env: 
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 
      
      - name: Terraform Plan for Corp London 
        uses: dflook/terraform-plan@v1
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          TERRAFORM_HTTP_CREDENTIALS: |
            github.com/foo/infrastructure.git=foo:${{ secrets.GH_ACCESS_TOKEN }}
        with:
          path: corp/london

[PeterBocan]

name: Build Main

on:
  push:
    branches:
      - "main"

concurrency: build-${{ github.repository }}

permissions:
  pull-requests: write
  contents: write

jobs:
  build-version:
    name: Create Build version
    runs-on: ubuntu-latest
    outputs:
      version: ${{ steps.output-version-step.outputs.version }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Create build version
        id: version
        uses: paulhatch/semantic-version@v4.0.3
        with:
          # The prefix to use to identify tags
          tag_prefix: "v"
          # A string which, if present in a git commit, indicates that a change represents a
          # major (breaking) change, supports regular expressions wrapped with '/'
          major_pattern: "(MAJOR)"
          # A string which indicates the flags used by the `major_pattern` regular expression. Supported flags: idgs
          major_regexp_flags: ""
          # Same as above except indicating a minor change, supports regular expressions wrapped with '/'
          minor_pattern: "(MINOR)"
          # A string which indicates the flags used by the `minor_pattern` regular expression. Supported flags: idgs
          minor_regexp_flags: ""
          # A string to determine the format of the version output
          format: "${major}.${minor}.${patch}"
          # If this is set to true, *every* commit will be treated as a new version.
          bump_each_commit: false
          # If true, the body of commits will also be searched for major/minor patterns to determine the version type.
          search_commit_body: false
          # The output method used to generate list of users, 'csv' or 'json'.
          user_format_type: "csv"

      - name: Set build version
        run: echo "VERSION=${{ steps.version.outputs.version }}" >> $GITHUB_ENV

      - name: Tag commit
        uses: tvdias/github-tagger@v0.0.1
        with:
          repo-token: "${{ secrets.GITHUB_TOKEN }}"
          tag: v${{ env.VERSION }}

      - name: Create release
        uses: ncipollo/release-action@v1
        with:
          token: "${{ secrets.GHA_PAT_JH }}"
          tag: v${{ env.VERSION }}
      - id: output-version-step
        run: echo "version=${{ env.VERSION }}" >> "$GITHUB_OUTPUT"

  # NOTE: PAY ATTENTION HERE. THIS STEP DETECTS WHICH DIRECTORIES CONTAIN CHANGES. 
  changes:
    name: Detect changes 
    needs: [build-version]
    runs-on: ubuntu-latest
    permissions:
      pull-requests: read
      contents: read
    outputs:
      proxy: ${{ steps.filter.outputs.proxy }}
      corp: ${{ steps.filter.outputs.corp }}
      dev: ${{ steps.filter.outputs.dev }}
      prod: ${{ steps.filter.outputs.prod }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            ireland:
              - 'corp/ireland/**'
            london:
              - 'corp/london/**'
            dev:
              - 'env/dev/**'
              - 'modules/**'
            prod:
              - 'env/prod/**'
              - 'modules/**'

  build-proxy:
    name: Build the branch
    needs: [build-version]
    runs-on: ubuntu-latest
    env: 
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 
      - name: Login to AWS ECR 
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: eu-west-2
        run: |
          docker logout
          aws ecr get-login-password --region "$AWS_REGION" | \
          docker login --username AWS --password-stdin "xxxx.dkr.ecr.$AWS_REGION.amazonaws.com"
      - name: Build, tag, and push image to Amazon ECR
        id: build
        env:
          ECR_REPOSITORY: api_proxy
          ECR_REGISTRY: xxxx.dkr.ecr.eu-west-2.amazonaws.com
          IMAGE_TAG: ${{ needs.build-version.outputs.version }}
        run: |
          cd api-proxy
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .

          echo "Pushing to dev ECR"
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          echo "IMAGE_PATH=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
    outputs:
      IMAGE_PATH: ${{ steps.build.outputs.IMAGE_PATH }}

  deploy-proxy:
    needs: [build-proxy]
    name: Deploy API proxy to dev
    uses: foo/.github/.github/workflows/deploy_image_ecs.yml@main
    secrets: inherit
    with:
      notify_on_failure: true
      aws_region: eu-west-2
      service_name: api-proxy
      image_name: ${{ needs.build-proxy.outputs.IMAGE_PATH }}
      ecs_cluster_name: dev
      environment_name: development
      task_family: dev-proxy
  
  terraform-dev:
    needs: [changes]
    if: ${{ needs.changes.outputs.dev == 'true' }}
    runs-on: ubuntu-latest 
    env: 
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    environment:
      name: development
    steps:
      - name: Checkout code
        uses: actions/checkout@v3 
      
      - name: Terraform Apply for Dev
        uses: dflook/terraform-apply@v1
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          TERRAFORM_HTTP_CREDENTIALS: |
            github.com/foo/infrastructure.git=foo:${{ secrets.GH_ACCESS_TOKEN }}
        with:
          path: env/dev 

[dflook]

Does a comment exist on the PR with the changes? It looks like the Detect changes job has different filters for each workflow.
I can imagine a change in modules/ for example would not generate a plan but it would try and apply it when merged, which would fail because the plan is missing.

[PeterBocan]

Yes it does. A plan is only generated when there are changes in that specific folder and the tf runs only if there are specific changes.

[F21]

I am also getting "Plan not found on PR" errors after upgrading from 1.35 to 1.42. I did not have time to debug this, but this is what happens:

1. Plan is created in a PR and the output is posted as a comment.
2. I merge the PR and the apply begins.
3. The apply fails due to some transient error (mainly due to azure's credentials being eventually consistent, causing issues).
4. I rerun the failed job and the transient error is gone, but after planning is completed, the apply fails due to "Plan not found on PR".

Previously, I was able to rerun failed jobs successfully, but not sure why it doesn't work anymore.

[PeterBocan]

I haven't managed to look into it and investigate. I am at first attributing it to the concurrency issues, because I am running two (or more) terraform applies in the same PR.

[PeterBocan]

@dflook any idea what might be going on?