Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Merge requests on GitLab cannot be applyed as there are pipelines stuck in running state.

When a commit is pushed a race condition happens between GitLab creating a merge request pipeline and Atlanits updating commit status.
We have additional checks running in .gitlab-ci defined jobs.

We observed 3 possible scenarios:

• Happy path command line logs | atlantis server logs:

  1. When a commit is pushed to the merge request branch, GitLab creates a pipeline and associated jobs
  2. This logic picks up the pipeline ID and updates the commit status accordingly
  3. All pipelines associated with the commit are successful, merge checks pass, and the merge request is mergeable and can be applied

• Split pipelines command line logs | atlantis server logs:

  1. GitLab does not create a pipeline in time
  2. Atlantis sets the Ref to the branch name instead of the MR ID (after waiting for 2 seconds)
  3. GitLab creates a pipeline and associated jobs after the commit is pushed
  4. Atlantis sets the commit status to running state, whitch creates a new pipeline and associated jobs with branch source instead of MR source
  5. This pipeline is considered as latest by GitLab, so the success of atlantis workflow jobs is reported there
  6. All pipelines associated with the commit are successful, merge checks pass, and the merge request is mergeable and can be applied

• Pipeline stuck in running state command line logs | atlantis server logs:

  1. GitLab does not create a pipeline in time
  2. Atlantis sets the Ref to the branch name instead of the MR ID (after waiting for 2 seconds)
  3. Atlantis sets the commit status to running state, whitch creates a new pipeline and associated jobs with branch source instead of MR source
  4. GitLab creates a pipeline and associated jobs after the commit is pushed
  5. GitLab consideres the merge request pipeline as latest, so the success of atlantis workflow jobs is reported there
  6. The branch source pipeline is left in running state
  7. The merge request is not mergeable and cannot be applied

Reproduction Steps

• Have at least one GitLab job triggered on merge request push event
• Push commits to the merge request branch until a pipeline stuck in running state is observed

Environment details

• Atlantis version: v0.33.0
• Deployment method: helm
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: yes
• Atlantis flags:

Additional Context

We chagned the retry logic as a POC, and we have not seen this issue since. Changes made can be seen here.

I would like to add some configuration that would allow us to add custom retry logic parameters in a backwards compatible way.
Keep the default as is - 2s retry once, but allow the user to add custom paramteres in cases like ours.