[SDCI-2079] Document automatic job retries (Preview) - hybrid approach #36147
+89
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,84 @@ | ||||||
| --- | ||||||
| title: Automatic Job Retries | ||||||
| further_reading: | ||||||
| - link: "/continuous_integration/pipelines" | ||||||
| tag: "Documentation" | ||||||
| text: "Explore Pipeline Execution Results and Performance" | ||||||
| - link: "/continuous_integration/pipelines/github/" | ||||||
| tag: "Documentation" | ||||||
| text: "Set up CI Visibility for GitHub Actions" | ||||||
| - link: "/continuous_integration/pipelines/gitlab/" | ||||||
| tag: "Documentation" | ||||||
| text: "Set up CI Visibility for GitLab" | ||||||
| - link: "/continuous_integration/troubleshooting/" | ||||||
| tag: "Documentation" | ||||||
| text: "Troubleshooting CI Visibility" | ||||||
| --- | ||||||
|
|
||||||
| <div class="alert alert-info">Automatic job retries are in Preview. To request access, contact your Datadog account team.</div> | ||||||
|
|
||||||
| ## Overview | ||||||
|
|
||||||
| Automatic job retries save developer time by re-running failures that are likely transient, such as network timeouts, infrastructure failures, or flaky tests. Genuine code defects are not retried. Datadog runs each failed job through an AI-powered error classifier. When the failure is identified as retriable, Datadog triggers a retry through the CI provider's API without manual intervention. | ||||||
|
|
||||||
| Automatic retries reduce the number of pipelines that developers re-run by hand, shorten feedback loops, and keep pipeline success metrics focused on non-transient failures. | ||||||
|
|
||||||
| ## How it works | ||||||
|
|
||||||
| 1. A CI job fails in your pipeline. | ||||||
| 2. Datadog's AI error classifier inspects the job's logs and error context to determine whether the failure is transient. | ||||||
| 3. If the failure is classified as retriable, Datadog requests a retry through the provider's API. | ||||||
| 4. Datadog retries each job up to a maximum number of attempts to prevent infinite retry loops. | ||||||
| 5. Datadog records the retry outcome on the original pipeline in CI Visibility. | ||||||
|
|
||||||
| ## Requirements | ||||||
|
|
||||||
| - CI Visibility enabled for your [GitHub Actions][1] or [GitLab][2] integration. | ||||||
| - [Datadog Source Code Integration][3] configured for the repositories where you want automatic retries. | ||||||
| - Indexed CI job logs for those repositories (see [Collect job logs for GitHub Actions][4] or [Collect job logs for GitLab][5]). | ||||||
| - Automatic job retries enabled for your organization (see the banner above for how to request access). | ||||||
|
|
||||||
| Automatic retries rely on the same AI error classifier used by [CI jobs failure analysis][6], which reads indexed job logs to decide whether a failure is transient. | ||||||
|
|
||||||
| ## Provider-specific behavior | ||||||
|
|
||||||
| {{< tabs >}} | ||||||
| {{% tab "GitLab" %}} | ||||||
|
|
||||||
| Datadog performs **smart retries** on GitLab: only the specific job classified as retriable is re-run. Other failed jobs (that aren't classified as retriable) and passing jobs aren't affected. | ||||||
|
|
||||||
| - Retries are triggered per job, as soon as the job fails. | ||||||
| - Smart retries work with GitLab.com (SaaS) and self-hosted GitLab instances reachable by the Datadog Source Code Integration. | ||||||
| - There is no additional CI cost beyond the retried job. | ||||||
|
|
||||||
| {{% /tab %}} | ||||||
| {{% tab "GitHub Actions" %}} | ||||||
|
|
||||||
| GitHub Actions imposes two provider-level limitations that shape how retries work: | ||||||
|
|
||||||
| - **Retries happen after the workflow finishes.** The GitHub API does not allow retrying an individual job while the rest of the workflow is still running. Datadog waits for the workflow to reach a final state before issuing retries. | ||||||
| - **All failed jobs are retried together.** The GitHub API does not support retrying a single job when other jobs in the workflow have also failed. Datadog reruns every failed job in the workflow through a single GitHub API call. This may increase your GitHub Actions compute usage. | ||||||
|
|
||||||
| ### Protected branches | ||||||
|
|
||||||
| The Datadog GitHub App's default permissions do not allow retries on protected branches. To enable automatic retries on a protected branch (for example, your default branch), grant the app Maintainer-level access. Review your organization's policies before expanding permissions. | ||||||
|
Contributor
There was a problem hiding this comment. Question here for a detail, is this org-wide or repo-wide setting for the Maintainer-level access?
Contributor
Author
There was a problem hiding this comment. this is per org AFAIK |
||||||
|
|
||||||
| {{% /tab %}} | ||||||
| {{< /tabs >}} | ||||||
|
|
||||||
| ## Limitations | ||||||
|
|
||||||
| - Each logical job is retried at most one time. | ||||||
|
Contributor
There was a problem hiding this comment. it's a bit confusing to me what does logical job stands for here, maybe you mean:
Suggested change
?
Contributor
Author
There was a problem hiding this comment. this means the same job id, maybe we should mention that or remove this line at all to avoid confusion, WDYT?
Contributor
There was a problem hiding this comment. yeah we can add some explanation on logical job concept if we want to keep, but it's just not clear to me why we need to specify retries by job ID, it's a very technical concept I think maybe we should just explain in the beginning of the doc how do we count for max number of retries so they know better what to consider for that setting, like "retries are counted for the pipelines/jobs executed in the same commit" |
||||||
| - Jobs classified as non-retriable (for example, compilation errors or asserted test failures) are never retried. | ||||||
| - If a job has already been retried manually or by provider-native retry rules, Datadog does not issue an additional retry. | ||||||
|
|
||||||
| ## Further reading | ||||||
|
|
||||||
| {{< partial name="whats-next/whats-next.html" >}} | ||||||
|
|
||||||
| [1]: /continuous_integration/pipelines/github/ | ||||||
| [2]: /continuous_integration/pipelines/gitlab/ | ||||||
| [3]: /integrations/guide/source-code-integration/ | ||||||
| [4]: /continuous_integration/pipelines/github/#collect-job-logs | ||||||
| [5]: /continuous_integration/pipelines/gitlab/#collect-job-logs | ||||||
| [6]: /continuous_integration/guides/use_ci_jobs_failure_analysis/ | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why here we say
up to a max numbut then later we specifyat most once?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
at most one the same job id, but the same job fingerprint is retried the N times, we don't have the settings yet, but once we have them we should update this doc