Skip to content

DPTP-5324: move periodic-ipi-deprovision-aws jobs to STS - #85496

Open
bear-redhat wants to merge 1 commit into
openshift:mainfrom
bear-redhat:issue/DPTP-5324
Open

bear-redhat wants to merge 1 commit into
openshift:mainfrom
bear-redhat:issue/DPTP-5324

Conversation

@bear-redhat

@bear-redhat bear-redhat commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

The periodic-ipi-deprovision-aws[-N] jobs authenticated to AWS with the long-lived access key in cluster-secrets-aws[-N] (.awscred). That key expired (AuthFailure on DescribeRegions), so clusters were being leaked.

Switch them to STS AssumeRoleWithWebIdentity, mirroring the credential setup ci-operator uses for ipi-aws jobs:

  • aws.sh: when AWS_STS_MODE=yes, build ~/.aws/config that exchanges a projected service-account token for the per-cluster home role and then chain-assumes the per-account target role. Backward compatible; the hypershift/konflux callers that still pass static creds are unaffected.
  • infra-build-farm-periodics.yaml: the 5 aws deprovision jobs now project an sts.amazonaws.com SA token and pass HOME_ROLE_ARN/TARGET_ROLE_ARN instead of mounting .awscred.

Add the ci.openshift.io/role: cloud-aws capacity selector so the dispatcher places these jobs on AWS build clusters. They were defaulting to build13 (a GCP cluster), where the AWS STS home-role secret is not synced and STS cannot work.

Summary by CodeRabbit

The five periodic AWS IPI deprovision jobs now support STS AssumeRoleWithWebIdentity authentication. The jobs use projected service-account tokens and chained hub and target role ARNs instead of static .awscred mounts.

The jobs target AWS build clusters with ci.openshift.io/role: cloud-aws. Jira validation confirms DPTP-5324, but the issue has no target version while the target branch expects 5.1.0.

Test results and review finding counts are unavailable.

The periodic-ipi-deprovision-aws[-N] jobs authenticated to AWS with the
long-lived access key in cluster-secrets-aws[-N] (.awscred). That key
expired (AuthFailure on DescribeRegions), so clusters were being leaked.

Switch them to STS AssumeRoleWithWebIdentity, mirroring the credential
setup ci-operator uses for ipi-aws jobs:

- aws.sh: when AWS_STS_MODE=yes, build ~/.aws/config that exchanges a
  projected service-account token for the per-cluster home role and then
  chain-assumes the per-account target role. Backward compatible; the
  hypershift/konflux callers that still pass static creds are unaffected.
- infra-build-farm-periodics.yaml: the 5 aws deprovision jobs now project
  an sts.amazonaws.com SA token and pass HOME_ROLE_ARN/TARGET_ROLE_ARN
  instead of mounting .awscred.

Add the ci.openshift.io/role: cloud-aws capacity selector so the
dispatcher places these jobs on AWS build clusters. They were defaulting
to build13 (a GCP cluster), where the AWS STS home-role secret is not
synced and STS cannot work.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 18, 2026
@openshift-ci-robot

openshift-ci-robot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@bear-redhat: This pull request references DPTP-5324 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.1.0" version, but no target version was set.

Details

In response to this:

The periodic-ipi-deprovision-aws[-N] jobs authenticated to AWS with the long-lived access key in cluster-secrets-aws[-N] (.awscred). That key expired (AuthFailure on DescribeRegions), so clusters were being leaked.

Switch them to STS AssumeRoleWithWebIdentity, mirroring the credential setup ci-operator uses for ipi-aws jobs:

  • aws.sh: when AWS_STS_MODE=yes, build ~/.aws/config that exchanges a projected service-account token for the per-cluster home role and then chain-assumes the per-account target role. Backward compatible; the hypershift/konflux callers that still pass static creds are unaffected.
  • infra-build-farm-periodics.yaml: the 5 aws deprovision jobs now project an sts.amazonaws.com SA token and pass HOME_ROLE_ARN/TARGET_ROLE_ARN instead of mounting .awscred.

Add the ci.openshift.io/role: cloud-aws capacity selector so the dispatcher places these jobs on AWS build clusters. They were defaulting to build13 (a GCP cluster), where the AWS STS home-role secret is not synced and STS cannot work.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
openshift-ci Bot requested review from jmguzik and pruan-rht September 18, 2026 16:49
@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: d9f19c7e-5ad2-49d5-8764-82e530be58ce

📥 Commits

Reviewing files that changed from the base of the PR and between 7301263 and 8b64483.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/infra-build-farm-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (1)
  • core-services/ipi-deprovision/aws.sh

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


Walkthrough

The AWS deprovisioning script now supports optional STS authentication. When enabled, it configures web-identity authentication for a hub role and chained assumption of the target role.

Changes

AWS STS authentication

Layer / File(s) Summary
Conditional STS profile setup
core-services/ipi-deprovision/aws.sh
When AWS_STS_MODE=yes, the script writes chained AWS profiles, sets the AWS configuration environment variables, and unsets AWS_SHARED_CREDENTIALS_FILE. Other modes preserve the existing behavior.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 8b644

No actionable merge-blocking risk was identified for the STS authentication migration.

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: moving the periodic AWS deprovisioning jobs to STS authentication.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The pull request changes only ci-operator/jobs/infra-build-farm-periodics.yaml and core-services/ipi-deprovision/aws.sh. The changes add job configuration and AWS STS shell logic. No Ginkgo test f…
Test Structure And Quality ✅ Passed PASS: The pull request changes only ci-operator/jobs/infra-build-farm-periodics.yaml and core-services/ipi-deprovision/aws.sh. These files contain no Ginkgo tests or Ginkgo constructs such as It
Microshift Test Compatibility ✅ Passed PASS: The pull request changes only core-services/ipi-deprovision/aws.sh and ci-operator/jobs/infra-build-farm-periodics.yaml. The diff adds no Ginkgo tests or It, Describe, Context, or `Whe…
Single Node Openshift (Sno) Test Compatibility ✅ Passed The pull request changes only core-services/ipi-deprovision/aws.sh and ci-operator/jobs/infra-build-farm-periodics.yaml. The changes add AWS STS configuration and update periodic job definitions. …
Topology-Aware Scheduling Compatibility ✅ Passed No topology-incompatible scheduling condition is introduced. The PR changes five Prow periodic jobs and an AWS shell script. The only scheduling-related additions are Prow cluster selection (build11
Ote Binary Stdout Contract ✅ Passed PASS: The pull request changes only a Bash deprovisioning script and its Kubernetes job definitions. The jobs invoke aws.sh, not an OTE or openshift-tests binary. The new cat command redirects o…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS. The authoritative pull-request diff changes only core-services/ipi-deprovision/aws.sh and ci-operator/jobs/infra-build-farm-periodics.yaml. These changes add AWS STS shell configuration and …
No-Weak-Crypto ✅ Passed PASS. The pull request adds AWS STS role configuration and projected service-account token wiring only. Exact searches of all added lines found no MD5, SHA1, DES, 3DES, RC4, Blowfish, ECB, custom cryp…
Container-Privileges ✅ Passed PASS: The pull request does not introduce any listed container privilege. The authoritative diff changes AWS deprovision jobs to use projected service-account tokens and changes their cluster and labe…
No-Sensitive-Data-In-Logs ✅ Passed No new sensitive-data logging is introduced. The changed script writes secret-backed role ARNs and a token file path to an AWS config file, but it does not print the config, token, environment, or cre…
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 18, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@bear-redhat: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ipi-deprovision-aws-3 N/A periodic Periodic changed
periodic-ipi-deprovision-aws N/A periodic Periodic changed
periodic-ipi-deprovision-aws-5 N/A periodic Periodic changed
periodic-ipi-deprovision-aws-2 N/A periodic Periodic changed
periodic-ipi-deprovision-aws-4 N/A periodic Periodic changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@bear-redhat

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@bear-redhat: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@deepsm007

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 18, 2026
@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bear-redhat, deepsm007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants