diff --git a/.github/workflows/bot-command.yml b/.github/workflows/bot-command.yml index 0a112dbf5435..56018b3559a1 100644 --- a/.github/workflows/bot-command.yml +++ b/.github/workflows/bot-command.yml @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); @@ -52,14 +52,14 @@ jobs: "`--disable-reuse-test ` *(OPTIONAL)* : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.\n\n" + "`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.\n\n" + "`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.\n\n" + - "`--stage-list \"A10-PyTorch-1, xxx\"` *(OPTIONAL)* : Only run the specified test stages. Supports wildcard `*` for pattern matching (e.g., `\"*PerfSanity*\"` matches all stages containing PerfSanity). Examples: \"A10-PyTorch-1, xxx\", \"*PerfSanity*\". Note: Does **NOT** update GitHub check status.\n\n" + + "`--stage-list \"A10-PyTorch-1, xxx\"` *(OPTIONAL)* : Only run the specified test stages. Supports wildcard `*` for pattern matching (e.g., `\"*PerfSanity*\"` matches all stages containing PerfSanity). Examples: \"A10-PyTorch-1, xxx\", \"*PerfSanity*\". The broad patterns `\"*\"` and `\"*Post-Merge*\"`, including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the `ci: post-merge approved` PR label. Note: Does **NOT** update GitHub check status.\n\n" + "`--gpu-type \"A30, H100_PCIe\"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: \"A30, H100_PCIe\". Note: Does **NOT** update GitHub check status.\n\n" + "`--test-backend \"pytorch, cpp\"` *(OPTIONAL)* : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: \"pytorch, cpp\" (does not run test stages with tensorrt or triton backend). Note: Does **NOT** update GitHub pipeline status.\n\n" + "`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.\n\n" + "`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.\n\n" + "`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.\n\n" + - "`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.\n\n" + - "`--extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard `*` for pattern matching. Examples: --extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\", --extra-stage \"*Post-Merge*\".\n\n" + + "`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the `ci: post-merge approved` PR label applied by an active member of `NVIDIA/trt-llm-ci-approvers`. The approval label remains in place when new commits are pushed.\n\n" + + "`--extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard `*` for pattern matching. Examples: --extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\", --extra-stage \"*Post-Merge*\". The broad patterns `\"*\"` and `\"*Post-Merge*\"`, including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the `ci: post-merge approved` PR label.\n\n" + "`--detailed-log ` *(OPTIONAL)* : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.\n\n" + "`--debug ` *(OPTIONAL)* : **Experimental feature**. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the `stage-list` parameter to access the appropriate container environment. Note: Does **NOT** update GitHub check status.\n\n" + "`--high-priority ` *(OPTIONAL)* : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.\n\n" + diff --git a/.github/workflows/post-merge-approval.yml b/.github/workflows/post-merge-approval.yml new file mode 100644 index 000000000000..31aa37370772 --- /dev/null +++ b/.github/workflows/post-merge-approval.yml @@ -0,0 +1,217 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Guard Post-Merge Approval Label + +on: + # Intentional: this runs static default-branch API logic only. It never checks + # out or executes PR code; it uses GitHub APIs to validate membership and + # manage this PR's approval label/comment. + pull_request_target: + types: [labeled] + +permissions: + contents: read + pull-requests: write + +jobs: + guard-post-merge-approval: + concurrency: + group: post-merge-approval-${{ github.event.pull_request.number }} + cancel-in-progress: false + if: >- + github.repository == 'NVIDIA/TensorRT-LLM' && + github.event.action == 'labeled' && + github.event.label.name == 'ci: post-merge approved' + runs-on: ubuntu-latest + steps: + - name: Validate post-merge approver + id: validate + if: github.event.action == 'labeled' + uses: actions/github-script@v8 + with: + github-token: ${{ secrets.TRTLLM_AGENT_SHARED_TOKEN }} + result-encoding: string + script: | + const approvalLabel = 'ci: post-merge approved'; + const owner = context.repo.owner; + const repo = context.repo.repo; + const issueNumber = context.payload.pull_request.number; + let actor = context.payload.sender?.login || context.actor; + let labelEventId = ''; + let timelineVerified = false; + + try { + const events = await github.paginate( + github.rest.issues.listEventsForTimeline, + { owner, repo, issue_number: issueNumber, per_page: 100 } + ); + const latestApprovalEvent = events + .filter( + (event) => + event.event === 'labeled' && + event.label?.name === approvalLabel + ) + .at(-1); + if (latestApprovalEvent) { + actor = latestApprovalEvent.actor?.login || actor; + labelEventId = String(latestApprovalEvent.id); + timelineVerified = true; + } else { + core.warning('Could not identify the latest post-merge approval event.'); + } + } catch (error) { + core.warning( + 'Could not read the latest post-merge approval event: ' + error.message + ); + } + + core.setOutput('validated_actor', actor); + core.setOutput('label_event_id', labelEventId); + if (!timelineVerified) { + return 'false'; + } + try { + const response = await github.request( + 'GET /orgs/{org}/teams/{team_slug}/memberships/{username}', + { + org: 'NVIDIA', + team_slug: 'trt-llm-ci-approvers', + username: actor, + } + ); + const authorized = response.data.state === 'active'; + console.log( + actor + ' active membership in NVIDIA/trt-llm-ci-approvers: ' + authorized + ); + return authorized ? 'true' : 'false'; + } catch (error) { + if (error.status === 404) { + console.log( + actor + ' is not an active member of NVIDIA/trt-llm-ci-approvers.' + ); + } else { + core.warning( + 'Could not verify post-merge approver ' + actor + ': ' + error.message + ); + } + return 'false'; + } + + - name: Clear unauthorized post-merge approval + if: always() && !cancelled() && steps.validate.outputs.result != 'true' + uses: actions/github-script@v8 + env: + VALIDATED_ACTOR: ${{ steps.validate.outputs.validated_actor }} + VALIDATED_LABEL_EVENT_ID: ${{ steps.validate.outputs.label_event_id }} + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + script: | + const approvalLabel = 'ci: post-merge approved'; + const actor = + process.env.VALIDATED_ACTOR || + context.payload.sender?.login || + context.actor; + const validatedEventId = process.env.VALIDATED_LABEL_EVENT_ID || ''; + const owner = context.repo.owner; + const repo = context.repo.repo; + const issueNumber = context.payload.pull_request.number; + + try { + const pullRequest = await github.rest.pulls.get({ + owner, + repo, + pull_number: issueNumber, + }); + const labelIsPresent = pullRequest.data.labels.some( + (label) => label.name === approvalLabel + ); + if (!labelIsPresent) { + console.log('Post-merge approval label is already absent; no cleanup needed.'); + return; + } + } catch (error) { + core.warning( + 'Could not read the current post-merge approval label state; continuing validation: ' + + error.message + ); + } + + try { + const events = await github.paginate( + github.rest.issues.listEventsForTimeline, + { owner, repo, issue_number: issueNumber, per_page: 100 } + ); + const latestApprovalEvent = events + .filter( + (event) => + event.event === 'labeled' && + event.label?.name === approvalLabel + ) + .at(-1); + const latestEventId = latestApprovalEvent + ? String(latestApprovalEvent.id) + : ''; + const latestActor = latestApprovalEvent?.actor?.login || ''; + if (!latestApprovalEvent) { + core.warning( + 'Could not identify the latest post-merge approval event; removing the label to fail closed.' + ); + } else if (!latestActor) { + core.warning( + 'Could not identify the latest post-merge approval actor; removing the label to fail closed.' + ); + } else if (!validatedEventId) { + core.warning( + 'The validation run did not bind an approval event; removing the label to fail closed.' + ); + } else if ( + latestEventId !== validatedEventId || + latestActor !== actor + ) { + console.log( + 'A newer post-merge approval event was found; leaving it for its own validation run.' + ); + return; + } + } catch (error) { + core.warning( + 'Could not re-check the latest post-merge approval event; removing the label to fail closed: ' + + error.message + ); + } + + try { + await github.rest.issues.removeLabel({ + owner, + repo, + issue_number: issueNumber, + name: approvalLabel, + }); + } catch (error) { + if (error.status !== 404) { + throw error; + } + } + + await github.rest.issues.createComment({ + owner, + repo, + issue_number: issueNumber, + body: + 'Removed the "' + approvalLabel + '" label because @' + actor + + ' could not be verified as an active member of ' + + 'NVIDIA/trt-llm-ci-approvers. Ask a member of that team to apply it.', + }); diff --git a/docs/source/developer-guide/ci-overview.md b/docs/source/developer-guide/ci-overview.md index 908315c3e3c1..1bbecd048e18 100644 --- a/docs/source/developer-guide/ci-overview.md +++ b/docs/source/developer-guide/ci-overview.md @@ -109,8 +109,15 @@ developers understand why the test is disabled. ### Triggering Post-merge tests -When you only need to verify a handful of post-merge tests, avoid the heavy -`/bot run --post-merge` command. Instead, specify exactly which stages to run: +Full `/bot run --post-merge` runs require the `ci: post-merge approved` PR +label because they can consume substantial shared GPU resources. The label must +be applied by an active member of the `NVIDIA/trt-llm-ci-approvers` GitHub team. +A GitHub workflow removes invalid approvals. The label remains in place when new +commits are pushed and can be removed manually when the approval no longer +applies. + +When you only need to verify a handful of post-merge tests, specify exactly +which stages to run: ```bash /bot run --stage-list "stage-A,stage-B" @@ -123,8 +130,14 @@ default pre-merge set: /bot run --extra-stage "stage-A,stage-B" ``` -Both options accept any stage name defined in `jenkins/L0_Test.groovy`. Being -selective keeps CI turnaround fast and conserves hardware resources. +Both options accept stage names and wildcard patterns defined in +`jenkins/L0_Test.groovy`. The broad `"*"` and `"*Post-Merge*"` selectors +require the same approval label, including when they appear in a comma-separated +list. Equivalent escaped or repeated-star forms are treated the same. Other +stage selectors, including explicit stage names and limited patterns such as +`"*PerfSanity*"`, retain their existing behavior. + +Being selective keeps CI turnaround fast and conserves hardware resources. ### Avoiding unnecessary `--disable-fail-fast` usage