Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/bot-command.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -52,14 +52,14 @@ jobs:
"`--disable-reuse-test ` *(OPTIONAL)* : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.\n\n" +
"`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.\n\n" +
"`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.\n\n" +
"`--stage-list \"A10-PyTorch-1, xxx\"` *(OPTIONAL)* : Only run the specified test stages. Supports wildcard `*` for pattern matching (e.g., `\"*PerfSanity*\"` matches all stages containing PerfSanity). Examples: \"A10-PyTorch-1, xxx\", \"*PerfSanity*\". Note: Does **NOT** update GitHub check status.\n\n" +
"`--stage-list \"A10-PyTorch-1, xxx\"` *(OPTIONAL)* : Only run the specified test stages. Supports wildcard `*` for pattern matching (e.g., `\"*PerfSanity*\"` matches all stages containing PerfSanity). Examples: \"A10-PyTorch-1, xxx\", \"*PerfSanity*\". The broad patterns `\"*\"` and `\"*Post-Merge*\"`, including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the `ci: post-merge approved` PR label. Note: Does **NOT** update GitHub check status.\n\n" +
"`--gpu-type \"A30, H100_PCIe\"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: \"A30, H100_PCIe\". Note: Does **NOT** update GitHub check status.\n\n" +
"`--test-backend \"pytorch, cpp\"` *(OPTIONAL)* : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: \"pytorch, cpp\" (does not run test stages with tensorrt or triton backend). Note: Does **NOT** update GitHub pipeline status.\n\n" +
"`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.\n\n" +
"`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.\n\n" +
"`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.\n\n" +
"`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.\n\n" +
"`--extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard `*` for pattern matching. Examples: --extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\", --extra-stage \"*Post-Merge*\".\n\n" +
"`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the `ci: post-merge approved` PR label applied by an active member of `NVIDIA/trt-llm-ci-approvers`. The approval label remains in place when new commits are pushed.\n\n" +
"`--extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard `*` for pattern matching. Examples: --extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\", --extra-stage \"*Post-Merge*\". The broad patterns `\"*\"` and `\"*Post-Merge*\"`, including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the `ci: post-merge approved` PR label.\n\n" +
"`--detailed-log ` *(OPTIONAL)* : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.\n\n" +
"`--debug ` *(OPTIONAL)* : **Experimental feature**. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the `stage-list` parameter to access the appropriate container environment. Note: Does **NOT** update GitHub check status.\n\n" +
"`--high-priority ` *(OPTIONAL)* : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.\n\n" +
Expand Down
217 changes: 217 additions & 0 deletions .github/workflows/post-merge-approval.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Guard Post-Merge Approval Label

on:
# Intentional: this runs static default-branch API logic only. It never checks
# out or executes PR code; it uses GitHub APIs to validate membership and
# manage this PR's approval label/comment.
pull_request_target:
types: [labeled]

permissions:
contents: read
pull-requests: write
Comment thread
coderabbitai[bot] marked this conversation as resolved.

jobs:
guard-post-merge-approval:
concurrency:
group: post-merge-approval-${{ github.event.pull_request.number }}
cancel-in-progress: false
if: >-
github.repository == 'NVIDIA/TensorRT-LLM' &&
github.event.action == 'labeled' &&
github.event.label.name == 'ci: post-merge approved'
runs-on: ubuntu-latest
steps:
- name: Validate post-merge approver
id: validate
if: github.event.action == 'labeled'
uses: actions/github-script@v8
with:
github-token: ${{ secrets.TRTLLM_AGENT_SHARED_TOKEN }}
result-encoding: string
script: |
const approvalLabel = 'ci: post-merge approved';
const owner = context.repo.owner;
const repo = context.repo.repo;
const issueNumber = context.payload.pull_request.number;
let actor = context.payload.sender?.login || context.actor;
let labelEventId = '';
let timelineVerified = false;

try {
const events = await github.paginate(
github.rest.issues.listEventsForTimeline,
{ owner, repo, issue_number: issueNumber, per_page: 100 }
);
const latestApprovalEvent = events
.filter(
(event) =>
event.event === 'labeled' &&
event.label?.name === approvalLabel
)
.at(-1);
if (latestApprovalEvent) {
actor = latestApprovalEvent.actor?.login || actor;
labelEventId = String(latestApprovalEvent.id);
timelineVerified = true;
} else {
core.warning('Could not identify the latest post-merge approval event.');
}
} catch (error) {
core.warning(
'Could not read the latest post-merge approval event: ' + error.message
);
}

core.setOutput('validated_actor', actor);
core.setOutput('label_event_id', labelEventId);
if (!timelineVerified) {
return 'false';
}
try {
const response = await github.request(
'GET /orgs/{org}/teams/{team_slug}/memberships/{username}',
{
org: 'NVIDIA',
team_slug: 'trt-llm-ci-approvers',
username: actor,
}
);
const authorized = response.data.state === 'active';
console.log(
actor + ' active membership in NVIDIA/trt-llm-ci-approvers: ' + authorized
);
return authorized ? 'true' : 'false';
} catch (error) {
if (error.status === 404) {
console.log(
actor + ' is not an active member of NVIDIA/trt-llm-ci-approvers.'
);
} else {
core.warning(
'Could not verify post-merge approver ' + actor + ': ' + error.message
);
}
return 'false';
}

- name: Clear unauthorized post-merge approval
if: always() && !cancelled() && steps.validate.outputs.result != 'true'
uses: actions/github-script@v8
env:
VALIDATED_ACTOR: ${{ steps.validate.outputs.validated_actor }}
VALIDATED_LABEL_EVENT_ID: ${{ steps.validate.outputs.label_event_id }}
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const approvalLabel = 'ci: post-merge approved';
const actor =
process.env.VALIDATED_ACTOR ||
context.payload.sender?.login ||
context.actor;
const validatedEventId = process.env.VALIDATED_LABEL_EVENT_ID || '';
const owner = context.repo.owner;
const repo = context.repo.repo;
const issueNumber = context.payload.pull_request.number;

try {
const pullRequest = await github.rest.pulls.get({
owner,
repo,
pull_number: issueNumber,
});
const labelIsPresent = pullRequest.data.labels.some(
(label) => label.name === approvalLabel
);
if (!labelIsPresent) {
console.log('Post-merge approval label is already absent; no cleanup needed.');
return;
}
} catch (error) {
core.warning(
'Could not read the current post-merge approval label state; continuing validation: ' +
error.message
);
}

try {
const events = await github.paginate(
github.rest.issues.listEventsForTimeline,
{ owner, repo, issue_number: issueNumber, per_page: 100 }
);
const latestApprovalEvent = events
.filter(
(event) =>
event.event === 'labeled' &&
event.label?.name === approvalLabel
)
.at(-1);
const latestEventId = latestApprovalEvent
? String(latestApprovalEvent.id)
: '';
const latestActor = latestApprovalEvent?.actor?.login || '';
if (!latestApprovalEvent) {
core.warning(
'Could not identify the latest post-merge approval event; removing the label to fail closed.'
);
} else if (!latestActor) {
core.warning(
'Could not identify the latest post-merge approval actor; removing the label to fail closed.'
);
} else if (!validatedEventId) {
core.warning(
'The validation run did not bind an approval event; removing the label to fail closed.'
);
} else if (
latestEventId !== validatedEventId ||
latestActor !== actor
) {
console.log(
'A newer post-merge approval event was found; leaving it for its own validation run.'
);
return;
}
} catch (error) {
core.warning(
'Could not re-check the latest post-merge approval event; removing the label to fail closed: ' +
error.message
);
}

try {
await github.rest.issues.removeLabel({
owner,
repo,
issue_number: issueNumber,
name: approvalLabel,
});
} catch (error) {
if (error.status !== 404) {
throw error;
}
}

await github.rest.issues.createComment({
owner,
repo,
issue_number: issueNumber,
body:
'Removed the "' + approvalLabel + '" label because @' + actor +
' could not be verified as an active member of ' +
'NVIDIA/trt-llm-ci-approvers. Ask a member of that team to apply it.',
});
Comment thread
chzblych marked this conversation as resolved.
21 changes: 17 additions & 4 deletions docs/source/developer-guide/ci-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,15 @@ developers understand why the test is disabled.

### Triggering Post-merge tests

When you only need to verify a handful of post-merge tests, avoid the heavy
`/bot run --post-merge` command. Instead, specify exactly which stages to run:
Full `/bot run --post-merge` runs require the `ci: post-merge approved` PR
label because they can consume substantial shared GPU resources. The label must
be applied by an active member of the `NVIDIA/trt-llm-ci-approvers` GitHub team.
A GitHub workflow removes invalid approvals. The label remains in place when new
commits are pushed and can be removed manually when the approval no longer
applies.

When you only need to verify a handful of post-merge tests, specify exactly
which stages to run:

```bash
/bot run --stage-list "stage-A,stage-B"
Expand All @@ -123,8 +130,14 @@ default pre-merge set:
/bot run --extra-stage "stage-A,stage-B"
```

Both options accept any stage name defined in `jenkins/L0_Test.groovy`. Being
selective keeps CI turnaround fast and conserves hardware resources.
Both options accept stage names and wildcard patterns defined in
`jenkins/L0_Test.groovy`. The broad `"*"` and `"*Post-Merge*"` selectors
require the same approval label, including when they appear in a comma-separated
list. Equivalent escaped or repeated-star forms are treated the same. Other
stage selectors, including explicit stage names and limited patterns such as
`"*PerfSanity*"`, retain their existing behavior.

Being selective keeps CI turnaround fast and conserves hardware resources.

### Avoiding unnecessary `--disable-fail-fast` usage

Expand Down
Loading