[CI] Introduce autorun_on_main feature

[hl475]

Purpose

Corresponding PR on vllm-project/ci-infra - vllm-project/ci-infra#200

This PR adds autorun_on_main: true into test label: LM Eval Small Models label: LM Eval Large Models (H100). As explained from PR on ci-infra side, we want to maintain limited/heavy tests to guard the trunk health at some degree.

The goal of this PR is to add a new feature - if a test/label has autorun_on_main as True AND the PR has been merged into main, then the corresponding test/label will run automatically (so no need to manually click to unblock).

The hope of the feature is adding some selected tests/labels to run automatically on PR to maintain trunk health. Some alternatives could be (1) merge queue, (2) new symbol other than ready. I am open to alternative, but it could change user experience a bit.

For any tests on broad to autorun_on_main, it will likely increase the CI cost. Please keep this in mind.

Test Plan

CI

Test Result

https://buildkite.com/vllm/ci/builds/38581/steps/canvas?sid=019a7696-a611-4e2f-8ce6-d0b873f4f5fa

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[hl475]

cc @yeqcharlotte @khluu

[khluu]

I'm not sure if the H100 queue capacity is enough to run per commit on main...

[hl475]

add autorun_on_main: true into label: LM Eval Small Models per suggestion

[yeqcharlotte]

Let's see how it goes!