Add autorun_on_main into CI pipeline

[hl475]

The goal of this PR is to add a new feature - if a test/label has autorun_on_main as True AND the PR has been merged (so running on main branch)if a test/label has ci_always_run as True AND the PR has been marked as ready, then the corresponding test/label will run automatically (so no need to manually click to unblock).

The hope of the feature is adding some selected tests/labels to run automatically on PR to maintain trunk health. Some alternatives could be (1) merge queue, (2) new symbol other than ready. I am open to alternative, but it could change user experience a bit. Previously we came up with ci_always_run, but it will run per rebase once ready, which might still be expensive.

[khluu]

Is this only on CI? (I'm asking because I saw you listed a fastcheck build in description, but the change is made to just test-template-ci.j2). Fastcheck currently only runs a set of 4-5 tests.

Instead of autorun_on_ready, can we name it something else like always_run or ci_always_run and add a description for it in the list of args in https://github.com/vllm-project/vllm/blob/main/.buildkite/test-pipeline.yaml#L8? We are currently associating ready label with launching CI, but that might change in the future.

[hl475]

Thanks @khluu !

Is this only on CI? (I'm asking because I saw you listed a fastcheck build in description, but the change is made to just test-template-ci.j2). Fastcheck currently only runs a set of 4-5 tests.

I want to make it only for CI not fastcheck - I used fastcheck pipeline to simulate the scenarios where ready label is not being applied, and ci pipeline when ready is being applied. Let me know if this is wrong!

Instead of autorun_on_ready, can we name it something else like always_run or ci_always_run and add a description for it in the list of args in https://github.com/vllm-project/vllm/blob/main/.buildkite/test-pipeline.yaml#L8?

Sure, I can rename it! and add the doc into test-pipeline.yaml together with this PR

[khluu]

Thanks @hl475 ! Let's merge the PR to add the field on vllm repo first, then we can merge this one.

[hl475]

Hi @khluu , per today's CI meeting discussion, I repurposed this PR a bit by introducing autorun_on_main instead of ci_always_run - the test will run automatically once merged into main branch.

I need to update the corresponding PR on vLLM side, but want to get some of your thoughts on this. Thanks!