Skip to content

FIX: Give scenario end-to-end tests time to start a scenario - #2512

Open
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/v1.1.0-Release-E2E-Fix
Open

FIX: Give scenario end-to-end tests time to start a scenario#2512
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/v1.1.0-Release-E2E-Fix

Conversation

@varunj-msft

Copy link
Copy Markdown
Contributor

Description

Scenarios in the end-to-end suite have been failing with an empty Error starting scenario: message.

Starting a scenario is a single request, and the server does the work before it answers: it runs the initializers, including loading each scenario's default datasets, resolves the target, and builds the scenario. That work grows with the number of registered scenarios and now exceeds the client's default read timeout of 60 seconds, so the client gives up while the server is still preparing the run. The message is empty because the underlying read timeout stringifies to nothing, so the report names neither the request that expired nor the timeout that governed it.

The timings confirm which timeout expired. Attempts are 150 seconds apart, which is the 60 second read timeout plus the 90 second delay between the flaky reruns, and the final attempt fails exactly 60 seconds after it starts because no delay follows it. The server answers other requests in the same session: the dataset tests pass before any scenario runs.

This passes an explicit, larger read timeout so the client waits for the setup it asked the server to do. It does not shrink the setup work itself — reducing what the default dataset initializer loads is still worth doing on its own, and would not remove the need for this timeout: measured locally with the large package registries excluded, the same setup still takes around seventy seconds, above the sixty second default.

Tests and Documentation

Test-only change to tests/end_to_end/test_scenarios.py; no product code is touched, and the --request-timeout option it passes already exists in pyrit_scan. The meaningful check is a live end-to-end run, which I plan to queue once this is on main, since that pipeline is scheduled only on main.

No notebooks are affected, so JupyText was not run.

Every scenario in the end-to-end suite has been failing since the middle of
August with an empty "Error starting scenario:" message. The message is empty
because the underlying read timeout stringifies to nothing, so the report
names neither the request that expired nor the timeout that governed it.

Starting a scenario is a single request, and the server does the work before
it answers: it runs the initializers, including loading each scenario's
default datasets, resolves the target, and builds the scenario. That work
grows with the number of registered scenarios, and it now exceeds the
client's default read timeout of 60 seconds, so the client gives up while the
server is still preparing the run. The empty message is the string form of
the resulting read timeout.

The timings confirm which timeout expired. Attempts are 150 seconds apart,
which is the 60 second read timeout plus the 90 second delay between the
flaky reruns, and the final attempt fails exactly 60 seconds after it starts
because no delay follows it. The server itself answers other requests in the
same session: the 108 dataset tests pass before any scenario runs.

Pass an explicit, larger read timeout for these tests so the client waits for
the setup it asked the server to do.

This makes the suite usable again but does not shrink the setup work itself.
Excluding the large package registries from the default dataset selection
would not remove the need for this timeout either: measured locally with
those registries excluded, the same setup still takes around seventy seconds,
above the sixty second default. The same growth also shows up as higher disk
use over a run, so reducing what the default dataset initializer loads
remains worth doing on its own.
@varunj-msft varunj-msft changed the title FIX Give scenario end-to-end tests time to start a scenario FIX: Give scenario end-to-end tests time to start a scenario Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants