[ci-scan] Build break: Windows Helix work items fail on unexpanded $HELIX_PYTHONPATH in gen-debug-dump-docs post-command

[github-actions]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1524472
Build error leg or test failing: Build / android-arm64 Release NativeAOT / Send to Helix — Windows Helix work items (e.g. System.Security.Cryptography.Tests, System.Net.Sockets.Tests)
Pull request: (rolling main build, not a PR build)

Error Details

gen-debug-dump-docs.py invokes the shell
variable $HELIX_PYTHONPATH using Unix syntax. On Windows work items cmd.exe does not
expand $HELIX_PYTHONPATH, so the command is not found and the work item exits with code 1
after the test run itself has already completed and reported results. Every Windows Helix
work item on the affected legs of runtime-extra-platforms fails this way, and the failure
recurs across consecutive builds (source build 1524472 and follow-up build 1526281). -->

D:\h\w\...\e>$HELIX_PYTHONPATH $HELIX_CORRELATION_PAYLOAD/gen-debug-dump-docs.py -buildid 1524472 -workitem $HELIX_WORKITEM_FRIENDLYNAME -jobid $HELIX_CORRELATION_ID -outdir $HELIX_WORKITEM_UPLOAD_ROOT -templatedir $HELIX_CORRELATION_PAYLOAD -dumpdir $HELIX_DUMP_FOLDER -productver 11.0.0
'$HELIX_PYTHONPATH' is not recognized as an internal or external command,
operable program or batch file.

D:\h\w\...\e>EXIT /b 1

Error Message

{
  "ErrorMessage": "'$HELIX_PYTHONPATH' is not recognized as an internal or external command",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

---

Filed by ci-failure-scan (detection only). ci-failure-fix walks open [ci-scan] KBEs and either opens a small fix PR or comments here to loop in owners — it never disables the test.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by CI Outer-Loop Failure Scanner · 360.6 AIC · ⌖ 15.3 AIC · ⊞ 18.6K · ◷

Report

Build	Repository	Test	Pull Request
1542078	dotnet/runtime	System.Net.Security.Unit.Tests.WorkItemExecution
1541801	dotnet/runtime	System.Net.WebSockets.Client.Tests.CancelTest_HttpClient_Loopback.SendAsync_Cancel_Success	#131908
1541017	dotnet/runtime	System.Net.Http.Functional.Tests.HttpClientHandlerTest_BasicAuth.RefreshesPreAuthCredentialsOnChange	#131876
1540640	dotnet/runtime	System.Net.Http.Functional.Tests.NtAuthTests.Http2_SessionAuthChallenge_SubsequentRequestsUseHttp11	#131862
1540423	dotnet/runtime	System.Net.Security.Unit.Tests.WorkItemExecution
1540188	dotnet/runtime	System.Net.WebSockets.Client.Tests.CancelTest_HttpClient_Http2Loopback.SendAsync_Cancel_Success	#131835
1538577	dotnet/runtime	System.Net.Ping.Functional.Tests.WorkItemExecution
1538004	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#131765
1536699	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1536279	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1535640	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1535116	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution	#131544
1534164	dotnet/runtime	System.Net.Ping.Functional.Tests.WorkItemExecution
1533212	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#131589
1532901	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#131569
1532615	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1532241	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#131552
1531891	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution	#131544
1531558	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#131275
1531367	dotnet/runtime	System.Net.Ping.Functional.Tests.WorkItemExecution	#131351
1530895	dotnet/runtime	System.Net.Ping.Functional.Tests.WorkItemExecution
1530106	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#131482
1529233	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1528851	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1528352	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#125436
1527467	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1527196	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#131375
1526936	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	15	28

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[matouskozak]

I see the same failure on maccatalyst x64 jobs

+ /usr/local/Cellar/python@3.14/3.14.6/Frameworks/Python.framework/Versions/3.14/bin/python3.14 /tmp/helix/working/B75209F8/p/gen-debug-dump-docs.py -buildid 1529233 -workitem IcuAppLocal.Tests -jobid b63e5a2a-ba0c-4583-91ba-ad08e7f64b7a -outdir /tmp/helix/working/B75209F8/w/AD2F0948/uploads -templatedir /tmp/helix/working/B75209F8/p -dumpdir /cores -productver 11.0.0
/usr/local/Cellar/python@3.14/3.14.6/Frameworks/Python.framework/Versions/3.14/Resources/Python.app/Contents/MacOS/Python: can't open file '/tmp/helix/working/B75209F8/p/gen-debug-dump-docs.py': [Errno 2] No such file or directory
+ exit 143

https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-b63e5a2aba0c458391/IcuAppLocal.Tests/1/console.cb7b36f7.log?helixlogtype=result