Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1236379	dotnet/maui	Checkout dotnet/maui@refs/pull/33213/merge to s	Log	dotnet/maui#33213
1236352	dotnet/maui	Checkout dotnet/maui@main to s	Log
1235484	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5675/merge to s	Log	dotnet/arcade-services#5675
1235495	dotnet/roslyn	Checkout dotnet/roslyn@main to s	Log
1235450	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122616/merge to s	Log	dotnet/runtime#122616
1235458	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5673/merge to s	Log	dotnet/arcade-services#5673
1235301	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122607/merge to s	Log	dotnet/runtime#122607
1235446	dotnet/maui	Checkout dotnet/maui@refs/pull/33214/merge to s	Log	dotnet/maui#33214
1235381	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122495/merge to s	Log	dotnet/runtime#122495
1235426	dotnet/maui	Checkout dotnet/maui@refs/pull/33216/merge to s	Log	dotnet/maui#33216
1235415	dotnet/maui	Checkout dotnet/maui@refs/pull/33215/merge to s	Log	dotnet/maui#33215
1235378	dotnet/maui	Checkout dotnet/maui@refs/pull/33209/merge to s	Log	dotnet/maui#33209
1235395	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122639/merge to s	Log	dotnet/runtime#122639
1235334	dotnet/maui	Checkout dotnet/maui@refs/pull/33213/merge to s	Log	dotnet/maui#33213
1235337	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64749/merge to s	Log	dotnet/aspnetcore#64749
1235314	dotnet/runtime	Checkout dotnet/runtime@refs/pull/119992/merge to s	Log	dotnet/runtime#119992
1235300	dotnet/maui	Checkout dotnet/maui@refs/pull/28983/merge to s	Log	dotnet/maui#28983
1235299	dotnet/maui	Checkout dotnet/maui@refs/pull/28983/merge to s	Log	dotnet/maui#28983
1235260	dotnet/maui	Checkout dotnet/maui@refs/pull/33075/merge to s	Log	dotnet/maui#33075
1235198	dotnet/maui	Checkout dotnet/maui@refs/pull/33176/merge to s	Log	dotnet/maui#33176
1235202	dotnet/roslyn	Checkout dotnet/roslyn@main to s	Log
1235197	dotnet/maui	Checkout dotnet/maui@refs/pull/33176/merge to s	Log	dotnet/maui#33176
1235172	dotnet/maui	Checkout dotnet/maui@refs/pull/33208/merge to s	Log	dotnet/maui#33208
1235133	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122638/merge to s	Log	dotnet/runtime#122638
1235127	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122495/merge to s	Log	dotnet/runtime#122495
1235119	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122602/merge to s	Log	dotnet/runtime#122602
1235090	dotnet/aspire	Checkout dotnet/aspire@refs/pull/13352/merge to s	Log	dotnet/aspire#13352
1235086	dotnet/maui	Checkout dotnet/maui@refs/pull/31651/merge to s	Log	dotnet/maui#31651
1235083	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122090/merge to s	Log	dotnet/runtime#122090
1235080	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64749/merge to s	Log	dotnet/aspnetcore#64749
1235077	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64749/merge to s	Log	dotnet/aspnetcore#64749
1235045	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1235073	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5636/merge to s	Log	dotnet/arcade-services#5636
1235047	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52038/merge to s	Log	dotnet/sdk#52038
1235026	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122631/merge to s	Log	dotnet/runtime#122631
1235037	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122616/merge to s	Log	dotnet/runtime#122616
1235032	dotnet/maui	Checkout dotnet/maui@refs/pull/33177/merge to s	Log	dotnet/maui#33177
1235018	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1234479	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64380/merge to s	Log	dotnet/aspnetcore#64380
1233578	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16417/merge to s	Log	#16417
2862425	dotnet-dotnet-docker	Checkout dotnet-dotnet-docker@dev/loganbussell/netiso to s
1232258	dotnet/maui	Checkout dotnet/maui@refs/pull/33168/merge to s	Log	dotnet/maui#33168
1231699	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5657/merge to s	Log	dotnet/arcade-services#5657
1229553	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5643/merge to s	Log	dotnet/arcade-services#5643
1228455	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1228452	dotnet/runtime	Checkout dotnet/runtime@release/10.0 to s	Log
1227964	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122441/merge to s	Log	dotnet/runtime#122441
1227948	dotnet/runtime	Checkout dotnet/runtime@release/8.0 to s	Log
1227903	dotnet/maui	Checkout dotnet/maui@refs/pull/33076/merge to s	Log	dotnet/maui#33076
1227332	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122412/merge to s	Log	dotnet/runtime#122412
1226905	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52117/merge to s	Log	dotnet/sdk#52117
1226819	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122394/merge to s	Log	dotnet/runtime#122394
1226888	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52115/merge to s	Log	dotnet/sdk#52115
1225769	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/59191/merge to s	Log	dotnet/aspnetcore#59191
1222271	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52047/merge to s	Log	dotnet/sdk#52047
1221834	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122225/merge to s	Log	dotnet/runtime#122225
1221128	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51847/merge to s	Log	dotnet/sdk#51847
1221004	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64656/merge to s	Log	dotnet/aspnetcore#64656
1221002	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64656/merge to s	Log	dotnet/aspnetcore#64656
1221003	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64656/merge to s	Log	dotnet/aspnetcore#64656
2853848	dotnet-sdk	Clone dotnet/sdk	Log	#55829
1220360	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121389/merge to s	Log	dotnet/runtime#121389
1219486	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122159/merge to s	Log	dotnet/runtime#122159
1219306	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16364/merge to s	Log	#16364
1218998	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64539/merge to s	Log	dotnet/aspnetcore#64539
1219000	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64539/merge to s	Log	dotnet/aspnetcore#64539
1219001	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64539/merge to s	Log	dotnet/aspnetcore#64539
1217310	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64594/merge to s	Log	dotnet/aspnetcore#64594
1216606	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122083/merge to s	Log	dotnet/runtime#122083
1214953	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5578/merge to s	Log	dotnet/arcade-services#5578
1214933	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5584/merge to s	Log	dotnet/arcade-services#5584
1213665	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121664/merge to s	Log	dotnet/runtime#121664
1213649	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121591/merge to s	Log	dotnet/runtime#121591
1213644	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121589/merge to s	Log	dotnet/runtime#121589
1213657	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121676/merge to s	Log	dotnet/runtime#121676
1213637	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121588/merge to s	Log	dotnet/runtime#121588
1213641	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121587/merge to s	Log	dotnet/runtime#121587
1213010	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51795/merge to s	Log	dotnet/sdk#51795
1212984	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121697/merge to s	Log	dotnet/runtime#121697
1212983	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121697/merge to s	Log	dotnet/runtime#121697
1212931	dotnet/runtime	Checkout dotnet/runtime@refs/pull/121952/merge to s	Log	dotnet/runtime#121952
1212962	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1092/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1092
2846765	dotnet-sdk	Checkout dotnet-sdk@refs/pull/55515/merge to s	Log	#55515
2846742	dotnet-sdk	Checkout dotnet-sdk@refs/pull/55514/merge to s	Log	#55514

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	43	84

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?