Start task failure: error pulling image: 503 The server is busy

[lachiemurray]

When provisioning a new cluster with more than 50 nodes we start to see a large proportion of start task failures (25+%) when pulling the docker image from an ACR.

error pulling image
configuration: received unexpected HTTP status: 503 The server is busy

I raised a support issue with the ACR team and they said we were being throttled and recommend that we attempt to retry pulling the image when we get a 503.

Note our image is quite large (~7 GB) which may be why we experience this issue whilst other do not.

Here are some of the ways I think we could mitigate this:

1. Retry pulling the image when 503 returned (as recommended by the ACR team)

2. Retry the entire start-task when any failure occurs (it looks like there is support for this in the batch SDK?)

3. Configure the docker daemon to pull fewer layers in parallel using the max-concurrent-downloads option. I looked at whether this could be done using a plugin but I think plugins would be run too late?

What do you think the best approach would be? Can you recommend any others?

[jafreck]

Thanks for reporting this and doing the investigations into solutions. A couple thoughts:

1. retrying image pull specifically is worth enabling in the event of server-side 500 errors. This should be enabled within the start task itself.
2. max_task_retry_count is the Batch feature. This should be enabled with a sensible default for any number of transient errors including docker pull fails. It might be worthwhile exposing this to users as well.
3. You are correct by saying plugins runs too late. So I think the above two solutions are the right way to go. Eventually AZTK will migrate to the Batch container feature (Internal: use native batch container feature #646), and with that we will likely lose some of the ability to fine tune the docker run time.