clientv3: backoff resetting LeaseKeepAlive stream #20718
Conversation
|
Hi @elias-dbx. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
|
||
| // retryConnWait is how long to wait before retrying request due to an error | ||
| retryConnWait = 500 * time.Millisecond | ||
| // retryConnMinBackoff is the starting backoff when retrying a request due to an error |
There was a problem hiding this comment.
How were these values chosen?
There was a problem hiding this comment.
They were chosen to be in line with the default exponential backoff parameters of other widely used client side libraries. For example, the aws-sdk-go-v2 library has a default max backoff of 20 seconds: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/retry/standard.go#L31
I kept the initial backoff to 500ms as that is the current backoff time.
|
/ok-to-test |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 18 files with indirect coverage changes @@ Coverage Diff @@
## main #20718 +/- ##
==========================================
+ Coverage 69.12% 69.28% +0.16%
==========================================
Files 422 422
Lines 34826 34838 +12
==========================================
+ Hits 24073 24138 +65
+ Misses 9352 9307 -45
+ Partials 1401 1393 -8 Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: elias-dbx, ronaldngounou The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest-required |
A large number of client leases can cause cascading failures within the etcd cluster. Currently, when the keepalive stream has an error we will always wait 500ms and then try to recreate the stream with LeaseKeepAlive(). Since there is no backoff or jitter, if the lease streams originally broke due to overload on the servers the retries can cause a cascading failure and put more load on the servers. We can backoff and jitter -- similar to what is done in watch streams -- in order to alleviate server load in the case where leases are causing the overload. Signed-off-by: Elias Carter <[email protected]>
elias-dbx
force-pushed
the
lease_backoffs
branch
from
54153a4 to
84e84de
Compare
3 participants
A large number of client leases can cause cascading failures within the etcd cluster. Currently, when the keepalive stream has an error we will always wait 500ms and then try to recreate the stream with LeaseKeepAlive(). Since there is no backoff or jitter, if the lease streams originally broke due to overload on the servers the retries can cause a cascading failure and put more load on the servers.
We can backoff and jitter -- similar to what is done in watch streams -- in order to alleviate server load in the case where leases are causing the overload.
Related to: #20717