RFC: Enhance Slow Store Scheduler to resolve network jitter problem #119
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: okJiang <[email protected]>
Signed-off-by: okJiang <[email protected]>
Signed-off-by: okJiang <[email protected]>
Signed-off-by: okJiang <[email protected]>
Signed-off-by: okJiang <[email protected]>
Signed-off-by: okJiang <[email protected]>
Signed-off-by: okJiang <[email protected]>
aravndr
reviewed
Jun 5, 2025
|
|
||
| Slow store scheduler is an existed scheduler in PD. Its current function is to detect whether the TiKV disk fails and quickly schedule the leaders away. Here is its steps: | ||
|
|
||
| 1. Each TiKV collects the timeout I/O requests locally (during each inspect-interval) |
There was a problem hiding this comment.
Does TIKV collect the timeout I/O requests for all other TIKV peers?
Member
Author
There was a problem hiding this comment.
In the first delivery version, it will not be collected, but in the second delivery version, it will be collected.
| #### TiKV <-> TiKV health check | ||
|
|
||
| When a tikv has a network problem, other tikvs will also experience request timeouts when accessing the problematic tikv. In this case, the slow score of the normal tikv may also increase, and if it lasts for a long time, it may even reach 100, triggering the slow store mechanism. | ||
| To avoid this problem, we can send the collected health information and its corresponding store_id to PD. If the `TimeoutRatio` of a node exceeds RatioMaxThresh, it is considered that the network of the node has a problem, and PD will filter out the request timeouts from normal nodes and then calculate the `TimeoutRatio`. |
There was a problem hiding this comment.
Could you explain how PD will filter the slow score elevation on the normal nodes?
Member
Author
There was a problem hiding this comment.
In the tikv <-> tikv health check mechanism, we calculate the score for each health check object in each node and feed all the scores back to PD. PD knows which two nodes the score comes from. Based on the distribution of all the scores, PD is able to distinguish which node has a problem.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ref tikv/pd#9359