EmrStepSensor to auto detect stepid's #26809
Unanswered
pedresnyman
asked this question in
Ideas
Replies: 1 comment
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Description
Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.
Use case / motivation
If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.
What happened
We ran into a throttle limit.
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded
With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.
Beta Was this translation helpful? Give feedback.
All reactions