Operator could release lock voluntarily #2315
Description
What would you like to be added/changed?
In our test_operator_velocity test suite we sometimes see failures because a single operator instance is holding a lock for a long time. Right now the operator instance holding the lock will wait until the FoundationDBCluster is fully reconcile before releasing the lock. This can be blocked or delayed for multiple reasons like slow scheduling of pods or when additional capacity is added to the Kubernetes cluster. One way to reduce the risk that a single operator instance is holding a lock for a long time, would be to release the lock voluntarily after a given duration. That way the other operator instances have a chance to take the lock and continue with their operations.