"jobs_plugin_push: no such pipeline..." when deploying

[saxap]

Hello.

Our service runs in Kubernetes and uses the http and jobs plugins to operate.

When deploying we catch an error: "jobs_plugin_push: no such pipeline ..." in terminating pods.

Could the issue be related to the order of plugin shutdown?
According to the endure logs, the plugins are being shut down not in a strict order, and sometimes the jobs plugin is being stopped before the http one. Shutdown is graceful.

Or do you have any other ideas?

[rustatian]

Hey @saxap 👋🏻

Shutdown does not have a specific order; only startup does. You can get this error because of the following reasons:

1. The Jobs driver (attached to the Jobs plugin) has already destroyed the pipeline, and your request was sent after that, but before the Jobs plugin was destroyed.
2. If you're deploying RR, RPC could be opened, but not all pipelines initialized at that moment. That can happen if you have a high RPS system that is sending requests right after RPC is available.

[saxap]

Thank you for answer.

The problem appears only in terminating pods.

So there is nothing we can do on RR side?

[rustatian]

Yeah, that can happen when you're still pushing data to the pods while terminating them. You may first stop all pushers/senders/whatever that send data via RPC to this POD, and then terminate it.
And yeah, on the RR side, you can't do much with that.