INTERNAL: RST_STREAM closed stream. HTTP/2 error code: FLOW_CONTROL_ERROR

[santoshgit66]

Hello gRPC team,

I’m running into an intermittent issue with gRPC-Java 1.60.1 where calls suddenly fail with:

"INTERNAL: RST_STREAM closed stream. HTTP/2 error code: FLOW_CONTROL_ERROR"

• Has anyone encountered this "INTERNAL: RST_STREAM closed stream. HTTP/2 error code: FLOW_CONTROL_ERROR" before?
• What could be the most common root causes for an HTTP/2 flow-control error in a gRPC application?
• Are there any recommended workarounds or configuration settings I should try (e.g. window sizing)?

Thanks!

[shivaspeaks]

It would be great if you share client & server configurations and client & server logs.

Taking a wild guess, I expect the problem is you are sending too large of a request. channelBuilder.maxInboundMessageSize() only controls what response sizes are allowed. The server has to opt-in to receiving larger requests via serverBuilder.maxInboundMessageSize().

[santoshgit66]

It would be great if you share client & server configurations and client & server logs.

Taking a wild guess, I expect the problem is you are sending too large of a request. channelBuilder.maxInboundMessageSize() only controls what response sizes are allowed. The server has to opt-in to receiving larger requests via serverBuilder.maxInboundMessageSize().

Hi @shivaspeaks, Yes, the user is performing a heavy data operation in the application. However, as shown below, we’ve already set Integer.MAX_VALUE on both the client and server, yet we’re still seeing the error.

• ManagedChannelBuilder<?> channelBuilder = Grpc.newChannelBuilderForAddress(host, port).maxInboundMessageSize(Integer.MAX_VALUE);

• ServerBuilder<?> serverBuilder = Grpc.newServerBuilderForPort(port).maxInboundMessageSize(Integer.MAX_VALUE ).addServices(//Services)

Also, please check part of the log.

io.grpc.StatusRuntimeException: INTERNAL: RST_STREAM closed stream. HTTP/2 error code: FLOW_CONTROL_ERROR
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
at chs.grpc.grpcBlockingStub.getData(SomeService.java:15098)
at jdk.internal.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)

Do you think a proxy or gateway between the client and server could be overriding these settings and causing the FLOW_CONTROL errors? Please let me know if you still need more details.

Thanks!

[kannanjgithub]

The max inbound message size is for the whole of the gRPC message. The flow control error reported here occurs when the sender (client in this case) sends a frame larger than the receiving window size of the server. The window size updates and their honoring are handled by gRPC.

[kannanjgithub]

It is possible that an intermediate proxy overshoots the flow control window and the server resets the stream and the proxy propagates the error back to the gRPC client.
Is the receiver a gRPC Java server as well and what version? 1.60.1 is quite old although there haven't been many changes to the ok-http transport code in gRPC Java.