Question on the DirectMemory Recycle Logic in OutputBufferSize of Jetty #13816
-
|
Jetty Version Jetty Environment ee10 Java Version Question
<New id="commonHttpConfig" class="org.eclipse.jetty.server.HttpConfiguration">
<Set name="secureScheme">https</Set>
<Set name="sendServerVersion">false</Set>
<Set name="outputBufferSize" type="int">
<Property name="jetty.outputBufferSize" default="32768"/>
</Set>
</New>
|
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 2 replies
-
|
You likely are constraining the memory too much, with your command line options, for the request rate you are imposing. You can configure Jetty's buffer pool with: The dump will detail how the buffer pool is being used. Furthermore, you can constrain the memory usage of the buffer pool by specifying the max amount of direct memory that the buffer pool will retain, using this constructor |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @sbordet replay, becauseI cannot expose the JMX remote access from my local. I setup the same traffic model but with lower tps in my local env and capture the MBean output. Please find below. |
Beta Was this translation helpful? Give feedback.
-
|
From the output, our application mainly use the bucket 4096 (InputBuffer for ReadListener) and 32768(Output for WriteListener), may I know will the buffer size keep increasing in these 2 buckets if the TPS keep increasing and encounter request timeout somehow. |
Beta Was this translation helpful? Give feedback.
-
|
Besides JMX MBean, may I know if any other way to spot the buffer in use? |
Beta Was this translation helpful? Give feedback.
-
|
From the buffer pool dump, it shows only very few acquires, so either this is a different pool that is not used much, or you have not run your load test. Consider that it may not be Jetty using the direct memory, but it may be some other component in your application. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @sbordet , finally, I dump the ArrayByteBufferPool, could you help to hv a look if any issue on that? Thank you in advanced. |
Beta Was this translation helpful? Give feedback.
-
|
The dump above shows that Jetty is only using 64.9 MiB of direct memory, out of a max of 268.4 MiB. Unfortunately, you have cut the dump before the line that reports the non-pooled acquisitions, so it is not possible to tell more. You have configured 1 GiB of max direct memory, so it does not seem it is Jetty causing your OOM problems. |
Beta Was this translation helpful? Give feedback.
-
|
A hypothesis of what may happen is that you are not ramping up the load at the beginning of your load test. As the execution is slower because the code is not yet JIT compiled, a larger number of buffer acquires arrives to the buffer pool, which will allocate them in excess to cope with the demand. If you perform a proper warm-up, and you ramp your load test, do you still see the OOME? |
Beta Was this translation helpful? Give feedback.
You likely are constraining the memory too much, with your command line options, for the request rate you are imposing.
You can configure Jetty's buffer pool with:
ArrayByteBufferPool.setStatisticsEnabled(true), and after a while you can take a server dump as explained here:https://jetty.org/docs/jetty/12.1/programming-guide/troubleshooting/component-dump.html
The dump will detail how the buffer pool is being used.
Furthermore, you can constrain the memory usage of the buffer pool by specifying the max amount of direct memory that the buffer pool will retain, using this constructor
https://javadoc.jetty.org/jetty-12.1/org/eclipse/jetty/io/ArrayByteBufferPool.html#%3Cinit%3E(int,int,int,i…