SharedLock ThreadLocal threadWriters memory leak may cause CPU usage to reach 100%

[huisman6]

Bug Report

Current Behavior

We have a long-running Spring Boot application in production, deployed on k8s , and occasionally one or two Pods experience 100% CPU usage.

After Heap Dump, it was found that most of the CPU was consumed inThreadLocalMap.expungeStaleEntry:

http-nio-8080-exec-64
  at java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(I)I ()
  at java.lang.ThreadLocal$ThreadLocalMap.remove(Ljava/lang/ThreadLocal;)V ()
  at java.lang.ThreadLocal.remove(Ljava/lang/Thread;)V ()
  at java.lang.ThreadLocal.remove()V ()
  at org.springframework.context.i18n.LocaleContextHolder.resetLocaleContext()V (LocaleContextHolder.java:70)
  at org.springframework.web.filter.RequestContextFilter.resetContextHolders()V (RequestContextFilter.java:120)
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(Ljakarta/servlet/http/HttpServletRequest;Ljakarta/servlet/http/HttpServletResponse;Ljakarta/servlet/FilterChain;)V (RequestContextFilter.java:103)

After further analysis, it was found that there were 23,776 ThreadLocal variables in the Tomcat Thread threadLocals Entries, most of which were io.lettuce.core.protocol.SharedLock$$Lambda, corresponding to the source code threadWriters , and too many ThreadLocal variables may cause the cleanup of StaleEntry to consume more CPU .

private final ThreadLocal<Integer> threadWriters = ThreadLocal.withInitial(() -> 0);

The value of the referent field for 20,827 ThreadLocalMap entries is io.lettuce.core.protocol.SharedLock threadWriters.

There are also thousands of ThreadLocalMap entries where the referent is null, the value is java.lang.Integer = 0. These are likely from threadWriters that have already been garbage collected. The cleanup of these stale entries consumes a significant amount of CPU.

Expected behavior/code

This code was introduced in version 6.4.0, and versions 6.4 and above are all affected by this ThreadLocal leak. The longer the process runs, the more likely it is to trigger 100% CPU usage.

Is it possible to modify threadWriters to static final to prevent ThreadLocal leaks?

private static final ThreadLocal<Integer> threadWriters = ThreadLocal.withInitial(() -> 0);

Environment

1. SpringBoot 3.4.8
2. Spring Data Redis 3.4.8
3. Lettuce 6.4.2.RELEASE , Netty 4.1.123.Final
4. JDK21 + Generational ZGC
5. Redis Version: 6.0

Lettuce is default based on the connection pool mode and does not share TCP connections: LettuceConnectionFactory.setShareNativeConnection(false).

the connection pool is apache commons-pool2, with a maximum number of connections: 200, a minimum number of idle connections: 15 , peak active connections: 60,
occasionally creating more TCP connections during a sudden surge of traffic, and closing them when idle.

[a-TODO-rov]

Hey, @huisman6
I am reviewing the scenario you described.
But in the meantime - is there a specific reason for you to use pooling ? We recommend using lettuce without pooling with sharing the native connection. That way you can get the most of it. Exception is if you need to use transactions. Check more here: https://redis.github.io/lettuce/advanced-usage/#is-connection-pooling-necessary

[huisman6]

Our application uses a Hybrid Cloud architecture, and the business has a high volume of requests to Redis. The peak Redis Command per second for a single application instance is around 5,000, and in most scenarios, mget is used, with an average of 300 keys per mget. We have observed that under high request volumes, using the shared native connection mode occasionally results in time-consuming spikes due to issues such as network packet loss or pipeline command or big keys causing Head Of Line, which affects application stability. Using the connection pool mode has better tolerance for "Head Of Line" and is relatively more stable.