CodegenException when using TomcatEmbeddedWebappClassLoader (fx. Spring Boot)

[michaelkrog]

I am having an issue wen running my application with QueryDSL on OpenJDK.

I have 2 classes that I generate QueryDsl classes for. One(Project) is a MongoEntity class that I annotated with @QueryEntity and the other(Service) is a POJO also with @QueryEntity.

I use com.querydsl.apt.QuerydslAnnotationProcessor for generating the classes.

When I compile and run it locally everything works fine. This is using the following JDK:

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

But when built for production using this JDK:

java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

...and then run in a container with this JDK:

openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

...then I get this exception for the Service class, but not the Project class:

com.mysema.codegen.CodegenException: Compilation of public class Q_2124358794_1275614662_1275614662_1195259493 {

    public static Iterable<dk.codezoo.meerkat.model.Service> eval(Iterable<dk.codezoo.meerkat.model.Service> service_, String a1) {
java.util.List<dk.codezoo.meerkat.model.Service> rv = new java.util.ArrayList<dk.codezoo.meerkat.model.Service>();
for (dk.codezoo.meerkat.model.Service service : service_) {
    try {
        if (service.getName().toLowerCase().contains(a1)) {
            rv.add(service);
        }
    } catch (NullPointerException npe) { }
}
return rv;    }

}

 failed.
warning: Supported source version 'RELEASE_6' from annotation processor 'org.jvnet.hudson.annotation_indexer.AnnotationProcessorImpl' less than -source '1.8'
/Q_2124358794_1275614662_1275614662_1195259493.java:3: error: package dk.codezoo.meerkat.model does not exist
    public static Iterable<dk.codezoo.meerkat.model.Service> eval(Iterable<dk.codezoo.meerkat.model.Service> service_, String a1) {
                                                                                                   ^
/Q_2124358794_1275614662_1275614662_1195259493.java:3: error: package dk.codezoo.meerkat.model does not exist
    public static Iterable<dk.codezoo.meerkat.model.Service> eval(Iterable<dk.codezoo.meerkat.model.Service> service_, String a1) {
                                                   ^
2 errors
1 warning

	at com.mysema.codegen.JDKEvaluatorFactory.compile(JDKEvaluatorFactory.java:74)
	at com.mysema.codegen.AbstractEvaluatorFactory.createEvaluator(AbstractEvaluatorFactory.java:128)
	at com.querydsl.collections.DefaultEvaluatorFactory.createEvaluator(DefaultEvaluatorFactory.java:151)
	at com.querydsl.collections.DefaultQueryEngine.evaluateSingleSource(DefaultQueryEngine.java:166)
	at com.querydsl.collections.DefaultQueryEngine.list(DefaultQueryEngine.java:80)
	at com.querydsl.collections.AbstractCollQuery.fetch(AbstractCollQuery.java:182)
	at dk.codezoo.meerkat.orchestration.DockerOrchestrationClient$DockerServiceClient.findAll(DockerOrchestrationClient.java:108)
	at dk.codezoo.meerkat.service.ServiceService.findAll(ServiceService.java:64)
	at dk.codezoo.meerkat.controller.ServiceController.list(ServiceController.java:62)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

[Shredder121]

package dk.codezoo.meerkat.model does not exist

Can you check this? Is it only happening with those JDKs?

[michaelkrog]

That package does exist. I have tried to unpack the jar just to check it, and it is there.
The two classes are both in the package. One works the other does not.

I just tried to force our build process to update the docker image it uses(openjdk:8-jdk), and that changed the JDK to this:

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

But to no avail. It fails with the same exception.

But I am only seeing it when I run it on OpenJDK.

[michaelkrog]

Okay. I just extracted the Jar built for production and ran that locally... And it fails...
So its not only OpenJDK.

Weird. If I built it with maven locally on 1.8.0_121(mac) and run it locally, there is not issue. If built it with 1.8.0_161(linux) and I run that locally, then it fails.

[michaelkrog]

I upgraded my mac to JDK 1.8.0_161 and still I am unable to reproduce it when I build i locally.

I then tried to login to our build server(JDK 1.8.0_161 linux), checked out the code, did a fresh build(mvn clean install) and copied the jar-file to my local machine and it fails.

So weird... I'll try to setup a virtual linux locally and reproduce it there.

[Shredder121]

Yes please do, I can't fathom where it would go wrong but can go right in some cases.
So I'd like to hear any progress!

[michaelkrog]

I think I found the source of the issue: classpath.

This is the start of the resolved classpath when I run the project via spring-boot:run from intellij:

/Users/michael/Development/meerkat/meerkat/target/classes:/Users/michael/.m2/repository/org/springframework/boot/spring-boot-starter-web/1.5.3.RELEASE/spring-boot-starter-web-1.5.3.RELEASE.jar:/Users/michael/.m2/repository/org/springframework/boot/spring-boot-starter/1.5.3.RELEASE/spring-boot-starter-1.5.3.RELEASE.jar:/...

And this is the start of the classpath when I run the final artifact locally from the commandline (java -jar meerkat.jar):

/Users/michael/Development/meerkat/meerkat/file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/classes!:/Users/michael/Development/meerkat/meerkat/file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/lib/spring-boot-starter-web-1.5.3.RELEASE.jar!:/Users/michael/Development/meerkat/meerkat/file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/lib/spring-boot-starter-1.5.3.RELEASE.jar!:/Users/michael/Development/meerkat/meerkat/file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/lib/spring-boot-1.5.3.RELEASE.jar!:...

Weird classpath

/Users/michael/Development/meerkat/meerkat/file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/classes! looks a bit weird, doesn't it?

[michaelkrog]

It comes down to

codegen/src/main/java/com/mysema/codegen/SimpleCompiler.java

Lines 71 to 72 in 60b65f9

	String decodedPath = URLDecoder.decode(url.getPath(), "UTF-8");
	paths.add(new File(decodedPath).getAbsolutePath());

The first URL retrieved in my case is jar:file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/classes!/

At line 71 it is being decoded to file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/classes!/

At line 72 the absolute path becomes /Users/michael/Development/meerkat/meerkat/file:/Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/classes!

[michaelkrog]

I am unable to make it work.
SimpleCompiler.getClassPath(parent); at https://github.com/querydsl/codegen/blob/master/src/main/java/com/mysema/codegen/JDKEvaluatorFactory.java#L57 returns a classpath that does not work when the application is run as an executable jar built with Spring Boot (https://docs.spring.io/spring-boot/docs/current/reference/html/build-tool-plugins-maven-plugin.html).

The classloader is a TomcatEmbeddedWebappClassLoader which resolves jars in the executable jar(Fx. /Users/michael/Development/meerkat/meerkat.jar!/BOOT-INF/lib/spring-boot-starter-tomcat-1.5.3.RELEASE.jar!). But the classpath returned from SimpleCompiler is then broken and classes can then not be resolved.

My knowledge about JavaCompiler and the codegen project is to sparse. I have been unable to create a fix.

[Shredder121]

Yes the issue is known, I'll continue with my fix

[Shredder121]

Thanks for looking into it though, much appreciated!

[matteo-gallo-bb]

@Shredder121 I just created a PR to add the support for Codegen to generate code when running in a fat JAR. It should fix this bug and others related to this.
I'm using this fix and it works for me with Spring Boot, Spring Data Key Value, spring-data-hazelcast and QueryDsl Collections.
The PR is: #43
If you can, please review it and let me know if there are possible improvements to do.