Weblogic 10.3 hangs on JNDI data source lookup during deployment

I have been working on a Spring based web application for a couple weeks. This application uses Hibernate Generic DAO’s with a JNDI template via spring to create the data source. This procedure works very well with DBUnit and Spring to test all of my data access objects in isolation and to get very high code coverage and am very excited about the health of this application.

After testing with DBUnit, I wanted to run Selenium functional tests in a container, so I choose Jetty to use locally as it was much faster for rapid development, and so far, have proven to work very well and has integrated seamlessly into my Maven build, Eclipse IDE and my entire Continuous Integration flow.

So comes time to deploy to our development container, WebLogic 10.3. I was going to use the WebLogic deploy plugin through Maven, but first I wanted to ensure the application would deploy manually. Unfortunately, I immediately had deployment issues with a well tested application. The deployments would hang for 45 minutes to an hour in a processing state, before finally failing the deployment.

The initial symptoms where already detailed on only one post I could find anywhere:

http://stackoverflow.com/questions/770438/hibernate-jpa-inheritance-issue-on-weblogic

[ERROR] Javassist Enhancement failed: com.xxx.domain.User
java.lang.NoSuchMethodError: pcGetManagedFieldCount

at com.xxx.domain.User.<clinit>(User.java)
at sun.misc.Unsafe.ensureClassInitialized(Native Method)
at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:25)

So I started to track down that symptom. The 1st place I looked after Stackoverflow, was BEA naturally. I found a change request #CR370788 on http://edocs.bea.com/wls/docs103/issues/known_resolved.html that talks about the issue:

In previous WLS releases, it was possible that some classes could be loaded without enhancement if the module in which the persistence.xml was defined was declared after some of the dependent modules referencing those persistent classes (for example, if SB.jar is dependent on persistent classes defined in domain.jar, and domain.jar is declared after SB.jar in an EAR). In this case, the dependent module which does not have the persistence.xml declared in it gets loaded, and the persistent entity classes are not enhanced. This can result in the following error:

java.lang.NoSuchMethodError: <span style="color: #ff0000;"><strong>pcGetManagedFieldCount</strong></span>
at com.bea.medrec.model.Prescription.
&lt;clinit&gt;(Prescription.java)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)

Workaround or Solution:

In WLS 10.3, the classloader hooks for runtime enhancement for all the EJBs are installed first. The webapp modules are then processed in order after their classloader initialization completes. This resolves the issue.

There is a narrow possibility that the NoSuchMethodError message could still occur if all of the following conditions are true:
(a) a custom class loader configuration is defined, (b) two webapp modules share the same classloader and persistence classes, (c) one of the modules has persistence.xml and the other module doesn’t, and (d) the webapp module that has the persistence.xml is not declared or occurs earlier in the order of the modules of the EAR. In this case, some entity classes may load without enhancements. To prevent this, the modules should be ordered in the opposite way.

In my case, I was not running multiple web modules, nor was there an issue with the persistence.xml declaration.

As I started looking into this error, I was searching through the server console logs….

Server Console Logs:

####<Apr 30, 2009 9:05:19 AM EDT> <Warning> <DeploymentService> <xxx.xxx.xxx.com>
<cmpAdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'>
<<WLS Kernel>> <> <> <1241096719211> <BEA-290064>
<Deployment service servlet encountered an Exception while handling the
deployment service message for request id "-1" from server "cmpServer01". Exception is:
<span style="color: #ff0000;"><strong>"java.lang.ClassNotFoundException: org.hibernate.HibernateException:
This error could indicate that a component was deployed on a  cluster member but not
other members of that cluster. Make sure that any component deployed on a
server that is part of a cluster is also deployed on all other members of that cluster</strong></span>
at weblogic.j2ee.ApplicationManager.loadClass(ApplicationManager.java:218)
at weblogic.j2ee.ApplicationManager.loadClass(ApplicationManager.java:85)
at weblogic.common.internal.WLObjectInputStream.resolveClass(WLObjectInputStream.java:61)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at weblogic.deploy.internal.targetserver.state.DeploymentState.readExternal
(DeploymentState.java:244)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.ArrayList.readObject(ArrayList.java:593)
at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at weblogic.common.internal.WLObjectInputStream.readArrayList(WLObjectInputStream.java:116)
at weblogic.deploy.service.internal.transport.DeploymentServiceMessage.readMessage
(DeploymentServiceMessage.java:322)
at weblogic.deploy.service.internal.transport.DeploymentServiceMessage.readExternal
(DeploymentServiceMessage.java:301)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.handleDeploymentServiceMessage
(DeploymentServiceServlet.java:618)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.internalDoPost
(DeploymentServiceServlet.java:235)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.access$000
(DeploymentServiceServlet.java:77)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet$1.run
(DeploymentServiceServlet.java:210)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.deploy.service.internal.transport.http.DeploymentServiceServlet.doPost
(DeploymentServiceServlet.java:207)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run
(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run
(WebAppServletContext.java:3498)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
".>

At this point, I felt I was running in circles. First I get a java.lang.NoSuchMethodError, at the same time I am getting a java.lang.ClassNotFoundException being thrown. All surrounding Hibernate, or my data access.

In my data access, I was using a custom controller to lookup a JNDI name, then if that was not available, use a basic data source.

<bean id="dataSourceFactory"
class="com.baselogic.configuration.DataSourceFactory"
init-method="init">
<property name="jndiDatasourceName" value="<strong>jdbc/DefaultDataSource</strong>"/>
<property name="basicDataSource" ref="<strong>basicDataSource</strong>"/>
</bean>

Then I realized, that it was WebLogic, not timing out the jndi lookup. My data source factory works on the assumption that a reasonable timeout will occurr if the requested JNDI data source name does not exist. This is not the case. I n my case, the above server console error was through around 5 minutes into the deployment, yet the server hangs for over an hour before finally returning a failed deployment. Once I create the JNDI data source with the proper name, the deployment worked smoothly.

Conclusion

In conclusion, the first thing to always address, is proper JNDI data source configuration before attempting a deployment. This can save hours of idle time waiting for deployment issue to arise.

The End…