0

We're integrating towards a weblogic server using RMI. We're running the client in a webapp on a tomcat server. We're having weird issues with the response times. Some of the request will respond in ~20ms, but some will take 10-12 seconds. There are no responses in between ~20ms and 10-12 seconds.

A few things are worth mentioning

  1. We are using wlfullclient 10.3.2. I had to remove some classes in it to be able to make it run on Tomcat. This was a package with Servlet.class. Ugly, I know, but I wasn't able to take my webapp work with wlthinclient or other packages. Also, we are using a custom rmi client made by the middleware guys and it works well on weblogic server we have been running for ages.

  2. When a call takes 10-12 seconds, it seems as if there is no tcp traffic before until the very end of this time span. I have only monitored port 7461 here.

  3. I have the host names in my /etc/hosts file so it shouldn't be a DNS problem

  4. I am experiencing this only on our test/prod server, not when I run the webapp on Tomcat on my work station

  5. From debugging the issue it seems that the hang happens in weblogic.security.Security.runAs().

Edit: I dumped the thread, and it seems I have some kind of locking issue with the socket.

"http-bio-8082-exec-45" daemon prio=10 tid=0x0000000002a96000 nid=0x318c runnable [0x00007f09ca8e3000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    - locked <0x000000076586d0e0> (a java.net.SocksSocketImpl)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at weblogic.socket.SocketMuxer.newSocket(SocketMuxer.java:342)
    at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:79)
    at weblogic.socket.BaseAbstractMuxableSocket.createSocket(BaseAbstractMuxableSocket.java:133)
    at weblogic.rjvm.t3.MuxableSocketT3.newSocketWithRetry(MuxableSocketT3.java:206)
    at weblogic.rjvm.t3.MuxableSocketT3.connect(MuxableSocketT3.java:375)
    at weblogic.rjvm.t3.ConnectionFactoryT3.createConnection(ConnectionFactoryT3.java:33)
    at weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1773)
    at weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1416)
    - locked <0x000000076586c7f8> (a weblogic.rjvm.ConnectionManagerClient)
    - locked <0x000000076586c7f8> (a weblogic.rjvm.ConnectionManagerClient)
    at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:437)
    at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:396)
    at weblogic.rjvm.RJVMImpl.ensureConnectionEstablished(RJVMImpl.java:303)
    at weblogic.rjvm.RJVMImpl.getOutputStream(RJVMImpl.java:347)
    at weblogic.rjvm.RJVMImpl.getRequestStreamInternal(RJVMImpl.java:609)
    at weblogic.rjvm.RJVMImpl.getRequestStream(RJVMImpl.java:560)
    at weblogic.rjvm.RJVMImpl.getOutboundRequest(RJVMImpl.java:786)
    at weblogic.rmi.internal.BasicRemoteRef.getOutboundRequest(BasicRemoteRef.java:159)
    at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:341)
    at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)

Any ideas?

oyvind.s
  • 232
  • 1
  • 15
  • Have a look at the following post to see what I did with the wlfullclient: http://stackoverflow.com/questions/19137191/rmi-from-tomcat-to-weblogic – oyvind.s Nov 01 '13 at 09:36
  • Why RMI? It's kind of legacy and not firewall-friendly. Can't you create some REST/SOAP wrapper deployed on weblogic to integrate with that application? I bet full WLS client is rather heavy and it will be way larger than your integration app. – Andrey Adamovich Nov 04 '13 at 13:40
  • If you suspect slowness is security related, then it may be your WebLogic domain security that holds it down. It's hard to advice anything without knowing more details. Try to simplify security settings as much as possible for the domain and deployed application. – Andrey Adamovich Nov 04 '13 at 13:43
  • The WLS client is heavy, but using a SOAP wrapper is something we've tried and I don't want to go into a discussion about it that solution here. I dont suspect security issues at this point, after seeing that the thread hangs on socketConnect() I would think that something holds onto that socket. – oyvind.s Nov 04 '13 at 13:49
  • 1
    well, socketConnect is a method that just waits for connection to be established, it says you nothing, but JVM is awaiting for network I/O. I would try to collect more thread dumps or run a profiler or debugger on weblogic side. – Andrey Adamovich Nov 04 '13 at 14:02
  • Good input, thanks, but what if I say that the weblogic side has not experienced this before and have been running for years. What can cause my server to experience these issues? Any system variables, software versions etc? – oyvind.s Nov 04 '13 at 14:24
  • It's hard to say, I've seen many weirdness happening to WebLogic that nobody could explain, even Oracle. First, you have to understand where the problem is, I would try to collect more thread dumps on both tomcat and weblogic server side to get better understanding what's happening. Also, since you have WebLogic I guess you have access to my.oracle.com and you can open a support case with Oracle to get a hand on that issue. But trust me, the amount of information you've given in this question is not enough for anyone to give you any silver-bullet advice. You need to dig for more information. – Andrey Adamovich Nov 04 '13 at 14:43

1 Answers1

0

The issue here was that weblogic uses other nodes in the cluster that what we had defined in the setup. There were 4 nodes, we had opened for 3 of them in the firewall. The last was unknown for us, and when weblogic tried to access it, it timed out and then got response from another server.

oyvind.s
  • 232
  • 1
  • 15