1

I'd like to do a loading test with Jmeter. On google compute engine, I've creared 5 VMs, one running tomcat server, one running jmeter master and three running jmeter-server. My test plan is just a HTTP Request to the splash page(i.e. default home page, http://***.***.***.***/8080) of tomcat. There are 5000 users in each slave, and the plan will be executed in 5 minutes.

Messages on the terminal of the jmeter master are as following:

Writing log file to: /opt/apache-jmeter-3.1/bin/jmeter.log
Creating summariser <summary>
Created the tree successfully using aggregate_report.jmx
Configuring remote engine: 10.138.0.4:1099
Configuring remote engine:  10.138.0.5:1099
Configuring remote engine:  10.138.0.7:1099
Starting remote engines
Starting the test @ Wed Jan 11 08:38:17 UTC 2017 (1484123897799)
Remote engines have been started
Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445
summary + 106604 in 00:00:12 = 9252.2/s Avg:   131 Min:    66 Max:  1439 Err:     0 (0.00%) Active: 2638 Started: 2638 Finished: 0
summary + 359500 in 00:00:30 = 11982.5/s Avg:   441 Min:    66 Max: 17390 Err:     0 (0.00%) Active: 9658 Started: 9658 Finished: 0
summary = 466104 in 00:00:42 = 11224.9/s Avg:   370 Min:    66 Max: 17390 Err:     0 (0.00%)
summary + 326600 in 00:00:30 = 10888.1/s Avg:   735 Min:     0 Max: 43081 Err: 127205 (38.95%) Active: 12167 Started: 12167 Finished: 0
summary = 792704 in 00:01:12 = 11083.7/s Avg:   521 Min:     0 Max: 43081 Err: 127205 (16.05%)

Error messages on the terminal of the jmeter-server are as following:

2017/01/11 08:39:39 ERROR - jmeter.samplers.BatchSampleSender: sampleOccurred java.rmi.ConnectIOException: Exception creating connection to: 10.138.0.6; nested exception is: 
        java.net.SocketException: Too many open files
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:631)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
        at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
        at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
        at com.sun.proxy.$Proxy2.processBatch(Unknown Source)
        at org.apache.jmeter.samplers.BatchSampleSender.sampleOccurred(BatchSampleSender.java:184)
        at org.apache.jmeter.samplers.DataStrippingSampleSender.sampleOccurred(DataStrippingSampleSender.java:111)
        at org.apache.jmeter.samplers.RemoteListenerWrapper.sampleOccurred(RemoteListenerWrapper.java:104)
        at org.apache.jmeter.threads.ListenerNotifier.notifyListeners(ListenerNotifier.java:67)
        at org.apache.jmeter.threads.JMeterThread.notifyListeners(JMeterThread.java:852)
        at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:505)
        at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:418)
        at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:249)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:460)
        at java.net.Socket.<init>(Socket.java:431)
        at java.net.Socket.<init>(Socket.java:211)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 15 more

I think this exception happened in the jmeter-server side. Maybe some configurations of the jmeter-server aren't correct. This links said using ulimit -n or editing limits.conf could fix this problem.
I tried writing the following snippet to the limits.conf file, but it didn't work.

* soft nofile 10240 
* hard nofile 20480

Tomcat Server side:
There are error messages in catalina.out file on the side of tomcat server:

12-Jan-2017 01:41:20.132 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
 java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:444)
        at java.lang.Thread.run(Thread.java:745)

I also configured the limits.conf file, but it still didn't work.

Community
  • 1
  • 1
niaomingjian
  • 3,472
  • 8
  • 43
  • 78
  • can you check the target server logs for any errors. It seems to be more of target server issue which is not able to handle incoming connections and resulting in error or the connections are not getting closed. – rachna bafna Jan 11 '17 at 19:42
  • There were similar errors, too many open files. – niaomingjian Jan 12 '17 at 02:11
  • It means your server is not able to handle the incoming requests. Can you set ulimit to infinite and run the test? – rachna bafna Jan 12 '17 at 04:29
  • How should I use ulimit -n command? I tried, but it didn't work. – niaomingjian Jan 12 '17 at 05:13
  • I suggest you check the server health in terms of CPU utilization and memory usage. Run a test with less number of users say 2000 and see if you run into same issue and based on that result increase/reduce the load. – rachna bafna Jan 12 '17 at 18:39

0 Answers0