2

Good day, all. I Met next problem: Work on project that need to deploy on different env with WAS and tomcat. WAS docer img used: common/base-ibm-websphere:8.0.0.10 - 8.0 Architecture: x86_64

 Name                  IBM WebSphere Application Server
Version               8.0.0.10
ID                    BASE
Build Level           cf101502.03
Build Date            1/16/15
Package               com.ibm.websphere.DEVELOPERSILAN.v80_8.0.10.20150116_1534
Architecture          x86-64 (64 bit)
Installed Features    IBM 64-bit SDK for Java, Version 6
                      EJBDeploy tool for pre-EJB 3.0 modules
                      Embeddable EJB container
                      Stand-alone thin clients and resource adapters

Tomcat docker img used: common/base-tomcat:7

java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) Server VM (build 24.80-b11, mixed mode)

From the start suppose something wrong going with WAS because it's much more slower with simplest request (x8 slower):

  long startMethodTime = System.nanoTime();
        long endMethodTime;
        long operationStartTime;
        long operationEndTime;
        Long result = 0L;
        Long resultSet = 0L;


        operationStartTime = System.nanoTime();
        List<ResultDTO> testDAOList = new ArrayList<ResultDTO>();
        for (int i = 0; i < 200000; i++) {
            ResultDTO testClass = new ResultDTO();
            testClass.setBody(String.valueOf(new Random().nextDouble()));
            testClass.setResult(String.valueOf(new Random().nextInt()));
            testClass.setSubject(String.valueOf(new Date()));

            testDAOList.add(testClass);
        }
        operationEndTime = System.nanoTime();
        String operationTime = String.valueOf(((operationEndTime -  operationStartTime) / 1000000));
        LOGGER.error("Creation object in cycle=" + operationTime);


        operationStartTime = System.nanoTime();
        for (ResultDTO testDAO : testDAOList) {
            result += testDAO.getResult().length();
        }
        operationEndTime = System.nanoTime();
        operationTime = String.valueOf((operationEndTime -  operationStartTime)/ 1000000);
        LOGGER.error("Incrementation in the loop of objects=" + operationTime);


        operationStartTime = System.nanoTime();
        Set<ResultDTO> sortedResults = new TreeSet<ResultDTO>(
                new Comparator<ResultDTO>() {
                    public int compare(ResultDTO r1, ResultDTO r2) {
                        int firstCompValue = (r1.getResult().length() + r1.getSubject().length()) % 2;
                        int secondCompValue = (r2.getResult().length() + r2.getSubject().length()) % 2;
                        return firstCompValue - secondCompValue;
                    }
                }
        );

        sortedResults.addAll(testDAOList);
        operationEndTime = System.nanoTime();
        operationTime = String.valueOf((operationEndTime -  operationStartTime) / 1000000);
        LOGGER.error("Sort  the Set of Objects=" + operationTime);

        for (ResultDTO testDAO : sortedResults) {
            resultSet += testDAO.getBody().length();
        }

        //Creation of 100 default objects
        long hundredObjectsCreationTime = 0;
        for (int i = 0; i < 100; i++) {
            operationStartTime = System.nanoTime();
            ResultDTO testClass = new ResultDTO();
            testClass.setBody(String.valueOf(new Random().nextDouble()));
            testClass.setResult(String.valueOf(new Random().nextInt()));
            testClass.setSubject(String.valueOf(new Date()));
            operationEndTime = System.nanoTime();

            hundredObjectsCreationTime += (operationEndTime -  operationStartTime);
        }

        LOGGER.error("100 objects creation time = " + hundredObjectsCreationTime);

        endMethodTime = System.nanoTime();
        operationTime = String.valueOf((endMethodTime - startMethodTime) / 1000000);
        LOGGER.error("Test method duration = " + operationTime);

It's executed on WAS in x8 slower than on Tomcat. Test results for comporation:

    Tomcat:
2018-03-26 21:39:44,347 ERROR [c.v.e.r.r.AboutResource] Creation object in cycle=1006
2018-03-26 21:39:44,381 ERROR [c.v.e.r.r.AboutResource] Incrementation in the loop of objects=33
2018-03-26 21:39:44,411 ERROR [c.v.e.r.r.AboutResource] Sort  the Set of Objects=29
2018-03-26 21:39:44,411 ERROR [c.v.e.r.r.AboutResource] 100 objects creation time = 405100 - microsec
2018-03-26 21:39:44,411 ERROR [c.v.e.r.r.AboutResource] Test method duration = 1070

WAS:

[26/03/18 22:26:39:944 UTC] 00000018 SystemOut     O ERROR [c.v.e.r.r.AboutResource] Creation object in cycle=8408
[26/03/18 22:26:40:069 UTC] 00000018 SystemOut     O ERROR [c.v.e.r.r.AboutResource] Incrementation in the loop of objects=124
[26/03/18 22:26:40:373 UTC] 00000018 SystemOut     O ERROR [c.v.e.r.r.AboutResource] Sort  the Set of Objects=303
[26/03/18 22:26:40:377 UTC] 00000018 SystemOut     O ERROR [c.v.e.r.r.AboutResource] 100 objects creation time = 3384400
[26/03/18 22:26:40:377 UTC] 00000018 SystemOut     O ERROR [c.v.e.r.r.AboutResource] Test method duration = 8841

On the WAS side configured only heap size (more than enoph) and db connection (that don't relate to this question).

ON what first of all I should pay attention when tuning IBM jVM / WAS configuration? (now i'm already investigating IBM manuals, possible u save my time for avoid smoke testing different not effective JVM properties).

Nolik
  • 131
  • 2
  • 4
  • 16
  • already try all from this page, jvm tuning advice's:https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/tprf_tunejvm_v61.html, nothing valuable impact. – Nolik Mar 27 '18 at 11:03
  • Add note: both evn launched in docker. – Nolik Mar 28 '18 at 12:48

2 Answers2

1

Some of the defaults between IBM Java and HotSpot are different so microbenchmarks (a few requests) are notoriously inaccurate. It's better to do a long load test. With that said, it could be many things. The first thing I suggest you try is to set -Xquickstart as a generic JVM argument:

The IBM® JIT compiler is tuned for long-running applications typically used on a server. You can use the -Xquickstart command-line option to improve the performance of short-running applications, especially for applications in which processing is not concentrated into a few methods. (https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.lnx.80.doc/diag/tools/jitpd_short_run.html)

If this doesn't help, then next thing is to enable verbosegc with -Xverbosegclog:${SERVER_LOG_ROOT}/verbosegc.%seq.log,20,50000 and check the proportion of time in garbage collection with the free IBM GCMV tool.

If this doesn't show much, then please provide more information about your operating system.

kgibm
  • 852
  • 10
  • 22
  • Provided test just show that in our app pure java execute much more slower (without db connections and so on). ~Same delay perf results we got for other API calls of our application where have java executions (~8 times slower). Try to add -Xquickstart option its didn't help. And additionally we provide big heap size for our app: -initialHeapSize 2560 that should be more than enough. But i still try to profile GC. – Nolik Mar 28 '18 at 12:33
  • 1
    Too large of a heap size can cause it's own problems with long GC times, so be careful, but looks like we'll need to dive deeper. Please run the Linux Performance MustGather during the issue: https://www-01.ibm.com/support/docview.wss?uid=swg21115785 – kgibm Mar 28 '18 at 14:57
1

Found problem. The enabled debug port slow down WAS java performance in 8 times!!! At lest on WAS 8.0.0.10 in docker.

Nolik
  • 131
  • 2
  • 4
  • 16
  • Ah yes, we've [seen this before](https://publib.boulder.ibm.com/httpserv/cookbook/Troubleshooting-Troubleshooting_Java-Troubleshooting_IBM_Java.html#Troubleshooting-Troubleshooting_IBM_Java-Debug_Mode). Unfortunately, there are so many possibilities that cause performance problems, so it's hard to know without data. In the future, when you have an issue, feel free to open a PMR and we'll help gather diagnostics and investigate and it should catch these sorts of problems pretty quickly. – kgibm Apr 03 '18 at 18:04