How to find which part of the source code is taking more time to execute in Java REST APIs built on SpringBoot?

Question

I have a REST Web Service built on Spring Boot and consuming some back-end REST APIs and executing database operations along with some Redis cache transactions. It is taking more than 7 or 8 seconds to get response when i hit the API through Postman tool. Initially it was 2 or 3 seconds. When i see the executing time by using Java Date API before and after every operations like database transaction, redis cache transaction, consuming other REST services etc. and they are like 20ms, 90ms, 500ms etc., respectively. I am unable to find where the API response is getting delayed while executing the Java Code. (Also some asynchronous code is present in btw.)

For the time being, i cannot put the execution time difference across the source code due to huge amount of code and multiple service implementations in a single API call.

Is there an open-source tool to find out the source code execution time to understand the performance or response time of a REST web service to further debug or improve the performance easily ?

Normally using tools like JMeter, we can see the performance results but how can we know which part of the code is actually taking a specific time like how SONAR code quality tool that detects which line of code may cause particular kind of bugs or issues or potential bugs or security threats.

Any help is highly appreciated.

Did you want to run the profiler in development only, or did you want to keep running it in production? — DV82XL, Jun 27 '20 at 11:52
hello, i would like to implement this in production, but want to successfully implement and do a POC in staging environments first. I am looking into your references and trying to understand them & implement it in spring boot application. sure will update you on this further and let you know if i face any issues. meanwhile if i can get any github example on this spf4j implementation in any spring boot application, that would be helpful. really thanks much for your answer! — Vamsi, Jun 27 '20 at 16:59
Are you still working on this? What are your requirements to consider this question answered? A POC of the profiling library shouldn't be too hard as examples are available online. A full POC in a Spring/REST application may take a while.. — DV82XL, Jun 30 '20 at 13:22
Hello! I have not found any POC related to Spring/REST and i couldn't get any basic config related stuff on spring boot for this library, still on research, may be i can try to get this figured out by this weekend. let me know if you have any other references or docs related to SPF4J — Vamsi, Jul 01 '20 at 15:18
I put together a basic POC using Spring Boot and spf4j, but it's pretty hard to set up due to lack of documentation. What version of Java are you running and are you using OpenJDK or Oracle JDK? — DV82XL, Jul 04 '20 at 23:47

DV82XL · Accepted Answer · 2020-07-06T13:33:07.170

Java Flight Recorder and Mission Control

Oracle's Java Flight Recorder is actually built into the Oracle JVM, but you need to use a JDK version that has JFR built in. Oracle also provides a set of tools called Java Mission Control that can be used to analyze and visualize the outputs of JFR. Of course, other open-source tools are also available to convert JFR outputs to flame graphs. JFR and JMC are free for development, but require commercial licenses to deploy into production. However, Oracle has kindly included JFR and JMC as of OpenJDK 11.

Spring AOP

Spring AOP can also be used to perform method profiling. See Profiling a Java Spring application and Spring App Method Profiling. This can help with simple debugging, but there is no built-in data visualization. Once you've identified problem methods (due to long or high-variance response times), you can annotate them with AOP and monitor them in a log file or using Spring Data ElasticSearch (advanced).

Simple Performance Framework for Java (SPF4J)

SPF4J is an open-source library. It's pretty easy to use but community support is lacking, so you will need to spend some time figuring out how it works. You can use annotations to flag and monitor specific methods or you can use AOP to profile the entire stack trace. The output is a series of ssdump files or Time-Series DataBase (TSDB) files, which you can visualize in a GUI tool that shows time graphs of min/max/average metrics and displays flame graphs to analyze the performance for all method calls in a stack trace. Here is a tutorial and sample code explaining how to profile a jaxrs REST API.

VisualVM

I was hesitant to add VisualVM to this list, since it is mostly used for CPU and memory profiling. It doesn't specialize in timing profiles. However, it is a popular option and, upon closer inspection, it does look like you can get some timing information as shown in the screenshot below. This tool would definitely be worth looking into as well.

IDE Profilers

There are also tools that come with IDEs, such as NetBeans Profiler or IntelliJ Ultimate Profiling Tools (commercial) that can also do the job. IntelliJ uses Java Flight Recorder (JFR) under the hood.