I'm investigating frequent invokation of futex syscall in the application that is transferring data from files to a socket that perf record shows.
I tried to attach to jvm with gdb and set catch point on futex syscall and then print backtraces to understand where it comes from. The problem was that bt fails with some internal error:
gdb -p <jvm_pid>
(gdb) catch syscall futex
Catchpoint 1 (syscall 'futex' [202])
(gdb) continue
Continuing.
Thread 2 "java" received signal SIGSTOP, Stopped (signal).
[Switching to Thread 0x7fa60385c700 (LWP 27739)]
0x00007fa602a0e9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) backtrace
/build/gdb-GT4MLW/gdb-8.1/gdb/frame.c:534: internal-error: frame_id get_frame_id(frame_info*): Assertion `fi->level == 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
So eventually I got the following:
Python Exception <class 'KeyboardInterrupt'> Quit:
#0 0x00007fa602a0e9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa60236e9bb in os::PlatformEvent::park (this=0x7fa5fc018200) at ./src/hotspot/src/os/linux/vm/os_linux.cpp:5987
which looks incomplete. Is gdb complaining about dynamic code generation so this might not be possible to trace such things with it?
I tried to print backtrace immediately after attaching and it worked fine:
gdp -p <jvm pid>
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa602a09d2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
Installing openjdk unwinder
(gdb) bt
#0 0x00007fa602a09d2d in __pthread_timedjoin_ex () at /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa60343a91d in ContinueInNewThread0 (continuation=continuation@entry=0x7fa603434c50 <JavaMain>, stack_size=1048576, args=args@entry=0x7ffd45c7ec50) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1056
#2 0x00007fa603436b1a in ContinueInNewThread (ifn=ifn@entry=0x7ffd45c7ed70, threadStackSize=<optimized out>, argc=<optimized out>, argv=0x556124e8f600, mode=mode@entry=0, what=what@entry=0x0, ret=0)
at ./src/jdk/src/share/bin/java.c:2037
#3 0x00007fa60343a9eb in JVMInit (ifn=ifn@entry=0x7ffd45c7ed70, threadStackSize=<optimized out>, argc=<optimized out>, argv=<optimized out>, mode=0, mode@entry=1, what=0x0,
what@entry=0x7ffd45c83c8c "xxx.xxx.xxx.xxx.Main", ret=<optimized out>) at ./src/jdk/src/solaris/bin/java_md_solinux.c:1103
#4 0x00007fa603437267 in JLI_Launch (argc=<optimized out>, argv=<optimized out>, jargc=<optimized out>, jargv=<optimized out>, appclassc=1, appclassv=0x0, fullversion=0x55612461e848 "1.8.0_191-8u191-b12-0ubuntu0.18.04.1-b12", dotversion=0x55612461e841 "1.8", pname=0x55612461e83c "java", lname=0x55612461e834 "openjdk", javaargs=0 '\000', cpwildcard=1 '\001', javaw=0 '\000', ergo=0)
at ./src/jdk/src/share/bin/java.c:304
#5 0x000055612461e691 in main (argc=<optimized out>, argv=<optimized out>) at ./src/jdk/src/share/bin/main.c:125
So is the problem with dynamic code generation? Maybe there is a workaround?
