4

I have installed lttng on my Raspbian running on a Raspberry Pi 3 Model B+.

I get the following error when following a first demo from the website (as root):

# lttng create my-kernel-session --output=/tmp/my-kernel-trace
Spawning a session daemon
Session my-kernel-session created.
Traces will be written in /tmp/my-kernel-trace

A process is started correctly I assume:

# ps aux | grep lttng-sessiond
root       601  0.0  0.5 129320  5404 ?        Ssl  12:00   0:00 lttng-sessiond --daemonize
root       652  0.0  0.0   4372   572 pts/0    S+   12:30   0:00 grep lttng-sessiond

The following command returns an error:

# lttng list --kernel
Error: Unable to list kernel events: Kernel tracer not available
Error: Command error

Note that the following command does not list anything:

# lsmod | grep lttng
#

Are the lttng kernel modules not loaded? Is this the cause of the problem?

By the way, I am able to trace an application as explained in https://lttng.org/docs/v2.9/#doc-tracing-your-own-user-application.

I have installed lttng following the Debian installation:

# apt-get install lttng-modules-dkms
# apt-get install liblttng-ust-dev
# apt-get install lttng-tools 
# apt-get install liblttng-ust-agent-java
# apt-get install python3-lttngust

And the following is installed:

# dpkg -l | grep lttng
ii  liblttng-ctl0:armhf               2.9.3-1                      armhf        LTTng control and utility library
ii  liblttng-ust-agent-java           2.9.0-2+deb9u1               all          LTTng 2.0 Userspace Tracer (Java agent library)
ii  liblttng-ust-agent-java-jni:armhf 2.9.0-2+deb9u1               armhf        LTTng 2.0 Userspace Tracer (Java agent JNI interface)
ii  liblttng-ust-ctl2:armhf           2.9.0-2+deb9u1               armhf        LTTng 2.0 Userspace Tracer (trace control library)
ii  liblttng-ust-dev:armhf            2.9.0-2+deb9u1               armhf        LTTng 2.0 Userspace Tracer (development files)
ii  liblttng-ust-java                 2.9.0-2+deb9u1               all          LTTng 2.0 Userspace Tracer (Java support library)
ii  liblttng-ust-java-jni:armhf       2.9.0-2+deb9u1               armhf        LTTng 2.0 Userspace Tracer (JNI interface)
ii  liblttng-ust-python-agent0:armhf  2.9.0-2+deb9u1               armhf        LTTng 2.0 Userspace Tracer (Python agent native library)
ii  liblttng-ust0:armhf               2.9.0-2+deb9u1               armhf        LTTng 2.0 Userspace Tracer (tracing libraries)
ii  lttng-modules-dkms                2.9.0-1                      all          Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii  lttng-tools                       2.9.3-1                      armhf        LTTng control and utility programs
ii  python3-lttngust                  2.9.0-2+deb9u1               all          LTTng 2.0 Userspace Tracer (Python 3 UST agent)

Some more info on my Linux distribution:

# uname -a
Linux realtimepi 4.14.52-rt34-v7+ #5 SMP PREEMPT RT Wed Aug 1 14:22:59 CEST 2018 armv7l GNU/Linux

And:

# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Did I incorrectly install lttng, is there a mismatch with my Linux distribution, are the lttng kernel modules not correctly loaded, or is there a different problem?

Thanks,

Bart

Bart Meyers
  • 51
  • 6

2 Answers2

3

From the uname -a output I can see that you are running a PREEMPT RT Linux kernel as you probably know because your hostname is realtimepi.

When you are running a kernel that was not packaged by your distribution you can't simply install the lttng-modules kernel modules as the Linux kernel header files installed on your machine would not match the ones expected by the modules package.

To use the lttng-modules, you will need to compile them from source. Have a look at the documentation on that subject: https://lttng.org/docs/v2.10/#doc-building-from-source. In your case, i believe you only need to build the modules from source.

Make sure you uninstall the lttng-modules-dkms package and then restart the lttng-sessiond process before trying again.

fdeslaur
  • 144
  • 2
  • 8
  • Thank you for your response. I will look into this and let you know whether compiling from source did the trick. – Bart Meyers Sep 20 '18 at 07:38
  • I am not able to build from source. I have installed all four prerequisites, and have linked my /lib/modules/\`uname -r\`/build to the folder I built my kernel from. When executing "make" in the lttng-modules-2.10.7 folder, I get the following output: `find: ‘/home/pi/lttng/lttng-modules-2.10.7/extra_version/patches/’: No such file or directory` Do you have any idea what the cause is and how to solve this? – Bart Meyers Sep 21 '18 at 13:47
  • That was with the downloaded package http://lttng.org/files/lttng-modules/lttng-modules-2.10.7.tar.bz2. If I create the folder in question `extra_version/patches/` I get a different error: `/bin/sh: 1: ./scripts/recordmcount: Exec format error`. With the code cloned from git I get the same error. – Bart Meyers Sep 21 '18 at 14:28
  • I solved this problem and was able to successfuly compile lttng-modules, but I get the same error when using lttng. I had to go into the kernel folder and execute `make scripts` to recompile the scripts, including recordmcount as mentioned in the error. I assume this was due to the fact that I cross-compiled my linux kernel on a different machine. OK! Next: `sudo make modules_install && sudo depmod -a`. Now I figured I need to install the other lttng packages (liblttng-ust-dev and lttng-tools) via apt-get. Now I am back to square one (Kernel tracer not available). Do you have any ideas? – Bart Meyers Sep 21 '18 at 15:16
  • I do get a list when executing `lsmod | grep lttng` though, which was different before (see above). – Bart Meyers Sep 21 '18 at 15:17
  • And the command `dpkg -l | grep lttng` does not list `lttng-modules-dkms`. Is this expected? – Bart Meyers Sep 21 '18 at 15:26
  • I also tried building the other packages (`liblttng-ust-dev` and `lttng-tools`) but I still get the `Kernel tracer not available` error. – Bart Meyers Sep 21 '18 at 16:13
  • okay so you successfully installed the modules. The problem is probably that you are running 2.10.X modules and 2.9.Y lttng-tools. There is a version mismatch. Checkout the 2.9 version of the modules and install them. – fdeslaur Oct 01 '18 at 15:33
  • That was indeed the case. Now I installed (built from source) all 2.9 versions over the current 2.10 versions, but I keep getting the same error. Should I uninstall the tools first? I have the feeling this might be caused by my cross-compiled kernel. – Bart Meyers Oct 03 '18 at 13:17
  • I also tried to compile the preempt_rt kernel on the Raspberry Pi itself from a fresh installation. That case, the kernel sources that are needed to compile lttng are available. After building lttng from source, I keep getting the same error. When installing `lttng-modules` I do get the warning `depmod: WARNING: /lib/modules/4.14.66-rt40-v7/extra/lttng-tracer.ko needs unknown symbol __migrate_disabled`. Note that this time my kernel version is `4.14.66-rt40-v7` and I use the 2.10 version of lttng. – Bart Meyers Oct 05 '18 at 15:03
  • Loaded modules (`lsmod | grep lttng`) are: `lttng_statedump`, `lttng_ftrace`, `lttng_kprobes`, `lttng_clock`, `lttng_lib_ring_buffer`, `lttng_kretprobes`. Should `lttng-tracer` appear? The command `sudo insmod /lib/modules/4.14.66-rt40-v7/extra/lttng-tracer.ko` yields the following error: `insmod: ERROR: could not insert module /lib/modules/4.14.66-rt40-v7/extra/lttng-tracer.ko: Unknown symbol in module`. Is this causing the `Kernel tracer not available` error? – Bart Meyers Oct 05 '18 at 15:12
  • The __migrate_disabled issue you are seeing is caused by a change in the RT kernel. A fix is in the pipeline. In the meantime, you can use the latest commits of the 2.10 branch you should get this work-around: https://lists.lttng.org/pipermail/lttng-dev/2018-September/028225.html – fdeslaur Oct 10 '18 at 14:11
  • I changed the code, compiled lttng from source, and now I can list the kernel events! Thanks a lot! – Bart Meyers Oct 15 '18 at 14:29
1

There is a problem with the RT kernel. This causes the following warning when compiling lttng-modules: depmod: WARNING: /lib/modules/4.14.66-rt40-v7/extra/lttng-tracer.ko needs unknown symbol __migrate_disabled

Although this is reported as a warning, it causes the error.

If you adapt the source code of the lttng-modules 2.10 branch as explained in this bug report and compile lttng-modules again, the warning is absent and the kernel events can be listed.

Thank you fdeslaur for this answer!

Bart Meyers
  • 51
  • 6