3

I'm helping develop an internal cloud infrastructure (OpenStack) to provide our departments with a fully on-premises alternative to AWS/Rackspace/SoftLayer/etc.

My boss just relayed a question to me from some internal customer who is asking us to support DTrace. I'm aware of the Oracle DTrace Project and the dtrace4linux and Linux' own native SystemTap (which I recall was initially built over Linux kprobes; but I gather than newer versions of SystemTap can tap into DTrace or the uprobes patches which were merged into the 3.5 kernels). I've also heard of Sysdig and LTTng and even fulltrace (what's the difference between the Linux kernels ftrace and uprobes support?)

My initial response was to ask whether this user has attempted to make effective use of ltrace and it's -S option to interleave system call tracing (à la strace).

But I also want to know more before the customer comes back.

My first and most important question is: which among these various options can be used on CentOS 6, 6.5, and 7.0 "out-of-the-box" (without replacing the stock kernels)? Same question for the Ubuntu 12.04 and 14.04 LTS releases?

Are any of these particularly suited or particularly bad for use from within cloud VM instances (OpenStack Nova, KVM)?

Is there a reasonably good and reasonably recent comparison of these options? What's my best bet for the time and effort investment involved with each of them?

What ever happened to Niels Provos' work on Systrace

fadden
  • 51,356
  • 5
  • 116
  • 166
Jim Dennis
  • 17,054
  • 13
  • 68
  • 116
  • 1
    Brendan Gregg has several overviews of linux perf/tracing on his website http://www.brendangregg.com/Slides/LinuxConEU2014_LinuxPerfTools.pdf Some tracers are listed - slide 37, 47, 48, 65-78: " Tracers: – perf_events, ftrace, eBPF, SystemTap, ktap, LTTng, dtrace4linux, sysdig" – osgx Apr 24 '15 at 03:44
  • I've seen Brendan's talk in person (at SCaLE). Those are excellent resources. I was just hoping for some emerging consensus and consolidation. – Jim Dennis Apr 24 '15 at 17:46

1 Answers1

2

IMHO SystemTap is a good choice for your customer. I worked a lot with DTrace and SystemTap and have to say that what achievable by DTrace is achievable by SystemTap

They have comparison with other tools https://sourceware.org/systemtap/wiki/SystemtapDtraceComparison (as expected, SystemTap is winner). It is also backward-compatible with DTrace USDT probes.

It works well with CentOS 6.x and 7.x stock kernels. Cannot say anything about Ubuntu, because I use Debian -- works fine, but Debian 7 stock kernel doesn't have uprobes. I've also used it under Xen, everything seem fine too.

Comparing with other tools you mentioned, however, SystemTap is more complex, because unlike Sysdig every performance data collection require scripting (it has examples, but not much, and doesn't have such thing as DTraceToolkit). Event LTTng project is developing user-friendly tools.

Also, one of famous DTrace developers, Brendan Gregg, currently working with Sysdig and SystemTap, that should mean something.

myaut
  • 11,174
  • 2
  • 30
  • 62