Ways to create linux threads besides of pthread_create on linux

Question

I intercepted pthread_create call to capture the relations among all the threads. But I found that some threads' creation were not recorded with only intercepting pthread_create. I also tried to intercept posix_spawn posix_spawnp and clone call. But there are still some threads that I don't know who create them running in my experiment. So are there any other ways to create threads on linux?

More specifically, I used LD_PRELOAD to intercept pthread_create call, the code fragment is shown below:

int  pthread_create(pthread_t  *thread,  const pthread_attr_t  *attr,  void  *(*start_routine)(void  *),  void  *arg){
  static void *handle = NULL;
  static P_CREATE old_create=NULL;
  if( !handle )
  {
      handle = dlopen("libpthread.so.0", RTLD_LAZY);
      old_create = (P_CREATE)dlsym(handle, "pthread_create");
  }
  pthread_t tmp=pthread_self();
  //print pthread_t pid

  int result=old_create(thread,attr,start_routine,(void *)temp);
  //print thread pid

  return result;
}

In this way, I captured all the thread creation process. The same goes for clone. But actually clone was not called by the application. Sometimes, I got a parent-child threads pair which the parent thread is not printed before. So I don't know whether there are other ways to create this parent thread.

More more specifically, the upper application is a Mapreduce job on JVM1.7. I want to observe all the threads and processes and their relation

Thank you.

Are you perhaps thinking about internal kernel threads? Because otherwise *all* user-space thread (and process) creation goes through the `clone` call in Linux. — Some programmer dude, Sep 29 '17 at 06:01
I think it's not because of internal kernel threads. Because I used `pthread_self` to get the `pid` of a threads. So they should be user space threads. But somstimes `pthread_self` shows some threads whose creation in not captured by the `pthread_create` or `clone`. — Chalex, Sep 29 '17 at 06:08
@Chalex: Give a specific example of what you claim, by showing some [MCVE](https://stackoverflow.com/help/mcve) which creates a thread you don't understand. I did not observe this (e.g. even [here](https://stackoverflow.com/q/43141659/841108) threads are created by `pthread_create`). So **edit your question** to improve it and motivate it (i.e. *why* do you care about such bizarre threads?). Also define what is a thread for you, and how you observe it (for me a thread is by definition created by `pthread_create`; and the scheduler cares about *tasks*, including kernel threads, processes....) — Basile Starynkevitch, Sep 29 '17 at 06:12
@Someprogrammerdude: even today you can create processes with the genuine `fork` and `vfork` system calls. If the `libc` is using `clone(2)` syscall for them, some other code (e.g. assembler code, or binaries statically linked with old libc) can directly use `fork` & `vfork` without `clone` — Basile Starynkevitch, Sep 29 '17 at 06:19
@BasileStarynkevitch Okay I'll give you that one could always call the old-fashioned system calls directly, but what *sane* person would do that? :) — Some programmer dude, Sep 29 '17 at 06:26
Anyone running an old enough Linux executable binary would use `fork` system call; e.g. some ELF statically linked executable I compiled 15 years ago and kept intact — Basile Starynkevitch, Sep 29 '17 at 06:27
@Basile Starynkevitch, thank you for your suggestion. I supplemented my question. — Chalex, Sep 29 '17 at 06:27
@Chalex: even with the edit, we don't understand *why* you care, and most importantly what actual command and program is exhibiting the weird behavior (having strange threads) you observe, and what are these strange threads and how do you observe these threads (what commands you use to observe that, with what actual output). So **edit your question even more** — Basile Starynkevitch, Sep 29 '17 at 06:29
@Someprogrammerdude: BTW, [bones](http://www.call-with-current-continuation.org/bones/) is an interesting real world program (a Scheme compiler) which does not use the `libc`, and is likely to `fork` directly when it needs fork — Basile Starynkevitch, Sep 29 '17 at 06:33
@ Basile Starynkevit. I have already posted how I observe these weird threads before. Once `pthread_create` is called, it will print the pid of both the parent and the child thread. But it often prints some parent threads that I don't know who created them. — Chalex, Sep 29 '17 at 06:36
If you intercept `clone` (or in general syscalls) through `LD_PRELOAD` tricks you may be missing some, since you are only intercepting the libc wrappers. Somebody may be invoking them using straight `sysenter`/`int 0x80`, which cannot be intercepted using loader tricks - you'd have to go down the `ptrace` road. — Matteo Italia, Sep 29 '17 at 06:36
Maybe it's because of they use `sysenter/int 0x80` directly. In that way, I cannot capture all the thread creation process. But I am not sure whether JVM works in this way. Thank you. — Chalex, Sep 29 '17 at 06:40
use `ptrace` :) And yeah I was doubtful about the `fork()` using clone now, but indeed it is true: `strace python -c 'import os; os.fork()'` — Antti Haapala -- Слава Україні, Sep 29 '17 at 06:41

score 1 · Accepted Answer · answered Sep 29 '17 at 07:50

(moving from the comment)

LD_PRELOAD tricks just let you intercept C calls to external libraries - in this particular case to lptrhead (for pthread_create) and to libc (for fork/clone); but to create threads a program can bypass them completely and talk straight to the kernel, by invoking such syscalls (clone in particular) using int 80h (on x86) or sysenter (on amd64).

Straight syscalls cannot be intercepted that easily, you generally need the help of the kernel itself - which generally happens through the ptrace interface - which incidentally is how stuff like strace and debuggers are implemented. You should look in particular at the PTRACE_O_TRACECLONE, PTRACE_O_TRACEVFORK and PTRACE_O_TRACEFORK options to trace the creation of new processes/threads, or straight PTRACE_SYSCALL to block on all syscalls.

The setup of ptrace is a bit laborious and I don't have much time now, but there are several examples on the Internet of the basic pthread loop that you'll surely be able to find/adapt to your objective.

Ways to create linux threads besides of pthread_create on linux

1 Answers1