8

For Linux user space processes it seems pretty easy to determine which processes are multithreading. You can use ps -eLf and look at the NLWP value for the number of threads, which also corresponds to the 'Threads:' value in /proc/$pid/status. Apparently back in the day of LinuxThreads the implementation was not POSIX compliant. But This stackoverflow answer says "POSIX.1 requires threads share a same process ID" which is apparently rectified in NPTL. So with NPTL it allows nifty displays of threads with commands like ps -eLf because the threads all share the same PID, and you can verify that under /proc/$pid/task/ and see all the thread subfolders belonging to that "parent" process.

I can't find similar thread groupings under a "parent" process for kernel processes spawned by kthreadd, and I suspect an implementation difference since a comment under this answer says "You can not use POSIX Threads in kernel-space" and the nifty thread grouping is a POSIX feature. Thus with ps -eLf I never see multiple threads listed for kernel processes created by kthreadd which have the square brackets around it, like [ksoftirqd/0] or [nfsd], unlike user-space processes created by init.

From the man page for pthreads (which is used in the user space): 

       A single process can contain multiple threads, all of which are
       executing the same program.  These threads share the same global
       memory (data and heap segments), but each thread has its own stack
       (automatic variables).

This however is precisely what I do not see for kernel "threads", in terms of one process containing multiple threads.

In short, I never see any of the processes listed by 'ps' that are children of kthreadd having a NLWP (Threads) value higher than one, which makes me wonder if any kernel process forks/parallelizes and multithreads like user-space programs do (with pthreads). Where do the implementations differ?

Practical example: Output from ps auxf for the NFS processes.

root         2  0.0  0.0      0     0 ?        S    Jan12   0:00 [kthreadd]
root      1546  0.0  0.0      0     0 ?        S    Jan12   0:00  \_ [lockd]
root      1547  0.0  0.0      0     0 ?        S    Jan12   0:00  \_ [nfsd4]
root      1548  0.0  0.0      0     0 ?        S    Jan12   0:00  \_ [nfsd4_callbacks]
root      1549  0.0  0.0      0     0 ?        S    Jan12   2:40  \_ [nfsd]
root      1550  0.0  0.0      0     0 ?        S    Jan12   2:39  \_ [nfsd]
root      1551  0.0  0.0      0     0 ?        S    Jan12   2:40  \_ [nfsd]
root      1552  0.0  0.0      0     0 ?        S    Jan12   2:47  \_ [nfsd]
root      1553  0.0  0.0      0     0 ?        S    Jan12   2:34  \_ [nfsd]
root      1554  0.0  0.0      0     0 ?        S    Jan12   2:39  \_ [nfsd]
root      1555  0.0  0.0      0     0 ?        S    Jan12   2:57  \_ [nfsd]
root      1556  0.0  0.0      0     0 ?        S    Jan12   2:41  \_ [nfsd]

By default when you start the rpc.nfsd service (at least with the init.d service script) it spawns 8 processes (or at least they have PIDs). If I wanted to write a multi-threaded version of NFS, which is implemented as a kernel module, with those nfsd "processes" as a frontend, why couldn't I group the default 8 different nfsd processes under one PID and have 8 threads running under it, versus (as is shown - and as is different than user space processes) eight different PIDs?

NSLCD is an example of a user space program that uses multithreading by contrast:

UID        PID  PPID   LWP  C NLWP STIME TTY          TIME CMD
nslcd     1424     1  1424  0    6 Jan12 ?        00:00:00 /usr/sbin/nslcd
nslcd     1424     1  1425  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd
nslcd     1424     1  1426  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd
nslcd     1424     1  1427  0    6 Jan12 ?        00:00:27 /usr/sbin/nslcd
nslcd     1424     1  1428  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd
nslcd     1424     1  1429  0    6 Jan12 ?        00:00:28 /usr/sbin/nslcd

The PID is the same but the LWP is unique per thread.

Update on the function of kthreadd

From this stackoverflow answer:

kthreadd is a daemon thread that runs in kernel space. The reason is that kernel needs to some times create threads but creating thread in kernel is very tricky. Hence kthreadd is a thread that kernel uses to spawn newer threads if required from there . This thread can access userspace address space also but should not do so . Its managed by kernel...

And this one:

kthreadd() is main function (and main loop) of daemon kthreadd which is a kernel thread daemon, the parent of all other kernel threads.

So in the code quoted, there is a creation of request to kthreadd daemon. To fulfill this request kthreadd will read it and start a kernel thread.

Community
  • 1
  • 1
SeligkeitIstInGott
  • 223
  • 2
  • 7
  • 3
    Ｍulti-threads share same resource like address space. Kernel threads only live in kernel space, and __all kernel threads share the kernel space naturally__. So, there is no need for kernel multi-threads at all. – Chris Tsui Jan 23 '16 at 06:25
  • Understood, but threads do more than share memory. The whole point of a thread is to parallelize so one program doesn't have to block or wait until another task returns an so it splits off the task into a new thread. How can you keep track of related threads working on the same task in the kernel? – SeligkeitIstInGott Jan 23 '16 at 17:53
  • I was using NLWP as not just an indicator that separate processes/tasks were sharing the same memory space but that in fact that they are related processes "executing the same program" as the pthreads man page says. This becomes important for performance tuning and metrics if you want to see whether you get a gain in performance by splitting a task into more threads. Does the kernel not optimize performance by parallelizing computational tasks in the same manner? Or are "threads" the wrong category to talk about parallelization (pthreads' man page seems to associate it though)? – SeligkeitIstInGott Jan 23 '16 at 18:00
  • pthread uses the `clone` system call to start threads. So user space threads are actually processes that share address space, fd list, signals, and so on. In kernel space, you can use `clone` with whatever flags to create processes that would appear grouped as threads or whatever else. – BitWhistler Jan 23 '16 at 18:06
  • I had read that fork() and pthread_create() both use clone() but with slightly different bitmask settings passed to it - so that much made sense. When you say "would appear grouped" for kernel space tasks can you give an example? Is that something you could display in Linux by examining something in particular that would indicate that grouping (under/proc or elsewhere)? Or would someone have to write a C program to gather the relevant data that indicates their grouping? Additional reading recommendations are welcome too. Thanks. – SeligkeitIstInGott Jan 23 '16 at 18:20
  • @SeligkeitIstInGott What do you mean by "threads do more than share memory" exactly? They also share file descriptors, but kernel threads don't have file descriptors. When you ask "how can you keep track", it's not clear what you're asking. What is the "you" that you think needs to keep track? – David Schwartz Jan 24 '16 at 00:09
  • 1
    @SeligkeitIstInGott, in Linux, there is one task (also called process or thread, doesn't matter at all) for each user thread (no matter it is a thread in multi-threads process or in a single-thread process). kernel thread is also task. From the view of process scheduling, kernel thread and user thread have no difference, as they are both tasks which is the basic scheduling unit. So, when you say parallel running, kernel threads are identical to user-space multi-threads. – Chris Tsui Jan 24 '16 at 01:24
  • @SeligkeitIstInGott, I cannot give you an example from the top of my head. When you call `clone` you give it flags to set the gid, pgid and some more things. I imagine ps, top, and friends use them to some extent. I believe the thread grouping comes from the /proc//task/ structure. – BitWhistler Jan 24 '16 at 10:01
  • @David Schwartz Roland in his answer (second paragraph) seemed to nail exactly what I was wondering about, despite my lack of knowledge and perhaps poor way of conveying what I really wanted to ask. – SeligkeitIstInGott Jan 24 '16 at 18:12
  • As for me saying "keep track", I was wondering if the OS had any way of telling you (via values in /proc or a tool that parsed it for you) what threads were working on the same task in the kernel (as Roland mentioned), which for user space threads does have such an indicator via the shared PID method of presenting the tasks under /proc/$pid/task and the number of threads value shown in /proc/$pid/status for "Threads". – SeligkeitIstInGott Jan 24 '16 at 18:15
  • I was just wondering if there was a similar way to see which kernel threads were working on the same task in a parallelized manner (using mutexes, locking, etc.) in a way that a user could determine with standard userland commands. – SeligkeitIstInGott Jan 24 '16 at 18:15
  • @ChrisTsui Thanks for clearing up my confusion. So every user space thread has a corresponding task in the kernel (to process its requests)? Could you take a crack at answering my question under Roland's answer about the difference between children of the kthreadd process and the init process. I think therein lies the source of my misunderstanding. – SeligkeitIstInGott Jan 24 '16 at 18:27

1 Answers1

12

There is no concept of a process in the kernel, so your question doesn't really make sense. The Linux kernel can and does create threads that run completely in kernel context, but all of these threads run in the same address space. There's no grouping of similar threads by PID, although related threads usually have related names.

If multiple kernel threads are working on the same task or otherwise sharing data, then they need to coordinate access to that data via locking or other concurrent algorithms. Of course the pthreads API isn't available in the kernel, but one can use kernel mutexes, wait queues, etc to get the same capabilities as pthread mutexes, condition variables, etc.

Calling these contexts of execution "kernel threads" is a reasonably good name, since they are closely analogous to multiple threads in a userspace process. They all share the (kernel's) address space, but have their own execution context (stack, program counter, etc) and are each scheduled independently and run in parallel. On the other hand, the kernel is what actually implements all the nice POSIX API abstractions (with help from the C library in userspace), so internal to that implementation we don't have the full abstraction.

Roland
  • 6,227
  • 23
  • 29
  • Very helpful answer. One of my difficulties was with the the terminology, and saying kernel "task" rather than "process" seems to be the correct way to designate it. And thank you for understanding what I was attempting to ask since I had some difficulty knowing how to even frame the question, and your second paragraph seemed to nail that about mutexes, wait queues, and locking. Is there anything in print or online that basically says what you said in an introductory manner, because I had difficulty finding anything straightforward on this topic. – SeligkeitIstInGott Jan 24 '16 at 18:04
  • I also have a question now on what kthreadd actually is and does. Whatever runs as a child of kthreadd: is it showing you actual kernel threads or is it showing you user space threads that have some kind of 1:1 mapping to kernel tasks (I think I read something about that)? Perhaps due to their ignorance as well some articles I've read treat children of kthreadd as actual kernel tasks, despite the fact they have a PID. And what is the main difference between children of the init process and children of kthreadd? – SeligkeitIstInGott Jan 24 '16 at 18:19
  • "despite the fact they have a PID" = meaning that they are processes, which is a user space only concept, thus can't be a representation of what is running in the kernel space. – SeligkeitIstInGott Jan 24 '16 at 18:38
  • Hmm... This page on "The Kernel Thread (kthread)" in "Solaris Internals: Core Kernel Components" (pg.283) seems to address some of this, but I still can't wrap my head around it: http://tinyurl.com/jl7elu5. Maybe I just need to read more about UNIX/Linux kernel internals. – SeligkeitIstInGott Jan 24 '16 at 18:48
  • 1
    They don't really have a PID. On Linux, the kernel has just one name space and process IDs, thread IDs, and IDs assigned to kernel tasks come from this same namespace. Various tools distinguish the different types of IDs that come from this same namespace in different ways. Just because something calls it a PID, you can't assume that it's in fact a user-space process. They may just mean a number assigned from the same namespace as PIDs and that appears in the slot a PID would appear if this kernel-scheduled entity were a user-space process. – David Schwartz Jan 24 '16 at 20:18
  • So what is it that kthreadd does and what is the primary difference between what you see spawned by it (the processes - if I can call them that [I am thoroughly confused on that point now] - listed by ps that have a PPID of 2, which is kthreadd, and square brackets around them) and processes spawned by init? Basically delineated by the difference in output between pstree -alp 1 and pstree -alp 2 which shows the children of init and kthreadd respectively. Additionally do the "processes" shown by pstree -alp 2 belong to user space or kernel space? Thanks. – SeligkeitIstInGott Jan 25 '16 at 01:18
  • BTW, if there is a reading resource that will spare me pestering you with all these questions I'm happy to be referred to something. – SeligkeitIstInGott Jan 25 '16 at 01:19
  • This appears to be relevant somewhat to my questions above: http://unix.stackexchange.com/questions/78583/can-ps-display-only-non-kernel-processes-on-linux . It is interesting how multiple answers there (and the subject title) use the phrase "kernel process(es)" when talking about ps output, so apparently I'm not the only one confused. :-) – SeligkeitIstInGott Jan 25 '16 at 01:38
  • @DavidSchwartz "Just because something calls it a PID, you can't assume that it's in fact a user-space process. They may just mean a number assigned from the same namespace as PIDs and that appears in the slot a PID would appear if this kernel-scheduled entity were a user-space process." That's an interesting observation, but that also makes the way that is implemented in Linux confusing. I created a new post about that here: http://stackoverflow.com/questions/35004259/linux-assigns-pids-to-kernel-threads-and-not-only-processes. Thanks for your responses. They're helping me sort this out more. – SeligkeitIstInGott Jan 26 '16 at 00:38
  • 1
    `init` is the first task which has user-space, and all of its children are forked (and then exec) from it. `kthreadd` is just a kernel thread to simplify creating other kernel thread, as there are other way to create kernel thread like `kernel_thread()`. All of these are done through `do_fork()` with different parameters. – Chris Tsui Jan 26 '16 at 12:21
  • @ChrisTsui Thanks for clarifying that. What's interesting though is that any kernel threads are visible to user space **at all**. They could simply be invisible/not displayed. Are there perhaps other kernel threads that run in kernel space which aren't visible to 'ps' and aren't children of kthreadd, or does kthreadd literally handle **all** kernel threads? Of course, kernel tasks are not strictly equivalent with kernel threads, so I do expect that there are some kernel tasks that are not visible in user space. – SeligkeitIstInGott Jan 26 '16 at 16:41