1

I have next code:

par.go

package main

import (
    "runtime";
    "time"
)

func main() {
    runtime.GOMAXPROCS(4)
    ch := make(chan int)
    n := 1
    for i := 0; i < n; i++ {
        go func() {
            time.Sleep(60 * time.Second)
            ch <- 1
        }();
    }
    for i := 0; i < n; i++ {
        <-ch
    }
}

I use next to run it:

$ go build par.go
$ time ./par

Then, confirm how many threads in this process:

$ ps -ef | grep par
shubunt+  3670 32131  0 12:35 pts/0    00:00:00 ./par
$ cat /proc/3670/status | grep -i threads
Threads:        5

You can see there are 5 threads.

If I change the value of n in code, then situations are next:

n := 100, Threads is 8
n := 10000, Threads is 9
n := 100000, Threads is 9
n := 1000000, Threads is 9
n := 2000000, Threads is 10

I know, go scheduler follow MPG model, here P = 4, so M = 4, M is 1:1 with KSE(Kernel threads). If any goroutine in any blocking status, the P will detached from current M, and find a idle M or new a M if can't find.

So, my question is: is time.Sleep really blocking goroutine? If not, why new threads there when I increase value of n from 1 to 2000000? If yes, there is 60 seconds there, why just scheduler new a little new M, I expect a lots of new threads there?

UPDATE:

Add another example from this.

test.go:

package main

import (
    "io/ioutil"
    "os"
    "runtime"
    "strconv"
)

func main() {
    runtime.GOMAXPROCS(2)
    data := make([]byte, 128*1024*1024)
    for i := 0; i < 200; i++ {
        go func(n int) {
            for {
                err := ioutil.WriteFile("testxxx"+strconv.Itoa(n), []byte(data), os.ModePerm)
                if err != nil {
                    println(err)
                    break
                }
            }
        }(i)
    }
    select {}
}

If not use Sleep, use real IO, the threads number will be 202 on my machine.

So, my question also related to the difference of above 2 examples, when I should worry about scheduler generate too many kernel threads for me?

atline
  • 28,355
  • 16
  • 77
  • 113

3 Answers3

4

[I]s time.Sleep really blocking goroutine?

Yes.

But how goroutines are actually scheduled to threads is a) complicated, b) different in every release, c) can be different from architecture to architecture and d) is not specified by the language. While the "MPG model" is an accurate model if how the scheduler works it is just a model.

If the scheduler determines that 10 threads are enough to not run 200'000 goroutines as they all are time.Sleeping then 10 threads are enough.

Basically there is nothing to worry or think about such stuff in Go (in stark contrast to other languages where extreme care has to be devoted to such peculiarities). "Blocking" just means that the next statement cannot be executed right away as the actual statement is not finished jet. This may happen for a plethora if reasons ranging from time.Sleep which does nothing except wait, waiting form RAM, waiting for disk or waiting for network data. Handling all in the same manner would simplify the scheduler but make it a bad one. So no, time.Sleep does not block the goroutine. The problem is "block goroutine" is not something with a defined meaning. And it need not be defined as there is nothing interesting to know about it.

Update:

[W]hen I should worry about scheduler generate too many kernel threads for me?

Never.

There a two different scenarios: A) writing normal, sensible production code and B) writing handcrafted code which is carefully designed to create lots of threads which are all waiting in disk IO to finish. Of course you can deliberately trick the scheduler and the OS and come up with a pathological program which creates too much threads (this problem that the scheduler can be tricked is addresses in issue #4056) but that is not something to worry about. Just do not deliberately do stupid things.

There are lots of ways to trick your computer. Writing racy code is one way. The race detector helps identifying them. Worrying about race conditions before writing them is a good thing (as this happens). Creating too many threads can happen and you can ask your OS for the thread count. And in the unlikely event that there are too many: Fix it. But this is unlikely. Its a bit like OOM errors: It is dead simple to write code which OOMs but there is nothing to worry constantly about OOM while writing code. If your experience OOM you redesign but you do not start any trivial project by worrying about OOM and what you need to know about OOM, how to prevent it and what to do about it. Unless you know already that your data uses lots of memory. Same here: If you know that your code will do massive concurrent disk-IO and this is intrinsic to the domain then you might worry about this and handle this in code but the techniques are the same in every language.

Volker
  • 40,468
  • 7
  • 81
  • 87
  • I'm not quite understand it. In fact, there is a topic here: https://stackoverflow.com/questions/28186361/why-does-it-not-create-many-threads-when-many-goroutines-are-blocked-in-writing, in it, it not use `sleep`, just use `io`. I tried it, it really generate 200+ threads, which I think maybe memory waste. So don't know what you said `there is nothing interesting to know about it` mean? I just want to know, in which scenario I should pay attention to avoid goroutine block, then don't let go scheduler generate too many threads for me. – atline Jun 23 '20 at 05:43
  • In fact, I also not understand, why we should talk `block` in goroutine. I mean if use epoll, all IO is non-blocked? Not same concept? – atline Jun 23 '20 at 05:45
  • And if we no need to worry about it, why https://github.com/golang/go/issues/4056 there? – atline Jun 23 '20 at 05:48
  • 2
    @atline You are conflating lots of things. The other question has the same problem than yours: Oversimplification of something which just is not specified. The issue you linked is something the people who _develop_ the scheduler are worrying about and they do have a clear understanding of what is going on on the OS and hardware level but this is nothing a developer in Go has to worry about. You are not going to learn anything from these experiments and you are not going to write better programs (a bit exaggerated for clearness). – Volker Jun 23 '20 at 07:10
  • I understood your points now, you main idea is: this is the behavior of golang, it's the thing of google needed to take care. For application developer we no need to think about of that, there is no way for a application developer to write a better one, so we no need to worry about that? My understanding for your points correct? – atline Jun 23 '20 at 07:15
  • 1
    "if use epoll, all IO is non-blocked?" Well, no. epoll is a mechanism on Linux which lets you watch file descriptors and there are other (older) such mechanisms like select. epoll is not some magic spell where you shout "epollius!" and all IO magically becomes "non-blocked" for whatever that may mean. The IO still is a blocking operation and all you can do with epoll is to be informed in a very effective way about which of several IO ops are ready to move foreward so you could try them again. – Volker Jun 23 '20 at 07:16
  • `The IO still is a blocking operation`, this really break all my understanding about `IO model`. What I always thought is `with epoll, or even poll/select`, the IO is non-block, but it's not async, but still sync io. Linux not design `aio` well, so `async io` moved to most of framework like `tornado` etc which in framework level wrap the done signal of `data move from kernel space to user space`, but it's really non-block, because if descriptor not ready, process won't block there. Give me some time to digest the new idea you bring to me, thanks. – atline Jun 23 '20 at 07:24
  • Yes, that's what I'm trying to tell you. These things are _not_ _specified_. If you rely on your application creating exactly 7 OS threads your application is not a valid Go program! On a different OS and compiled with a different compiler or even the same compiler but a different version of the compiler your program might create 1 or 1000 threads). Of course it is sensible to know about fundamental OS limitations like that file descriptors and threads, but this is all abstracted away in Go. – Volker Jun 23 '20 at 07:27
  • I upvote this, but I need more time to digest all things before I come back, thanks patience on this. – atline Jun 23 '20 at 07:31
  • If you do IO on a keyboard and the user doesn't type there simply is no IO and reading 5 keystrokes cannot happen until the user types and so this _is_ blocking. With epoll/select you just do not block a thread until the user typed. You say "inform me once these keystrokes are there or when the network packet arrived or when xyz" and do some other work. Lots of languages make it very hard do do several thing concurrently without falling into the rabbit hole of a "blocking syscall" which ruins application performance. Go is _designed_ to be different. – Volker Jun 23 '20 at 07:32
  • JavaScript is famous for its complicated framework if dealing with this type of concurrency. Go's philosophy is radically different and more like "The language takes care that concurrent code is properly scheduled on the hardware. How, when and why is absolutely no business of the developer, he cannot know and he doesn't need to know." There are a _few_ knobs like GOMAXPROCS which somehow control this scheduling: If you run 200 Go programs on a machine with 128 cores you might want to limit each program to use maybe just 12 cores simultaneously. – Volker Jun 23 '20 at 07:38
  • [illustrated-tales-of-go-runtime-scheduler](https://news.knowledia.com/US/en/articles/illustrated-tales-of-go-runtime-scheduler-6e42a862a8a774956997e3804b53f69821a64430) & [The Go netpoller](https://morsmachine.dk/netpoller) make me understand what you mean `"block goroutine" is not something with a defined meaning.` which seems make me understand a little about my concern, thanks although I am not agree with your points that developers no need to understand the detail of the scheduler. Anyway, by pass the arguments. – atline Jun 24 '20 at 14:41
3

Summarize my final understanding about this question in case anyone interested:

  1. For time.Sleep, if I set to launch 20 goroutines, next is the results of delve:

     ./dlv attach 2070
     Type 'help' for list of commands.
     (dlv) threads
     * Thread 2070 at 0x44d003 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
       Thread 2071 at 0x44d003 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
       Thread 2072 at 0x44d003 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
       Thread 2092 at 0x44d003 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
       Thread 2093 at 0x44d003 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
     (dlv) grs
       Goroutine 1 - User: /home/shubuntu1/gotrial/new/par.go:19 main.main (0x452686)
       Goroutine 2 - User: /usr/lib/go-1.12/src/runtime/proc.go:302    runtime.gopark (0x425f3f)
       Goroutine 3 - User: /usr/lib/go-1.12/src/runtime/proc.go:302 runtime.gopark (0x425f3f)
       Goroutine 4 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 5 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 6 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 7 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 8 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 9 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 10 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 11 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 12 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 13 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 14 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 15 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 16 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 17 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 18 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 19 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 20 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 21 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 22 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
       Goroutine 23 - User: /usr/lib/go-1.12/src/runtime/proc.go:307 time.Sleep (0x43cc69)
     * Goroutine 24 - User: /usr/lib/go-1.12/src/runtime/lock_futex.go:228 runtime.notetsleepg (0x409064)
     [24 goroutines]
    

    You can see, there are many goroutines in time.Sleep while just a little threads, these threads are in process of runtime.futex.

    From Understanding Golang sleep() function and also this, we could know golang sleep will involve a lock, so this possible a reason that some additional thread launched, but the lock quickly released, so not so many threads there, many goroutines still could reuse the same thread.

  2. For ioutil.WriteFile, if I set to launch 20 goroutines, next is the results of delve:

     ./dlv attach 2455
     Type 'help' for list of commands.
     (dlv) threads
     * Thread 2455 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2456 at 0x4530f3 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
       Thread 2457 at 0x460f0a /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6
       Thread 2458 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2459 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2460 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2461 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2462 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2463 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2464 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2465 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2466 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2468 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2469 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2470 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2471 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2472 at 0x460f0a /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6
       Thread 2473 at 0x460f0a /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6
       Thread 2474 at 0x460e90 /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
       Thread 2475 at 0x460f0a /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6
       Thread 2476 at 0x460f0a /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6
       Thread 2477 at 0x4530f3 /usr/lib/go-1.12/src/runtime/sys_linux_amd64.s:536 runtime.futex
     (dlv) grs
       Goroutine 1 - User: /home/shubuntu1/gotrial/newnew/test.go:24 main.main (0x4754b0)
       Goroutine 2 - User: /usr/lib/go-1.12/src/runtime/proc.go:302 runtime.gopark (0x429d9f)
       Goroutine 3 - User: /usr/lib/go-1.12/src/runtime/proc.go:302 runtime.gopark (0x429d9f)
       Goroutine 4 - User: /usr/lib/go-1.12/src/runtime/proc.go:302 runtime.gopark (0x429d9f)
       Goroutine 5 - User: /usr/lib/go-1.12/src/runtime/proc.go:302 runtime.gopark (0x429d9f)
       Goroutine 6 - User: /usr/lib/go-1.12/src/runtime/proc.go:302 runtime.gopark (0x429d9f)
       Goroutine 7 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6 (0x460f0a) (thread 2457)
       Goroutine 8 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6 (0x460f0a) (thread 2476)
       Goroutine 9 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2461)
       Goroutine 10 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6 (0x460f0a) (thread 2473)
       Goroutine 11 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2468)
       Goroutine 12 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2469)
     * Goroutine 13 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2455)
       Goroutine 14 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2459)
       Goroutine 15 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2460)
       Goroutine 16 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2471)
       Goroutine 17 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2464)
       Goroutine 18 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2462)
       Goroutine 19 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2466)
       Goroutine 20 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2474)
       Goroutine 21 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2470)
       Goroutine 22 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2463)
       Goroutine 23 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6 (0x460f0a) (thread 2472)
       Goroutine 24 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2458)
       Goroutine 25 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:53 syscall.Syscall6 (0x460f0a) (thread 2475)
       Goroutine 26 - User: /usr/lib/go-1.12/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x460e90) (thread 2465)
     [26 goroutines]
     (dlv)
    

    You can see, there are many goroutines in the process of syscall, while also there are many threads in syscall, not like sleep in runtime.futex which just invove lock. The ioutil involve real block syscall. This is because epoll not works for regular files on linux, so not like network io which have a network poller handle blocking interface with a non-block low level sys call, see this.

    So, I guess if we can't bear too many threads for regular file operation, then in application level, above design not reasonable. Maybe we need a series of dedicated file operation goroutines, and all file related requirements should be delegated to these limited number of dedicated goroutines with channel.

atline
  • 28,355
  • 16
  • 77
  • 113
1

I've read this discussion a few times and I think I understand the crux of you confusion. I was also a bit confused but I believe I understand this now.

  • Will time.Sleep block a go-routine?

Yes, but all that means is a few Kb of memory.

  • Will time.Sleep block a thread?

No, not any more. Very early versions of Go on some architectures may have had this problem but it is not an issue anymore. time.Sleep is handled by the Go runtime not with any system call.

  • Why does 2 million simultaneous sleeps only create a few threads?

I'm not sure exactly but 2 or 3 threads is insignificant next to 2 million go-routines.

  • Will I/O block a thread?

Yes but each blocked go-routine may not equate to one blocked thread. Eg on Linux Go uses epoll (on Windows IOCompletionPorts) to handle multiple network connections on the same thread.

Andrew W. Phillips
  • 3,254
  • 1
  • 21
  • 24