17

I'm using isolcpus to assign a subset of the total CPU cores only for kernel tasks. This means that I can use the reminder for other tasks. I noticed when I run the tests/benchmarking tool from go, all the kernel processors are used (using htop). However, when I use taskset -c 3-10 go test -bench ., only one CPU core shows that is running the command. I was expecting the workload to be distributed to cores 3-10. Is it possible to achieve this?

Update: Thanks to @JimB for suggestion: I'm running the following golang code on the cpus cores reserved for kernel cores using taskset and works fine. But when scheduling outside those cores only one cpu core is used instead of the entire range specified in taskset.

package main

import (
    "fmt"
    "golang.org/x/sys/unix"
    "testing"
    "time"
)


func TestSchedSetaffinity(t *testing.T) {

    var newMask unix.CPUSet
    newMask.Set(0)

    err := unix.SchedSetaffinity(0, &newMask)
    if err != nil {
            fmt.Printf("SchedSetaffinity: %v", err)
    }

    for i := 0; i < 50; i++ {
            fmt.Println("Hello world")
            time.Sleep(2 * time.Second)
    }
}
Andrei
  • 7,509
  • 7
  • 32
  • 63
  • 3
    IIRC affinity masks should be preserved across forks, so I would assume the child process has the same settings as the `go test` process, but perhaps the runtime's thread management is somehow bypassing this. You could also try directly calling [`SchedSetaffinity`](https://pkg.go.dev/golang.org/x/sys/unix#SchedSetaffinity) with a pid of 0. – JimB Jan 25 '22 at 15:22
  • @JimB Brilliant! This seems to work. Thank you! – Andrei Jan 25 '22 at 20:13
  • I'm going back on this, but this seemed to work on a machine without `isolcpus`. When I tried to run on a machine with `isolcpus` it seems that only one CPU core is being exercised. Same behavior with `cpusets`. I'm wondering what is being started when not using taskset vs using taskset. Had a look at the number of threads started by the processor and it seems that it is changing every time I query for this info. Sometimes is 10, other times are 8. By looking at `htop` is seems that all the kernel cpus are exercises without taskset. – Andrei Jan 27 '22 at 17:46
  • I'm not 100% sure what the question is here, but if you have a Go program with several goroutines, setting CPU affinity for a goroutine requires LockOSThread, else the runtime can (and will) put some other goroutine on that OS thread. It would also be useful to know what runtime.GOMAXPROCS(0) returns. That is the number of OS threads the runtime will try to use. – David Chase Jan 28 '22 at 20:15
  • It's a different command, but I use numactl on a BIG.little arm64 box for benchmarking on the big cores, and that yields the expected results -- the Go process uses exactly the specified cores, all of them, and no others. – David Chase Jan 28 '22 at 20:19
  • @DavidChase I have the program above and I run it with `go test example.go`. I run `htop` in a separate terminal and run the filter mode. Although it is a single process I can see threads created and allocated to multiple cpu cores. When I run taskset with a range, only the first element from the range is used for all those threads – Andrei Jan 31 '22 at 18:04
  • @DavidChase also tried to use LockOSThread with the same effect. It seems that I can get what I like with taskset when I specify the cpu range to cpu cores outside `isolcpus` – Andrei Jan 31 '22 at 18:06
  • Further digging into this: my understanding is that golang will start various components in separate threads (like garbage collector). When I do htop I can see the threads started for my golang process. One runs the workload and the rest are golang's components. When no taskset is used these seem to be distributed across multiple cores. When taskset is used with range, only the first cpu core is used. – Andrei Feb 01 '22 at 17:48
  • `When taskset is used with range, only the first cpu core is used.` do the golang components also be a part of that same CPU core? – Neenad May 25 '22 at 10:14
  • @NeenadIngole yes all get scheduled to the same core – Andrei May 26 '22 at 18:12

1 Answers1

0

isolcpus does not work well with taskset and go. To take advantage of all allocated cpu cores you need to use chrt.

For example:

taskset -c 3-10 chrt 1 ./my_go_workload_binary

The assumption is that the cpu cores 3-10 in the isolcpus set.

Andrei
  • 7,509
  • 7
  • 32
  • 63