2

I'm trying to use the kernel's cpuset to isolate my process. To obtain this, I follow the instructions(2.1 Basic Usage) from kernel doc cpusets, however, it didn't work in my environment.

I have tried in both my centos7 server and my ubuntu16.04 work pc, but neither did work.

  1. centos kernel version:
[root@node ~]# uname -r
3.10.0-327.el7.x86_64
  1. ubuntu kernel version:
4.15.0-46-generic

What I have tried is as follows.

root@Latitude:/sys/fs/cgroup/cpuset# pwd
/sys/fs/cgroup/cpuset
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.cpus
0-3
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.mems
0
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.cpu_exclusive
1
root@Latitude:/sys/fs/cgroup/cpuset# cat cpuset.mem_exclusive
1
root@Latitude:/sys/fs/cgroup/cpuset# find . -name cpuset.cpu_excl
usive | xargs cat
0
0
0
0
0
1
root@Latitude:/sys/fs/cgroup/cpuset# mkdir my_cpuset
root@Latitude:/sys/fs/cgroup/cpuset# echo 1 > my_cpuset/cpuset.cpus
root@Latitude:/sys/fs/cgroup/cpuset# echo 0 > my_cpuset/cpuset.mems
root@Latitude:/sys/fs/cgroup/cpuset# echo 1 > my_cpuset/cpuset.cpu_exclusive
bash: echo: write error: Invalid argument
root@Latitude:/sys/fs/cgroup/cpuset#

It just printed the error bash: echo: write error: Invalid argument. Google it, however, I can't get the correct answers.

As I pasted above, before my operation, I confirmed that the cpuset root path have enabled the cpu_exclusive function and all the cpus are not been excluded by other sub-cpuset.

By using ps -o pid,psr,comm -p $PID, I can confirm that the cpus can be assigned to some process if I don't care cpu_exclusive. But I have also proved that if cpu_exclusive is not set, the same cpus can also be assigned to another processes.

I don't know if it is because some pre-setting are missed.

What I expected is "using cpuset to obtain exclusive use of cpus". Can anyboy give any clues?

Thanks very much.

xjas
  • 75
  • 2
  • 11
  • Did you achieve your goal(using cpuset to obtain exclusive use of cpus)?I did the same test in Ubuntu16.04, i found that there still some user processes running on the cores which i want to isolate.I have set these user processes to another cpuset already.Any hint?Thank you. – sunshilong369 May 25 '20 at 07:56

1 Answers1

4

i believe it is a mis-understanding of cpu_exclusive flag, as i did. Here is the doc https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt, quoting:

If a cpuset is cpu or mem exclusive, no other cpuset, other than
a direct ancestor or descendant, may share any of the same CPUs or
Memory Nodes.

so one possible reason you have bash: echo: write error: Invalid argument, is that you have some other cgroup cpuset enabled, and it conflicts with your operations of echo 1 > my_cpuset/cpuset.cpu_exclusive

please run find . -name cpuset.cpus | xargs cat to list all your cgroup's target cpus.

assume you have 12 cpus, if you want to set cpu_exclusive of my_cpuset, you need to carefully modify all the other cgroups to use cpus, eg. 0-7, then set cpus of my_cpuset to be 8-11. After all these cpus configurations , you can set cpu_exclusive to be 1.

But still, other process can still use cpu 8-11. Only the tasks that belongs to the other cgroups will not use cpu 8-11

for me, i had some docker container running, which prevents me from setting my cpuset cpu_exclusive

with kernel doc, i do not think it is possible to use cpus exclusively by cgroup itself. One approach (i know this approach is running on production) is that we isolate cpus, and manage the cpu affinity/cpuset by ourselves

xgwang
  • 614
  • 7
  • 21
  • Thanks for your reply. Please forgive my stupidness, maybe my situation is not same as you have encounterd. From what I have pased above, I just use the script `find . -name cpuset.cpu_exclusive | xargs cat` to confirm is there any process have consume cpu cores exclusively. It says no process did that. So I wonder is there any situations that I have not taken into account? – xjas May 15 '19 at 11:56
  • update the answer a bit, please run `find . -name cpuset.cpus | xargs cat` to list all your cgroups target cpu. if they have overlap, you cannot set cpu_exclusive. – xgwang May 16 '19 at 03:38
  • Yes, great, it did work as you have advised. Thanks for solving my confusion. But still I think it might be reasonable that the cgpoup modify the other cpusets for me after I have make my_cpusets exclusive. The reason why I think like that is I read a article: [kubernetes cpu-management](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/), which said the kubelet can make the specified pod exculsively consume some cores. Thanks again. – xjas May 16 '19 at 05:39
  • imho, i do no think it is reasonable to ask cgroup to do that. if it changes the configurations on parent/sibling scope, then with continuous changes due to different applications/users requirements, the cgroup subset could be a mess. so we need a solution like k8s to do this, and cgroup just focus on its own feature, in a limit scope. – xgwang May 17 '19 at 06:31
  • The explanation is quite reasonable, and I must dig into the k8s management becauset it seems I did not understand it fully. Thanks for your consideration again. – xjas May 21 '19 at 09:54