Docker on RHEL 6 Cgroup mounting failing

Question

I'm trying to get my head around something that's been working on a Centos+Vagrant, but not on our providers RHEL (Red Hat Enterprise Linux Server release 6.5 (Santiago)). A sudo service docker restart hands this:

Stopping docker:                                             [  OK  ]
Starting cgconfig service: Error: cannot mount cpuset to /cgroup/cpuset: Device or resource busy
/sbin/cgconfigparser; error loading /etc/cgconfig.conf: Cgroup mounting failed
Failed to parse /etc/cgconfig.conf                           [FAILED]

Starting docker:                                              [  OK  ]

The service starts okey enough, but images cannot run. A mounting failed error is shown when I try. And the startup-log also gives a warning or two. Regarding the kernelwarning, centos gives the same and has no problems as Epel should resolve this:

WARNING: You are running linux kernel version 2.6.32-431.17.1.el6.x86_64, which might be unstable running docker. Please upgrade your kernel to 3.8.0.
2014/08/07 08:58:29 docker daemon: 1.1.2 d84a070; execdriver: native; graphdriver:
[1233d0af] +job serveapi(unix:///var/run/docker.sock)
[1233d0af] +job initserver()
[1233d0af.initserver()] Creating server
2014/08/07 08:58:29 Listening for HTTP on unix (/var/run/docker.sock)
[1233d0af] +job init_networkdriver()
[1233d0af] -job init_networkdriver() = OK (0)
2014/08/07 08:58:29 WARNING: mountpoint not found

Anyone had any success overcoming this problem or should I throw in the towel and wait for the provider to update to RHEL 7?

Yes, sorry for not putting up the answer here @CamJackson. Have done so now :) — judoole, Oct 07 '14 at 07:35

score 7 · Accepted Answer · answered Jun 23 '15 at 04:33

7

I have the same issue.

enter image description here

(1) check cgconfig status

# /etc/init.d/cgconfig status

if it stopped, restart it

# /etc/init.d/cgconfig restart

check cgconfig is running

enter image description here

(2) check cgconfig is on

# chkconfig --list cgconfig

cgconfig 0:off 1:off 2:off 3:off 4:off 5:off 6:off

if cgconfig is off, turn it on

enter image description here

(3) if still does not work, may be some cgroups modules is missing. In the kernel .config file, make menuconfig, add those modules into kernel and recompile and reboot

after that, it should be OK

enter image description here

answered Jun 23 '15 at 04:33

tonysok

627
1
7
13

1

How do I do (3)? Can you expand on that? – codecowboy Sep 01 '15 at 13:03
Very helpful. Thank! – PRIHLOP Apr 26 '18 at 09:10
Changed this to accepted answer as it is more descriptive and has more upvotes. – judoole Sep 04 '18 at 07:50

score 1 · Answer 2 · answered Oct 07 '14 at 07:34

1

I ended up asking the same question at Google Groups and in the end finding a solution with some help. What worked for me was this:

umount cgroup
sudo service cgconfig start

The project of making Docker work was put on halt all the same. Later a problem of network connection for the containers. This took to much time to solve and had to give up.

answered Oct 07 '14 at 07:34

judoole

1,382
2
10
20

1

Yeah I found your thread there as well and tried this solution. Unfortunately it didn't work for me :( I get `umount: cgroup: not found`, followed by `cannot mount cpuset to /cgroup/cpuset: No such file or directory`, and then `error loading /etc/cgconfig.conf` (among other messages). Full logs can be seen [here](https://snap-ci.com/camjackson/writeitdown/branch/master/logs/defaultPipeline/33/dockerise) (right down the bottom). That build is actually on centos, not rhel. Not sure how much difference that makes. – Cam Jackson Oct 07 '14 at 12:37
3

I thought centos would be easier, but maybe not :) – judoole Oct 07 '14 at 12:51
Maybe the provider has some limitations? Not too familiar with Snap-ci myself. Is it for continous integration and running tests? – judoole Oct 07 '14 at 12:53
Yeah, it's a hosted CI service, and I'm beginning to suspect it might be something to do with the platform that's causing problems. I actually work for the company that makes it (albeit out of an office in a different country), so I shouldn't have any trouble getting support help. Thanks anyway :) – Cam Jackson Oct 07 '14 at 13:02
Looks nice 4 sure! Maybe spinning up a plain as vanilla CoreOS on a Vagrant and trying to the same there could give some answers also, or in any case see if it works at all. I used CoreOS with Vagrant before trying to deploy to Rhel6 and that worked fine actually. Or just plainly ask them if Docker works of course :) – judoole Oct 07 '14 at 13:10
@CamJackson , did you solved this ? I'm with the same problem. – ton Apr 15 '15 at 15:44
@ton You mean specifically on SnapCI? No I didn't. I was told they don't support docker yet, so for now I'm just using the built in Heroku deployment mechanism. If I really want to build docker images in the future, I'll probably set up a GoCD server in EC2. – Cam Jackson Apr 18 '15 at 06:30

score 1 · Answer 3 · answered May 23 '15 at 04:55

So I spent the whole day trying to rig docker to work on my vps. I was running into this same error. Basically what it came down to was the fact that OpenVZ didn't support docker containers up until a couple months ago. Specifically this RHEL update:

https://openvz.org/Download/kernel/rhel6/042stab105.14

Assuming this is your problem, or some variation of it, the burden of solving it is on your host. They will need to follow these steps:

https://openvz.org/Docker_inside_CT

score 1 · Answer 4 · answered Sep 03 '15 at 21:16

In my case

/etc/rc.d/rc.cgconfig start

was generating

Starting cgconfig service: Error: cannot mount cpu,cpuacct,memory to /cgroup/cpu_and_mem: Device or resource busy /usr/sbin/cgconfigparser; error loading /etc/cgconfig.conf: Cgroup mounting failed Failed to parse /etc/cgconfig.conf

i had to use:

/etc/rc.d/rc.cgconfig restart

and it automagicly umouted and mounted groups

Stopping cgconfig service: Starting cgconfig service:

score 1 · Answer 5 · answered Jul 27 '16 at 12:36

it seems like the cgconfig service not running,so check it!

# /etc/init.d/cgconfig status
# mkdir -p /cgroup/cpuacct /cgroup/memory /cgroup/devices /cgroup/freezer net_cls /cgroup/blkio
# cat /etc/cgconfig.conf |tail|grep "="|awk '{print "mount -t cgroup -o",$1,$1,$NF}'>cgroup_mount.sh
# sh ./cgroup_mount.sh
# /etc/init.d/cgconfig restart
# /etc/init.d/docker restart

score 0 · Answer 6 · answered Apr 07 '15 at 09:26

0

This situation occurs when the kernel is booted with cgroup_disable=memory and /etc/cgconfig.conf contains memory = /cgroup/memory;

This causes only /cgroup/cpuset to be mounted instead of the full set.

Solution: either remove cgroup_disable=memory from your kernel boot options or comment out memory = /cgroup/memory; from cgconfig.conf.

answered Apr 07 '15 at 09:26

Brian Lund Larsen

56
3

Unfortunately I'm not able to test this any longer, but I would be happy to accept this as an answer if peeps believe it's a better alternative. – judoole Apr 16 '15 at 07:53
Does this require a reboot? Commenting memory = /cgroup/memory doesn't work for me. – codecowboy Sep 01 '15 at 13:00

score 0 · Answer 7 · answered Apr 29 '15 at 17:43

The cgconfig service startup uses mount and umount which requires an extra privilege bump from docker.

See the --privileged=true flag here for more info.

I was able to overcome this issue by starting my container with:
docker run -it --privileged=true my-image.

Tested in Centos6, Centos6.5.

Docker on RHEL 6 Cgroup mounting failing

7 Answers7