Packer docker build exits code 137 when running runit cookbook

Question

I'm trying to use Packer to build a docker image of the webapp I'm working on. Whenever I run packer build, when it gets to the step that it runs the runit recipe, I would get Build 'docker' errored: Error executing Chef: Non-zero exit status: 137

I looked into 137, and found out this is the exit code commonly associated with a kill -9. In most cases this would imply that the system is running critically low on memory, and the system is attempting to compensate.

I tried to find the smallest possible reproduction, and I came up with this packer configuration:

{
    "builders":[{
        "type": "docker",
        "pull": false,
        "image": "silkstart/basic_server",
        "export_path": "image.tar",
        "run_command":[
            "-d",
            "-i",
            "-t",
            "--memory-reservation",
            "1G",
            "{{.Image}}",
            "/bin/bash"
        ]
    }],
    "provisioners":[
        {
            "type": "chef-solo",
            "cookbook_paths": ["cookbooks", "vendor/cookbooks"],
            "data_bags_path": "data_bags",
            "roles_path": "roles",
            "environments_path": "environments",
            "run_list": [
                "recipe[runit]"
            ]
        }
    ],
    "post-processors": [
        {
            "type": "docker-import",
            "repository": "silkstart/docker_test",
            "tag": "0.1"
        }
    ]
}

When I run packer build on this configuration, this is my output:

TMPDIR=/opt/shared packer build packer_files/docker_test.json
docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Starting docker container...
    docker: Run command: docker run -v /opt/shared/packer-docker484290992:/packer-files -d -i -t --memory-reservation 1G silkstart/basic_server /bin/bash
    docker: Container ID: 1f87b0cf1fe71f07b580ae6b18415a79c23a1a32a40f5f0366be90f160977a50
==> docker: Provisioning with chef-solo
    docker: Installing Chef...
    docker: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
    docker: Dload  Upload   Total   Spent    Left  Speed
    docker: 100 20022  100 20022    0     0  45092      0 --:--:-- --:--:-- --:--:-- 45196
    docker: Getting information for chef stable  for ubuntu...
    docker: downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=&p=ubuntu&pv=14.04&m=x86_64
    docker: to file /tmp/install.sh.23/metadata.txt
    docker: trying curl...
    docker: url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
    docker: md5 5cfc19d5a036b3f7860716bc9795a85e
    docker: sha256  e0b42748daf55b5dab815a8ace1de06385db98e29a27ca916cb44f375ef65453
    docker: version 12.6.0downloaded metadata file looks valid...
    docker: downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
    docker: to file /tmp/install.sh.23/chef_12.6.0-1_amd64.deb
    docker: trying curl...
    docker: Comparing checksum with sha256sum...
    docker:
    docker: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
    docker:
    docker: You are installing an omnibus package without a version pin.  If you are installing
    docker: on production servers via an automated process this is DANGEROUS and you will
    docker: be upgraded without warning on new releases, even to new major releases.
    docker: Letting the version float is only appropriate in desktop, test, development or
    docker: CI/CD environments.
    docker:
    docker: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
    docker:
    docker: Installing chef
    docker: installing with dpkg...
    docker: Selecting previously unselected package chef.
    docker: (Reading database ... 17195 files and directories currently installed.)
    docker: Preparing to unpack .../chef_12.6.0-1_amd64.deb ...
    docker: Unpacking chef (12.6.0-1) ...
    docker: Setting up chef (12.6.0-1) ...
    docker: Thank you for installing Chef!
    docker: Creating directory: /tmp/packer-chef-solo
    docker: Creating directory: /tmp/packer-chef-solo/cookbooks-0
    docker: Creating directory: /tmp/packer-chef-solo/cookbooks-1
    docker: Creating directory: /tmp/packer-chef-solo/roles
    docker: Creating directory: /tmp/packer-chef-solo/data_bags
    docker: Creating directory: /tmp/packer-chef-solo/environments
    docker: Creating configuration file 'solo.rb'
    docker: Creating JSON attribute file
    docker: Executing Chef: sudo chef-solo --no-color -c /tmp/packer-chef-solo/solo.rb -j /tmp/packer-chef-solo/node.json
    docker: [2016-01-29T06:42:48+00:00] INFO: Forking chef instance to converge...
    docker: [2016-01-29T06:42:48+00:00] INFO: *** Chef 12.6.0 ***
    docker: [2016-01-29T06:42:48+00:00] INFO: Chef-client pid: 207
    docker: [2016-01-29T06:42:50+00:00] INFO: Setting the run_list to ["recipe[runit]"] from CLI options
    docker: [2016-01-29T06:42:50+00:00] INFO: Run List is [recipe[runit]]
    docker: [2016-01-29T06:42:50+00:00] INFO: Run List expands to [runit]
    docker: [2016-01-29T06:42:50+00:00] INFO: Starting Chef Run for 1f87b0cf1fe7
    docker: [2016-01-29T06:42:50+00:00] INFO: Running start handlers
    docker: [2016-01-29T06:42:50+00:00] INFO: Start handlers complete.
    docker: [2016-01-29T06:42:52+00:00] INFO: Processing service[runit] action nothing (runit::default line 20)
    docker: [2016-01-29T06:42:52+00:00] INFO: Processing execute[start-runsvdir] action nothing (runit::default line 24)
    docker: [2016-01-29T06:42:52+00:00] INFO: Processing execute[runit-hup-init] action nothing (runit::default line 33)
    docker: [2016-01-29T06:42:52+00:00] INFO: Processing apt_package[runit] action install (runit::default line 64)
    docker: [2016-01-29T06:42:55+00:00] INFO: Processing cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] action create (dynamically defined)
    docker: [2016-01-29T06:42:55+00:00] INFO: cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] created file /var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed
    docker: [2016-01-29T06:42:55+00:00] INFO: cookbook_file[/var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed] updated file contents /var/chef/cache/preseed/runit/runit-2.1.1-6.2ubuntu3.seed
    docker: [2016-01-29T06:42:55+00:00] INFO: apt_package[runit] pre-seeding package installation instructions
==> docker: Killing the container: 1f87b0cf1fe71f07b580ae6b18415a79c23a1a32a40f5f0366be90f160977a50
Build 'docker' errored: Error executing Chef: Non-zero exit status: 137

I'm not entirely sure what is causing the code 137, and any help would be appreciated.

Update 1

I'm including a gist of the full debug output from Chef. It's much more verbose, mainly due it would seem to all of the attempts Ohai makes to get information.

https://gist.github.com/jrstarke/4c5f3b432aaee70c7f77

No references in here seem to suggest an out of memory error, at least on the docker host.

score 1 · Answer 1 · answered Feb 02 '16 at 05:54

After much much digging, I found the problem. The underlying problem, and the solution were both found on an issue in cloudfoundry-incubator/garden-linux.

Apparently as part of the setup process one of the post init scripts for runit executes a kill -s HUP 1. Why I'm not entirely sure, but as they noted there, doing a trap '' HUP right before my apt-get install runit and a trap HUP afterwards totally solved my problem.

score 0 · Answer 2 · answered Jan 29 '16 at 07:16

0

Check the OOM log on the host machine. Also you can use the execute_command configuration value to turn the log level to debug.

answered Jan 29 '16 at 07:16

coderanger

52,400
4
52
75

I've added a gist to my question with the log output and kern.log entries for the run. Nothing that looks like an OOM error though. – Jamie Starke Jan 29 '16 at 15:30

score 0 · Answer 3 · answered Aug 02 '17 at 12:14

0

This answer seemed to work for me: https://stackoverflow.com/a/42398166/2878244

I had to increase the memory resources assigned to docker by going to the Docker Tab > Preferences > Advanced

answered Aug 02 '17 at 12:14

wcyn

3,826
2
31
25

Packer docker build exits code 137 when running runit cookbook

3 Answers3