Azure cloud-init: Failed to install packages

Question

I have a simple cloud-init file:

#cloud-config
package_upgrade: true

packages:
  - symlinks
  - wget
  - rsync
  - git

but it is failing for my Azure (log provided at the end).
This is the first time that I'm using cloud-init and on Azure.
What could be wrong and how can I fix it?

PS. my VM:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:    11
Codename:   bullseye

$ uname -rm
5.10.0-6-cloud-amd64 x86_64

Failed log:

2021-05-28 14:44:26,704 - util.py[DEBUG]: Cloud-init v. 20.4.1 running 'modules:final' at Fri, 28 May 2021 14:44:26 +0000. Up 25.34 seconds.
2021-05-28 14:44:26,748 - stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.debian.Distro'>
2021-05-28 14:44:26,749 - stages.py[DEBUG]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) with frequency once-per-instance
2021-05-28 14:44:26,749 - handlers.py[DEBUG]: start: modules-final/config-package-update-upgrade-install: running config-package-update-upgrade-install with frequency once-per-instance
2021-05-28 14:44:26,756 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/1dd6dd52-1a5f-4a41-9810-da5e50583762/sem/config_package_update_upgrade_install - wb: [644] 24 bytes
2021-05-28 14:44:26,756 - helpers.py[DEBUG]: Running config-package-update-upgrade-install using lock (<FileLock using file '/var/lib/cloud/instances/1dd6dd52-1a5f-4a41-9810-da5e50583762/sem/config_package_update_upgrade_install'>)
2021-05-28 14:44:26,757 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/1dd6dd52-1a5f-4a41-9810-da5e50583762/sem/update_sources - wb: [644] 22 bytes
2021-05-28 14:44:26,757 - helpers.py[DEBUG]: Running update-sources using lock (<FileLock using file '/var/lib/cloud/instances/1dd6dd52-1a5f-4a41-9810-da5e50583762/sem/update_sources'>)
2021-05-28 14:44:26,757 - subp.py[DEBUG]: Running command ['apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] with allowed return codes [0] (shell=False, capture=False)
2021-05-28 14:44:27,179 - util.py[DEBUG]: apt-update [apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet update] took 0.422 seconds
2021-05-28 14:44:27,180 - util.py[WARNING]: Package update failed
2021-05-28 14:44:27,180 - util.py[DEBUG]: Package update failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 86, in handle
    cloud.distro.update_package_sources()
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 205, in update_package_sources
    self._runner.run("update-sources", self.package_command,
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 199, in package_command
    util.log_time(logfunc=LOG.debug,
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 293, in subp
    raise ProcessExecutionError(stdout=out, stderr=err,
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']
Exit code: 100
Reason: -
Stdout: -
Stderr: -
2021-05-28 14:44:27,386 - subp.py[DEBUG]: Running command ['apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'dist-upgrade'] with allowed return codes [0] (shell=False, capture=False)
2021-05-28 14:44:27,464 - util.py[DEBUG]: apt-upgrade [apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet dist-upgrade] took 0.078 seconds
2021-05-28 14:44:27,465 - helpers.py[DEBUG]: update-sources already ran (freq=once-per-instance)
2021-05-28 14:44:27,465 - subp.py[DEBUG]: Running command ['apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'symlinks', 'wget', 'rsync', 'git'] with allowed return codes [0] (shell=False, capture=False)
2021-05-28 14:44:27,526 - util.py[DEBUG]: apt-install [apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install symlinks wget rsync git] took 0.061 seconds
2021-05-28 14:44:27,526 - util.py[WARNING]: Failed to install packages: ['symlinks', 'wget', 'rsync', 'git']
2021-05-28 14:44:27,527 - util.py[DEBUG]: Failed to install packages: ['symlinks', 'wget', 'rsync', 'git']
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 100, in handle
    cloud.distro.install_packages(pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 112, in install_packages
    self.package_command('install', pkgs=pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 199, in package_command
    util.log_time(logfunc=LOG.debug,
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 293, in subp
    raise ProcessExecutionError(stdout=out, stderr=err,
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'symlinks', 'wget', 'rsync', 'git']
Exit code: 100
Reason: -
Stdout: -
Stderr: -
2021-05-28 14:44:27,527 - cc_package_update_upgrade_install.py[WARNING]: 2 failed with exceptions, re-raising the last one
2021-05-28 14:44:27,527 - handlers.py[DEBUG]: finish: modules-final/config-package-update-upgrade-install: FAIL: running config-package-update-upgrade-install with frequency once-per-instance
2021-05-28 14:44:27,527 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
2021-05-28 14:44:27,528 - util.py[DEBUG]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 848, in _run_modules
    ran, _r = cc.run(run_name, mod.handle, func_args,
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 124, in handle
    raise errors[-1]
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 100, in handle
    cloud.distro.install_packages(pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 112, in install_packages
    self.package_command('install', pkgs=pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 199, in package_command
    util.log_time(logfunc=LOG.debug,
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2348, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 293, in subp
    raise ProcessExecutionError(stdout=out, stderr=err,
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'symlinks', 'wget', 'rsync', 'git']
Exit code: 100
Reason: -
Stdout: -
Stderr: -

Maybe Microsoft Azure Linux Agent (waagent) is the root cause, it may has keeped an apt process in provisioning. After long struggle with waagent, I've switched to Ubuntu (which use cloud-init for provisioning by default). — silverfox, Aug 04 '21 at 14:50
That could be the answer, could you elaborate as an answer please? I can accept it @silverfox. — xpt, Aug 04 '21 at 19:00

Richard Cheney · Answer 1 · 2022-03-09T12:28:09.570

4

Newer versions of apt use the option string, so you can extend the timeout and add retries to the commands. For example:

sudo apt update -o Dpkg::Lock::Timeout=60 -o Acquire::Retries=60

You can include this in your cloud config files. (Example cloud-init file.)

apt:
  conf: |
    Acquire::Retries "60";
    DPkg::Lock::Timeout "60";

This creates a file, /etc/apt/apt.conf.d/94cloud-init-config. These options help to avoid immediate lock errors when waagent is competing.

Additional info:

man apt.conf
ls /usr/share/doc/apt/examples/
cloud-config-apt.txt

edited Mar 09 '22 at 12:28

answered Mar 09 '22 at 10:59

Richard Cheney

41
3

1

Tried again today, not working for me, still getting `E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)` if you believe your solution is working, please provide fully working example. thanks. – xpt Jun 19 '22 at 14:14
I've also been having issues getting this working. I think this works for something like `upgrade`, but not for `update`. – siwyd Jun 30 '23 at 14:32

score 1 · Answer 2 · answered Nov 19 '21 at 23:45

1

The waagent is holding the apt lock during VM provisioning preventing cloud-init from installing the packages.

E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/

Workaround: Use Azure Custom Script Extensions instead.

References:

answered Nov 19 '21 at 23:45

SleepyTonic

506
5
11

Sounds like Microsoft Azure Linux Agent (waagent) ***is*** the root cause, same as what silverfox pointed out before. thx. – xpt Nov 20 '21 at 15:08
Further discussion: https://github.com/MicrosoftDocs/azure-docs/issues/82500 — there's guidance in that thread about masking/disabling waagent-apt.service, which reportedly worked around the problem for some people, but it didn't address it for me. Also note that cloud-init only claims to be supported in a subset of Debian images, currently nothing later than Debian 10: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/using-cloud-init#debian – Nathaniel Irons Feb 14 '22 at 06:21
Thanks for the input @NathanielIrons. So sad that such basic problem hasn't been solved for over a year now. If you have found a native cloud-init workaround (instead of Ms' Azure Custom Script Extensions), please post an answer as I tried today with the above 2-upvote "solution" but still failed. – xpt Jun 19 '22 at 14:19

Azure cloud-init: Failed to install packages

2 Answers2

Linked