AWS Patch Manager Fails on Ubuntu 22.04

Question

UPDATE: As of Feb 1,2023 I paid the money and opened a ticket with AWS and this is the response.. Keep waiting:

Reaching out to the internal team in terms of Patch Manager for Ubuntu 22.04, we were notified that the addition of Ubuntu 22.04 for Patch Manager is currently a “work-in-progress” and is on its roadmap to be released in the near future. While no exact ETA is currently available, you can expect to see this release in the latter half of the first quarter of this year. I would suggest you to keep an eye on our SSM documentation [1] and What’s New page [2] for latest updates. I do apologize for the inconvenience this has caused you. Please bear with us for sometime as the SSM team works on getting this release out in the near future.

SSM documentation - https://docs.aws.amazon.com/systems-manager/latest/userguide/patch-manager-prerequisites.html What’s New - https://aws.amazon.com/new/

I've tested this on an upgraded Ubuntu 22.04 host and also on a CLEAN new Ubuntu 22.04 host from official AWS - Canonical image and attempts to patch either host using the AWS SSM RunPatchBaseLine fail.

Std out from command is reporting: root [ERROR]: Error loading entrance module.

Std Error is reporting: "Error loading patching payloadfailed to run commands: exit status 156

Output in the /var/log/amazon/ssm/amazon-ssm-agent.log for the failed commands looks like:

"/usr/bin/python3
/usr/bin/apt-get
Reading package lists...
Building dependency tree...
Reading state information...
python3-apt is already the newest version (2.3.0ubuntu2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Using python binary: 'python3'
Using Python Version: Python 3.10.4
/usr/bin/curl
/usr/bin/wget
06/10/2022 16:52:49 root [INFO]: Downloading payload from https://s3.dualstack.us-east-2.amazonaws.com/aws-ssm-us-east-2/patchbaselineoperations/linux/payloads/patch-baseline-operations-1.87.tar.gz
06/10/2022 16:52:49 root [INFO]: Attempting to import entrance file os_selector
06/10/2022 16:52:49 root [ERROR]: Error loading entrance module.
Traceback (most recent call last):
  File \"/var/log/amazon/ssm/patch-baseline-operations/common_startup_entrance.py\", line 164, in execute
    entrance_module = __import__(module_name)
  File \"/var/log/amazon/ssm/patch-baseline-operations/os_selector.py\", line 11, in \u003cmodule\u003e
    import common_os_selector_methods
  File\"/var/log/amazon/ssm/patch-baseline-operations/common_os_selector_methods.py\", line 11, in \u003cmodule\u003e
    from patch_common.baseline_override import load_baseline_override
  File \"/var/log/amazon/ssm/patch-baseline-operations/patch_common/baseline_override.py\", line 6, in \u003cmodule\u003e
    from patch_common.downloader import download_file, load_json_file, is_access_denied
  File \"/var/log/amazon/ssm/patch-baseline-operations/patch_common/downloader.py\", line 1, in \u003cmodule\u003e
    import boto3
  File \"/var/log/amazon/ssm/patch-baseline-operations/boto3/__init__.py\", line 16, in \u003cmodule\u003e
    from boto3.session import Session
  File \"/var/log/amazon/ssm/patch-baseline-operations/boto3/session.py\", line 17, in \u003cmodule\u003e
    import botocore.session
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/session.py\", line 29, in \u003cmodule\u003e
    import botocore.configloader
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/configloader.py\", line 19, in \u003cmodule\u003e
    from botocore.compat import six
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/compat.py\", line 25, in \u003cmodule\u003e
    from botocore.exceptions import MD5UnavailableError
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/exceptions.py\", line 15, in \u003cmodule\u003e
    from botocore.vendored import requests
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/__init__.py\", line 58, in \u003cmodule\u003e
    from . import utils
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/utils.py\", line 26, in \u003cmodule\u003e
    from .compat import parse_http_list as _parse_list_header
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/compat.py\", line 7, in \u003cmodule\u003e
    from .packages import chardet
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/__init__.py\", line 3, in \u003cmodule\u003e
    from . import urllib3
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/__init__.py\", line 10, in \u003cmodule\u003e
    from .connectionpool import (
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/connectionpool.py\", line 38, in \u003cmodule\u003e
    from .response import HTTPResponse
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/response.py\", line 9, in \u003cmodule\u003e
    from ._collections import HTTPHeaderDict
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/_collections.py\", line 1, in \u003cmodule\u003e
    from collections import Mapping, MutableMapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
06/10/2022 16:52:49 root [ERROR]: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
Traceback (most recent call last):
  File \"/var/log/amazon/ssm/patch-baseline-operations/common_startup_entrance.py\", line 164, in execute
    entrance_module = __import__(module_name)
  File \"/var/log/amazon/ssm/patch-baseline-operations/os_selector.py\", line 11, in \u003cmodule\u003e
    import common_os_selector_methods
  File \"/var/log/amazon/ssm/patch-baseline-operations/common_os_selector_methods.py\", line 11, in \u003cmodule\u003e
    from patch_common.baseline_override import load_baseline_override
  File \"/var/log/amazon/ssm/patch-baseline-operations/patch_common/baseline_override.py\", line 6, in \u003cmodule\u003e
    from patch_common.downloader import download_file, load_json_file, is_access_denied
  File \"/var/log/amazon/ssm/patch-baseline-operations/patch_common/downloader.py\", line 1, in \u003cmodule\u003e
    import boto3
  File \"/var/log/amazon/ssm/patch-baseline-operations/boto3/__init__.py\", line 16, in \u003cmodule\u003e
    from boto3.session import Session
  File \"/var/log/amazon/ssm/patch-baseline-operations/boto3/session.py\", line 17, in \u003cmodule\u003e
    import botocore.session
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/session.py\", line 29, in \u003cmodule\u003e
    import botocore.configloader
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/configloader.py\", line 19, in \u003cmodule\u003e
    from botocore.compat import six
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/compat.py\", line 25, in\u003cmodule\u003e
    from botocore.exceptions import MD5UnavailableError
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/exceptions.py\", line 15, in \u003cmodule\u003e
    from botocore.vendored import requests
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/__init__.py\", line 58, in \u003cmodule\u003e
    from . import utils
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/utils.py\", line 26, in \u003cmodule\u003e
    from .compat import parse_http_list as _parse_list_header
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/compat.py\", line 7, in \u003cmodule\u003e
    from .packages import chardet
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/__init__.py\",line 3, in \u003cmodule\u003e
    from . import urllib3
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/__init__.py\", line 10, in \u003cmodule\u003e
    from .connectionpool import (
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/connectionpool.py\", line 38, in \u003cmodule\u003e
    from .response import HTTPResponse
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/response.py\", line 9, in \u003cmodule\u003e
    from ._collections import HTTPHeaderDict
  File \"/var/log/amazon/ssm/patch-baseline-operations/botocore/vendored/requests/packages/urllib3/_collections.py\", line 1, in \u003cmodule\u003e
    from collections import Mapping, MutableMapping
ImportError: cannotimport name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
",
      "standardError": "Error loading patching payloadfailed to run commands: exit status 156"

This appears to be an issue with the newer Python that is included with Ubuntu 22.04. Have you upgraded SSM? It's not clear to me that Ubuntu 22.04 is supported yet. — stdunbar, Jun 10 '22 at 17:56
Sorry for delays.. I've updated all software and packages. checked snap packages. Etc. I went do far as to install Pyton2.7 and hacked the symlink for python3 and forced SSM to run on Python2, then I get the error: "An error occurred (UnsupportedOperatingSystem) when calling the GetDeployablePatchSnapshotForInstance operation" I wish I could find any documentation on SSM that says UB 22.04 is supported or not. — Scott Dunt, Jun 16 '22 at 22:18
`UnsupportedOperatingSystem` certainly seems to indicate that it's not supported. [This link](https://docs.aws.amazon.com/systems-manager/latest/userguide/prereqs-operating-systems.html) (scroll to Ubuntu) only goes to 20.10 as of this writing. — stdunbar, Jun 16 '22 at 23:02
I also updated the snap version of amazon-ssm-agent to the release canditate version Name Version Rev Tracking Publisher Notes amazon-ssm-agent 3.1.1511.0 5992 latest/candidate aws✓ classic And it still fails with the error 156.. — Scott Dunt, Jun 17 '22 at 15:34
i totally agree, that's in part why I started this thread. AW offers multiple UB 22.04 images to create new instances from. So, why is patching of 22.04 not supported.. OR more clearly documented? — Scott Dunt, Jun 24 '22 at 14:04

score 2 · Answer 1 · answered Sep 11 '22 at 19:07

2

This latest ubuntu 22.0 ain't supported yet.

answered Sep 11 '22 at 19:07

Gurlal

111
1
9

20.04 also isn't supported anymore. SSM support for Ubuntu is just hopelessly broken. – user1751825 Mar 27 '23 at 21:22

Scott Dunt · Accepted Answer · 2023-08-07T12:58:05.707

Problems with the current - new SSM Patch manager STILL NOT working on Ubuntu 22.04, may, this is NOT conclusive, but might be related to the new "services need restart" popup that got added to Ubuntu 22.04.

My test instance failed to patch last night, Patch manager, 'Detailed Status' was "terminated". And I cannot get to the 'View output' section of the SSM ui for that patch run.

When I ran apt update on the command line it worked.. BUT, I got a 'services need to be restarted" popup, which interactively I could respond to.. I am wondering if that is messing with the python scripts that are doing the patching..

Take a read through of: How to stop ubuntu pop-up "Daemons using outdated libraries" when using apt to install or update packages?

And:

https://askubuntu.com/questions/1367139/apt-get-upgrade-auto-restart-services

I have updated my 'test' instance and will monitor to see if that helps.

As of 4/4/2023 the AWS Link https://docs.aws.amazon.com/systems-manager/latest/userguide/prereqs-operating-systems.html shows that 22.04 is supported by SSM patch manager.. Now I need to test it to make sure thats' true..

score 0 · Answer 3 · answered Apr 18 '23 at 07:02

I did some digging into this and found the following:

Ubuntu 22.04 LTS is not supported in Patch Manager at time of writing. Please see: https://repost.aws/questions/QUzbb6-VpuTfuu-SK77Dg2CA/patch-baseline-operations-fail-with-python-3-10-6

It is supported in System Manager but that is not the same. Please see the following for a list of supported OS types and Python versions here: https://docs.aws.amazon.com/systems-manager/latest/userguide/patch-manager-prerequisites.html#python-version

Ubuntu 22.04 makes use of Python 3.10.x, but the highest Patch Manager currently supports for Ubuntu is up to Python 3.9.x

Yes, It seems AWS maintains TWO DIFFERENT lists of what is supported.. 22.04 is "supported" for SSM.. but its NOT supported for Patch manager???? Thanks for adding the link to the supposed versions for PATCH MANAGER to the discussion. IMO UB 22.04 has been out for OVER a YEAR. Which makes this look a whole lot like, "Run AWS's versions of Linux or else suffer the consequences..." — Scott Dunt, Apr 19 '23 at 12:24

score 0 · Answer 4 · answered Jul 20 '23 at 13:16

0

We're finding 22.04/Python 3.10 is not supported now although, when AWS Patch Manager initially added 22.04 to their list in April, May's updates ran ok on our 22.04 servers. Last 2 months however we get errors wrt to the /var/log/amazon/ssm/patch-baseline-operations/jmespath/visitor.py file

Curiously when we attempt updates the whole folder "patch-baseline-operations" gets blown away. The boxes where this folder remains, in terms of its datestamp, tally with the last good update date on those boxes ie the May date

answered Jul 20 '23 at 13:16

amartinez

169
2
14

Agreed, according to AWS Docs. 22.04 and Python 3.10 should be a supported combination: https://docs.aws.amazon.com/systems-manager/latest/userguide/patch-manager-prerequisites.html#supported-os – Scott Dunt Jul 21 '23 at 14:14
Thanks Scott - are you finding its not, similar to us? – amartinez Jul 24 '23 at 07:19

AWS Patch Manager Fails on Ubuntu 22.04

4 Answers4