Can I force AWS CLI (v1) to use UTF encoding?

Question

Edit: Note this is an older question, from a time when AWS CLI was v1 - as noted in comments, there are other likely better solutions with v2

I'm using AWS CLI on Windows to query items from DynamoDb. Some of these items include non-ASCII characters.

When the query hits those items, it dies with an error

'charmap' codec can't encode character u'u010d' in position....

After hours of searching, I finally stumbled across a hackish workaround; under the AWSCLI\encodings directory, I copied utf_8.pyc over cp1252.pyc. This allows me to continue, but of course is ugly.

Before resorting to that, I also tried setting environment variables such as LANG, LC_ALL, LC_CTYPE to various permutations of en-US.UTF-8 or similar, all with no effect that I could see.

Does anyone know how (or is it even possible) to tell AWS CLI to use a particular encoding?

For anyone ending up here while experiencing this issue with AWS CLI v2 on Windows in Powershell, see the following solution https://stackoverflow.com/a/57134096/664054. This question is about CLI v1, where the recommendation now is to use CLI v2 instead — WhiteKnight, Jan 29 '23 at 15:06

score 5 · Accepted Answer · answered Dec 27 '16 at 11:56

5

Since you're using the command line interface, a change to the terminal's encoding scheme should fix the issue.

Type:

chcp 65001

in the console (for UTF-8; you may also try different encodings) and retry your operations.

answered Dec 27 '16 at 11:56

Igor Sowinski

179
1
8

6

For this to work, in the AWSCLI\encodings directory I had to copy utf_8.pyc to cp65001.pyc (which didn't exist), but that's less dirty than what I had done previously. Thanks! :) – Rick Riensche Dec 28 '16 at 14:20
Bless you @RookieRick, I was trying to get this to work for a couple hours until I finally came across your brilliant comment. – Dmitry K. Jun 04 '17 at 20:35
1

This option did not work for me calling `aws dynamodb scan --table-name foo --region us-east-2` on Windows 10 from a Powershell 6 session. However, @Marek's solution (see below) did. – Adam Jul 01 '19 at 02:21

score 5 · Answer 2 · edited Jun 06 '19 at 20:14

Maybe it could help as well - the issue with translation from AWS and storing results to the file (or powershell variable): With error:

aws translate translate-text --text "Investigation" --source-language-code "auto" --target-language-code "PL" >> a.txt

'charmap' codec can't encode character '\u015a' in position 1: character maps to

Adding env. variable fixes the problem

set PYTHONIOENCODING=UTF-8

aws translate translate-text --text "Investigation" --source-language-code "auto" --target-language-code "PL" >> a.txt

The same in powershell:

PS C:\Users\???\Documents> $aws = aws translate translate-text --text "Request" --source-language-code "auto"--target-language-code "PL"

'charmap' codec can't encode character '\u015b' in position 4: character maps to <undefined>

    PS C:\Users\???\Documents> exit

    C:\Users\???\Documents>set PYTHONIOENCODING=UTF-8

    C:\Users\???\Documents>powershell
    Windows PowerShell
    Copyright (C) 2016 Microsoft Corporation. All rights reserved.

    PS C:\Users\???\Documents> $aws = aws translate translate-text --text "Request" --source-language-code "auto"
    --target-language-code "PL"

    PS C:\Users\???\Documents> $aws

    {
        "TranslatedText": "Prośba",
        "SourceLanguageCode": "en",
        "TargetLanguageCode": "pl"
    }

Tried this solution on Windows 10, and it worked. If you're setting the environment variable from a Powershell session, the syntax is `$env:PYTHONIOENCODING='UTF-16'`. — Adam, Jul 01 '19 at 02:20

score 4 · Answer 3 · answered Jul 26 '18 at 15:37

4

I've reinstalled AWS CLI using upgraded MSI installers which now use Python 3 instead of Python 2 and the unknown encoding error is now gone.

answered Jul 26 '18 at 15:37

kenorb

155,785
88
678
743

score 3 · Answer 4 · edited Oct 22 '19 at 16:16

3

I am using git bash on windows 10 and set PYTHONIOENCODING=UTF-8 was not actually did env change. Used export PYTHONIOENCODING=UTF-8 then overcame the charmap error

edited Oct 22 '19 at 16:16

Yaser Darzi

1,480
12
24

answered Oct 22 '19 at 14:48

John Du

31
1

score 0 · Answer 5 · answered Jun 24 '20 at 07:25

0

for Windows 10 and cli is installed by python

Error: 'charmap' codec can't encode characters in position XX-XX: character maps to

Solution: run following command in command-line

set PYTHONIOENCODING=UTF-8

answered Jun 24 '20 at 07:25

Ezhil Arasan

450
3
5

Can I force AWS CLI (v1) to use UTF encoding?

5 Answers5