UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 0: ordinal not in range (256)

Question

I have a script that runs just fine on the local but fails with a UnicodeEncodeError message on a gitlab runner.

Actual error message: UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 0: ordinal not in range(256)

What does this mean and how do I resolve this?

Below is the piece of code.

import requests
def get_subgroups(group_id):
    subgroups_ids = []
    subgroup_qry = 'https://gitlab.example.com/api/v4/groups'+group_id+'subgroups?pagination=keyset&per_page=500'
    response = requests.get(subgroup_qry, headers=gl_auth_header)
    if response.status_code == 200:
        subgroups = json.loads(response.text)
        try:
            for subgroup in subgroups:
                subgroup_ids.append(subgroup['id'])
        except Exception as e:
            print(e)
        return subgroups_ids
    else:
        print("Invalid response", response.status_code)

sub_groups = get_subgroups(GROUP_ID)

Below is the complete error message:

Traceback (most recent call last):
188  File "gitlab_scripts/protected_vars.py", line 95, in <module>
189    sub_groups = get_subgroups(GROUP_ID)
190  File "gitlab_scripts/protected_vars.py", line 37, in get_subgroups
191    response = requests.get(subgroup_qry, headers=gl_auth_header)
192  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/api.py", line 75, in get
193    return request('get', url, params=params, **kwargs)
194  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/api.py", line 61, in request
195    return session.request(method=method, url=url, **kwargs)
196  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
197    resp = self.send(prep, **send_kwargs)
198  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/sessions.py", line 645, in send
199    r = adapter.send(request, **kwargs)
200  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/requests/adapters.py", line 440, in send
201    resp = conn.urlopen(
202  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
203    httplib_response = self._make_request(
204  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
205    conn.request(method, url, **httplib_request_kw)
206  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
207    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
208  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/http/client.py", line 1282, in request
209    self._send_request(method, url, body, headers, encode_chunked)
210  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/http/client.py", line 1323, in _send_request
211    self.putheader(hdr, value)
212  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/site-packages/urllib3/connection.py", line 224, in putheader
213    _HTTPConnection.putheader(self, header, *values)
214  File "/usr/local/lib/pyenv/versions/3.10.2/lib/python3.10/http/client.py", line 1255, in putheader
215    values[i] = one_value.encode('latin-1')
216 UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 0: ordinal not in range(256)

why don't use UTF-8? or at least CP1252. But you should really avoid any non-Unicode encodings nowadays — phuclv, Aug 28 '22 at 03:08
[python requests.get() returns improperly decoded text instead of UTF-8?](https://stackoverflow.com/q/44203397/995714), [python json load set encoding to utf-8](https://stackoverflow.com/q/46408051/995714) — phuclv, Aug 28 '22 at 07:56
Thanks for sharing, but sadly none of the solutions worked :( — blackPanther, Aug 28 '22 at 19:41
Why has this been marked as duplicate? I don't see how the linked post answers the question. — Rik Mulder, Jan 11 '23 at 20:31

score 0 · Answer 1 · answered Aug 28 '22 at 03:03

0

You have the unicode character LEFT DOUBLE QUOTATION MARK \u201c in your gl_auth_header. Which requests is trying to encode in latin 1 encoding which it cannot represent. Remove the left quote from your headers.

answered Aug 28 '22 at 03:03

nlta

1,716
1
7
17

Thanks, how do you know it's the gl_auth_header? BTW, I'm reading it's value from environment and it looks fine. – blackPanther Aug 28 '22 at 05:40
Lines 211 and 213 in the stack trace say put_header. So that's probably encoding the header you include. Can you include gl_auth_header in your post so we can see it? – nlta Aug 28 '22 at 19:20
`gl_token = os.environ.get('GL_DATA_GROUP_TOKEN')` `gl_auth_header = {'PRIVATE-TOKEN': gl_token}` – blackPanther Aug 28 '22 at 19:24
@blackPanther what you've pasted here has a 0x27 single quote but I don't think that's actually what present in the ENV variable based on the error message. – nlta Sep 03 '22 at 05:49

UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 0: ordinal not in range (256)

1 Answers1