I am trying to scrape data from an API using the requests module:
r = requests.get(url, headers=headers)
pprint(r.json())
The response is 200 and the result is showing up in both Postman and Thunderclient. However, in requests (and also axios), r.text responds with this (only an extract):
'�哠a�B��6}����՛c�\x0e\x1bz:�\x1c'
'�M��7/�V�n3ґ7/6�Ӽ�tqy*����G�v?���M�!\x1fH��y�\x19��8\r'
'�\x0c'
'H���f3d���\x16<t�\x1f�\x7f��7��0=\x0b'
'��ť3~!\x05\x1f'
'\x13��E�=��q]L�`s-h�}��Ӂ4od��}ټ��\x00��b2y��z_�Kp^�_�iy���c����/o�a��u�4g��=�����͋Mf6TC��C���d4&H���\\�)g6\x7fv\x1c'
'��f^ev�K\x16�=��\x16/ԯ�\x1f'
'�\x17w�y\x15����gnm��keE�^��N˔f4���I�(�P}|bz�E�x�4Ӂy����5\x10:��Y0��0Oˎ�7�U���$&�_����2��"�+\x1d'
'�y��$U�\x12�2ɢV1�pa� k�2o¯\x11P~\x00���C�C?���cI�aa\x01�q�j:\x1e'
'\x17��c��v��#���4��a���� ��z����(uU��^�|�P���T�n�7\r'
...
'�����H�%����$�m�m�\x07U��$�PQkL\x0b'
"'J�qt{���V���4\x17r�լ�\x0b"
'\n'
'��O��O]ݬ�\x04')
and r.json() responds with this error:
JSONDecodeError Traceback (most recent call last)
File ~/path_to_venv/python3.10/site-packages/requests/models.py:910, in Response.json(self, **kwargs)
909 try:
--> 910 return complexjson.loads(self.text, **kwargs)
911 except JSONDecodeError as e:
912 # Catch JSON-related errors and raise as requests.JSONDecodeError
913 # This aliases json.JSONDecodeError and simplejson.JSONDecodeError
File /usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
343 if (cls is None and object_hook is None and
344 parse_int is None and parse_float is None and
345 parse_constant is None and object_pairs_hook is None and not kw):
--> 346 return _default_decoder.decode(s)
347 if cls is None:
File /usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
333 """Return the Python representation of ``s`` (a ``str`` instance
334 containing a JSON document).
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
File /usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
JSONDecodeError Traceback (most recent call last)
/path_to_project/app/main.py in <cell line: 7>()
23 # getting data
24 r = requests.get(url, headers=headers)
----> 26 pprint(r.json())
File ~/path_to_venv/python3.10/site-packages/requests/models.py:917, in Response.json(self, **kwargs)
915 raise RequestsJSONDecodeError(e.message)
916 else:
--> 917 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
JSONDecodeError: [Errno Expecting value] UƟQ��0a^[�%�'��F����/��1��0-�q=^��֧���"sڎ����(Jf"J����E.�p0쨧���!5����m�2K��tͮ�nI1$���R=��r�%KO%[�7z�H�*�H�
Request Headers:
User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Response headers:
{'Date': 'Fri, 03 Jun 2022 15:47:15 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'nginx', 'Vary': 'Accept-Encoding, Accept-Encoding', 'X-PX-SCORE': '0', 'x-amzn-RequestId': '8133bf64-dcd9-4492-ad32-8aeecddafb3d', 'country-code': 'BE', 'x-amz-apigw-id': 'TJv4cGjcoAMFcZQ=', 'X-Amzn-Trace-Id': 'Root=1-629a2d02-3c1891b224cea4886826fed0;Sampled=0', 'Via': '1.1 3d65275b81abaf880be10de6f2c71e9a.cloudfront.net (CloudFront), 1.1 7eefe38d76087dfad8e2f0b7702246ee.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'IAD89-P1, IAD89-C3', 'X-Cache': 'Miss from cloudfront', 'X-Amz-Cf-Id': 'n3lQfxKSnMOZMzUAnDGK3zgNHRmZ3FgrCyGdkSAtnnOzh53krckmIg==', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Content-Encoding': 'br'}