-1

I am getting this error every time i try to extract the json in this api Request_URL='https://freeserv.dukascopy.com/2.0/api/group=quotes&method=realtimeSentimentIndex&enabled=true&key=bsq3l3p5lc8w4s0c&type=swfx&jsonp=_callbacks____1kvynkpid'

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

import json
import requests
import pandas as pd

r = requests.get(Request_URL)
df = pd.DataFrame(r.json())
Bayrem
  • 17
  • 10
  • 1
    JSON that you are working with is malformed – ᴀʀᴍᴀɴ Nov 14 '21 at 17:19
  • how do I fix that ? – Bayrem Nov 14 '21 at 17:38
  • Put `print()` calls in your program to discover what you are getting back from `requests.get()`. It's clearly not what your code is assuming. – BoarGules Nov 14 '21 at 17:47
  • print `r.json()` and see what is the problem – ᴀʀᴍᴀɴ Nov 14 '21 at 17:47
  • the same error again, JSONDecodeError: Expecting value: line 1 column 1 (char 0) – Bayrem Nov 14 '21 at 18:37
  • I am sure @ᴀʀᴍᴀɴ (I will get the right user right one of these days) meant to say `print(r.text)` to see the string that is coming back to see why it does not parse as valid `JSON`. Of course, we already *know* that the call to `r.json()` throws an exception so there is no way that "printing" it is going to be helpful. – Booboo Nov 14 '21 at 21:12
  • @Booboo yeah My bad, I meant `r.text` – ᴀʀᴍᴀɴ Nov 14 '21 at 21:15
  • @ᴀʀᴍᴀɴ It's off-topic but: Why ᴀʀᴍᴀɴ and not ARMAN, which is more readily typed in English-speaking countries (this is an English-speaking website)? I had to copy and paste your name. What character set is that from? – Booboo Nov 14 '21 at 21:23
  • I don't remember why did I write it like this, I didn't know it causes a problem for mentioning @Booboo – ᴀʀᴍᴀɴ Nov 14 '21 at 21:26
  • 1
    @ᴀʀᴍᴀɴ When you want to send a message to somebody you can type `@` followed by the first couple of letters of their name and SO will show you all the matching names of people who have posted messages with that prefix and then just select from those matches. But if I type in @AR, nothing comes up and I did not even initially realize your name was not composed of ASCII characters. I have no idea how to enter ᴀʀᴍᴀɴ from my keyboard so I had to copy and paste this from your comment. Not a big deal but it took me a few seconds to realize what was going on. – Booboo Nov 14 '21 at 21:31
  • I believe you posted `Request_URL` incorrectly since I got a 404 Not found error. I suspect that it is ``'https://freeserv.dukascopy.com/2.0/api?group=quotes&method=realtimeSentimentIndex&enabled=true&key=bsq3l3p5lc8w4s0c&type=swfx&jsonp=_callbacks____1kvynkpid'`, that is with '...?group=...' instead of '.../group=...', but you will have to verify that. – Booboo Nov 14 '21 at 21:33
  • Does this answer your question? [JSONDecodeError: Expecting value: line 1 column 1 (char 0)](https://stackoverflow.com/questions/16573332/jsondecodeerror-expecting-value-line-1-column-1-char-0) – Ulrich Eckhardt Nov 14 '21 at 22:50
  • Yes it does, thanks @UlrichEckhardt – Bayrem Nov 15 '21 at 08:50
  • @Bayrem, the only thing I did is put "[python] JSONDecodeError" into the search field above to find that question. Do that before writing another question and waiting for answers, you'll have answers much faster. – Ulrich Eckhardt Nov 15 '21 at 11:51
  • much appreciated advice @UlrichEckhardt ! – Bayrem Nov 15 '21 at 22:12

1 Answers1

0

The problem is that the response coming back is in JSONP format. That is, it is JavaScript consisting of a call to a function with an argument that is a JavaScript structure (which should be a valid JSON string if it had single quotes around it, but there is no guarantee that it is). In part it looks like:

_callbacks____1kvynkpid([{"id":"10012" ...])

So we need to first remove the JavaScript call, which are the leading characters up to and including the first ( character and the final ):

import requests
import json

request_url = 'https://freeserv.dukascopy.com/2.0/api?group=quotes&method=realtimeSentimentIndex&enabled=true&key=bsq3l3p5lc8w4s0c&type=swfx&jsonp=_callbacks____1kvynkpid'

r = requests.get(request_url)
text = r.text
idx = text.index('(')
# skip everything up to and including opening '(' and then skip closing ')'
text = text[idx+1:-1]
print(json.loads(text))

Prints:

[{'id': '10012', 'title': 'ESP.IDX/EUR', 'date': '1636925400000', 'long': '71.43', 'short': '28.57'}, {'id': '10104', 'title': 'AUS.IDX/AUD', 'date': '1636925400000', 'long': '70.59', 'short': '29.41'}, {'id': '10266', 'title': 'NLD.IDX/EUR', 'date': '1636925400000', 'long': '73.48', 'short': '26.52'},

... data too big too fully reproduce

 {'id': '82862', 'title': 'MAT/USD', 'date': '1636925400000', 'long': '70.27', 'short': '29.73'}, {'id': '82866', 'title': 'ENJ/USD', 'date': '1636925400000', 'long': '72.16', 'short': '27.84'}]

In this case the structure when interpreted as a string adhered to the JSON format and so we were able to parse it with json.loads(). But what if the JavaScript structure had been (in part):

[{'id':'10012'}]

This is both legal JavaScript and legal Python, but not legal JSON because strings must be enclosed within double-quotes for it to be valid JSON. But since it is legal Python, we could use ast.literal_eval:

import requests
import ast

request_url = 'https://freeserv.dukascopy.com/2.0/api?group=quotes&method=realtimeSentimentIndex&enabled=true&key=bsq3l3p5lc8w4s0c&type=swfx&jsonp=_callbacks____1kvynkpid'

r = requests.get(request_url)
text = r.text
idx = text.index('(')
# skip everything up to and including opening '(' and then skip closing ')'
text = text[idx+1:-1]
print(ast.literal_eval(text))

Of course, for the current situation both json.loads and ast.literal_eval happened to work. However, if the JavaScript structure had been:

[{id:'10012'}]

This is valid JavaScript but, alas, not valid Python and cannot be parsed with either json.loads or ast.literal_eval.

Booboo
  • 38,656
  • 3
  • 37
  • 60