How to specify requests to return data as JSON?

Question

I'm new to the Python requests module and I'm trying to export game data from a user from the Lichess.org API (a chess website).

Here is my code:

import requests, json

url = "https://www.lichess.org/api/games/user/mbellm"

r = requests.get(url, params={"max":2, "analysed":"true", "clocks":"true", "evals":"true", "opening":"true"})
r_text = r.content.decode("utf-8"))
print(r_text)
#data = json.loads(r_text)

The output received in r_text however seems to not be in JSON/NDJSON format, but rather in PGN format (a type of chess notation). In the section I linked to the API documentation above, it states that you can specify whether to receive data in JSON or PGN format but I don't see anywhere where it says how to choose the format which you receive it in.

How can I set my code to return the fetched data in JSON/NDJSON format?

Current output from print(r_text):

[Event "Rated Bullet game"]
[Site "https://lichess.org/IqlbjkHX"]
[Date "2019.01.13"]
[Round "-"]
[White "mbellm"]
[Black "Ruediruempel"]
[Result "0-1"]
[UTCDate "2019.01.13"]
[UTCTime "22:45:48"]
[WhiteElo "1097"]
[BlackElo "1202"]
[WhiteRatingDiff "-8"]
[BlackRatingDiff "+7"]
[Variant "Standard"]
[TimeControl "60+0"]
[ECO "C50"]
[Termination "Normal"]

1. e4 e5 2. Nf3 Nf6 3. Bc4 Nc6 4. d3 Bc5 5. Nc3 Nd4 6. O-O O-O 7. Nxd4 Bxd4 8. Nd5 Nxd5 9. Bxd5 c6 10. c3 cxd5 11. cxd4 dxe4 12. Qg4 exd3 13. dxe5 d6 14. Bd2 Bxg4 15. exd6 Qxd6 16. f3 Bf5 17. Rad1 h6 18. a3 Bh7 19. Bb4 Qd4+ 20. Kh1 Qxb2 21. Rxd3 Bxd3 22. Rc1 Qxc1+ 23. Be1 Qxe1# 0-1


[Event "Rated Classical game"]
[Site "https://lichess.org/sgNWdvkn"]
[Date "2019.01.13"]
[Round "-"]
[White "Stalingrad_1"]
[Black "mbellm"]
[Result "1-0"]
[UTCDate "2019.01.13"]
[UTCTime "04:15:39"]
[WhiteElo "1656"]
[BlackElo "1732"]
[WhiteRatingDiff "+13"]
[BlackRatingDiff "-13"]
[Variant "Standard"]
[TimeControl "900+15"]
[ECO "D00"]
[Termination "Normal"]

1. d4 d5 2. e3 Nf6 3. c4 c6 4. Qb3 Nbd7 5. Nc3 Nb6 6. cxd5 Nfxd5 7. Nxd5 cxd5 8. Bb5+ Bd7 9. Nf3 e6 10. Bxd7+ Qxd7 11. Ne5 Qc7 12. Bd2 Bd6 13. Rc1 Qe7 14. Qb5+ Kf8 15. f4 Bxe5 16. dxe5 Nc4 17. Bb4 1-0

In your "get" request, try setting the "Accept" or "Content-type" header to "application/json". I haven't read the API documentation so I'm only guessing so leaving this as a comment — John, Jan 15 '19 at 02:37
@John to confirm then the `request.get` line would say `r = requests.get(url, params={"max":2, "analysed":"true", "clocks":"true", "evals":"true", "opening":"true"}, headers={"Accept":"application/json"} )` — Matt, Jan 15 '19 at 02:39
that looks good, I also amended to try "Content-type" header as well, but your syntax looks good. — John, Jan 15 '19 at 02:40
@John Ah but you led me to the solution, rather than `"application/json"` the syntax is `"application/x-ndjson"` — Matt, Jan 15 '19 at 02:44

Henry Woody · Accepted Answer · 2019-01-15T04:54:48.140

4

It looks like the default format for this endpoint is application/x-chess-pgn, however application/x-ndjson is also available. (This from the API docs)

To tell the API you want application/x-ndjson, you can use the Accept header.

For example:

import requests, json

url = "https://www.lichess.org/api/games/user/mbellm"

r = requests.get(
    url,
    params={"max":2, "analysed":"true", "clocks":"true", "evals":"true", "opening":"true"},
    headers={"Accept": "application/x-ndjson"}
)
r_text = r.content.decode("utf-8")
print(r_text)

Sample of r_text:

{"id":"0ngCNyCN","rated":true,"variant":"standard","speed":"classical","perf":"classical","createdAt":1547495368806,"lastMoveAt":1547498443462,"status":"resign","players":{"white":{"user":{"id":"mbellm","name":"mbellm"},"rating":1715,"ratingDiff":-9,"analys

Edit

Since the response from this API is ndjson or Newline Delimed JSON, calling json.loads will not work properly on the response data. You can split the response data on the new line character "\n" and then call json.loads on each substring. Note that the response ends with "\n", so be sure not to call json.loads on the final substring.

Here's an example:

games = [json.loads(s) for s in r_text.split("\n")[:-1]]

edited Jan 15 '19 at 04:54

answered Jan 15 '19 at 02:44

Henry Woody

14,024
7
39
56

1

@Matt for sure! I'm getting a `JSONDecodeError` from calling `json.loads(r_text)`, but i think that's because the response has a few JSON objects inside of it separated by newline (`\n`) characters. If you split `r_text` on `"\n"` then call `json.loads` on each substring (not the last one though) you can parse the response – Henry Woody Jan 15 '19 at 02:49
Yeah maybe because this is NDJSON rather than regular JSON `json.loads` doesn't work. I wonder if json has a method for converting NDJSON to JSON... Would be a bit cleaner but thank you for the follow-up regardless – Matt Jan 15 '19 at 02:51
1

@Matt oh yeah good point, and yeah I'll bet it's worth looking into – Henry Woody Jan 15 '19 at 02:53
one last question, the code you posted in your answer doesn't actually seem to yield the `\n` in the output of `r_text` since after it is decoded to utf-8 the `\n`s get removed. Because of that I can't seem to figure out how you are getting `json.loads` to work on it. Would you mind specifying in your answer the exact code you used to get it to load into JSON format – Matt Jan 15 '19 at 03:03
1

Try `[json.loads(x) for x in r_text.split("\n")[:-1]]` – Henry Woody Jan 15 '19 at 03:05
been working on this for the last couple hours, maybe you can help. Are you able to use the `indent=2` parameter on the newly created json (`output = json.dumps(new_json, indent=2`), I can't get it to work and I need the JSON to be somewhat readable – Matt Jan 15 '19 at 04:29
1

@Matt yeah that's working for me, using exactly the code in my answer and for either `games` or `games[0]` it seems to work fine. What's the error you're getting? – Henry Woody Jan 15 '19 at 04:57
No error message but it just isn't indenting when I print `games2` after calling `games2 = json.dumps(games, indent=2)` – Matt Jan 15 '19 at 05:00
1

Nevermind, it just worked now I have no idea what I've been doing wrong all this time. Thanks so much for your help – Matt Jan 15 '19 at 05:02
@HenryWoody Just a curious question, how does one figure out to use the headers and to use the 'Accept' key? What is the thought process behind this? – ycx Jan 15 '19 at 17:47
@ycx Generally, information about the HTTP transaction (request/response) goes in the [Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). [Accept](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) is the header for deciding what format the server should send its responses in. All of this is just convention though, so it's up to the people making the API to actually implement these things in this way. – Henry Woody Jan 15 '19 at 18:48

score 2 · Answer 2 · answered Jan 15 '19 at 02:45

According to API and , you should try to add a header in ur request, e.g.:

headers = {
    "accept":"application/x-ndjson"
}

so ur request should become:

r = requests.get(url, params={"max":2, "analysed":"true", "clocks":"true", "evals":"true", "opening":"true"}, headers=headers)

the result should be(FYI):

{"id":"0ngCNyCN","rated":true,"variant":"standard","speed":"classical","perf":"classical","createdAt":1547495368806,"lastMoveAt":1547498443462,"status":"resign","players":{"white":{"user":{"id":"mbellm","name":"mbellm"},"rating":1715,"ratingDiff":-9,"analysis":{"inaccuracy":6,"mistake":8,"blunder":5,"acpl":83}},"black":{"user":{"id":"mecfvm","name":"mecfvm"},"rating":1766,"ratingDiff":9,"analysis":{"inaccuracy":9,"mistake":5,"blunder":3,"acpl":63}}},"winner":"black","opening":{"eco":"C50","name":"Italian Game: Giuoco Pianissimo, Italian Four Knights Variation","ply":9},"moves":"e4 e5 Nf3 Nc6 Bc4 Bc5 d3 Nf6 Nc3 d6 h3 Be6 Bxe6 fxe6 Bg5 h6 Bh4 Qd7 a3 O-O-O Na4 g5 Nxc5 dxc5 Bg3 Nd4 Nxe5 Qe8 O-O Nh5 Bh2 Nf4 Re1 Rf8 Nf3 Nxf3+ Qxf3 e5 Bxf4 gxf4 a4 a6 Red1 Rg8 Kh2 h5 c3 Qf7 Ra3 Rg7 b4 cxb4 cxb4 Kb8 b5 Qe7 Rb3 axb5 axb5 Rdg8 Rg1 Qe6 Rb2 Qb6 Qxh5 Qd4 Rd2 Qc3 Rdd1 Qb2 Qf3 Qxb5 d4 exd4 Rxd4 Qe5 Qd3 Qe7 Ra4 Qe5 Rd4 f3+ g3 c5 Rd8+ Rxd8 Qxd8+ Ka7 Qd5 Qxd5 exd5 Rd7 Rd1 b5 g4 c4 Kg3 c3 Kxf3 b4 Ke4 c2 Rc1 b3 Kd3 Rxd5+ Kc3 Rb5 Kb2 Ka6 f4 Ka5 f5 Kb4 f6 Ra5 Rxc2 Ra2+ Kc1","analysis":[{"eval":12},{"eval":37},{"eval":23},{"eval":15},{"eval":10},{"eval":24},{"eval":7},{"eval":31},{"eval":0},{"eval":18},{"eval":0},{"eval":24},{"eval":58},{"eval":7},{"eval":0},{"eval":0},{"eval":-53,"best":"g5d2","variation":"Bd2 Nd4 Na4 Nxf3+ Qxf3 Bd4 c3 Bb6 Qg3 Qe7 O-O Nh5 Qg4 Qf7","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Bd2."}},{"eval":-21},{"eval":-71,"best":"c3a4","variation":"Na4 Bb6 c3 g5 Bg3 O-O-O Nd2 Qg7 Nxb6+ axb6 a4 d5 Qe2 h5","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Na4."}},{"eval":-43},{"eval":-46},{"eval":0},{"eval":0},{"eval":12},{"eval":12},{"eval":200,"best":"d7d6","variation":"Qd6 O-O Rhg8 b4 a6 Re1 g4 hxg4 Rxg4 Bh4 Rg6 Rb1 cxb4 axb4","judgment":{"name":"Mistake","comment":"Mistake. Best move was Qd6."}},{"eval":192},{"eval":196},{"eval":175},{"eval":270,"best":"h8f8","variation":"Rf8 f3","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Rf8."}},{"eval":166,"best":"c2c3","variation":"c3 Nb5 Qe2 Kb8 Qe3 Nd6 Bh2 Rg8 b4 Nf7 Rfd1 Nxe5 Bxe5 Qf7","judgment":{"name":"Mistake","comment":"Mistake. Best move was c3."}},{"eval":180},{"eval":151},{"eval":150},{"eval":-451,"best":"b2b4","variation":"b4","judgment":{"name":"Blunder","comment":"Blunder. Best move was b4."}},{"eval":-32,"best":"f4h3","variation":"Nxh3+","judgment":{"name":"Blunder","comment":"Blunder. Best move was Nxh3+."}},{"eval":-18},{"eval":75,"best":"f4d3","variation":"Nxd3 Qg3 Nf4 Qc3 b6 b4 Qb5 Bxf4 gxf4 Qe5 Qc6 Red1 Rxd1+ Rxd1","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Nxd3."}},{"eval":79},{"eval":77},{"eval":0,"best":"b2b4","variation":"b4 Qc6 Qh5 Rfe8 Reb1 Kb8 b5 Qd7 Kh2 c4 dxc4 Qd2 Qf3 Qxc2","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was b4."}},{"eval":40},{"eval":0},{"eval":39},{"eval":27},{"eval":24},{"eval":26},{"eval":119,"best":"c8b8","variation":"Kb8","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Kb8."}},{"eval":0,"best":"a1b1","variation":"Rab1 Rd6 b4 cxb4 Rxb4 c5 Rb2 Rdg6 Rg1 Qg7 a5 Kb8 Rb6 Rg5","judgment":{"name":"Mistake","comment":"Mistake. Best move was Rab1."}},{"eval":46},{"eval":50},{"eval":41},{"eval":45},{"eval":75},{"eval":87},{"eval":134},{"eval":131},{"eval":195,"best":"e7e6","variation":"Qe6","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Qe6."}},{"eval":174},{"eval":174},{"eval":165},{"eval":211},{"eval":103,"best":"b3a3","variation":"Ra3 Rg5 Kh1 Kc8 Ra4 Qb3 Ra8+ Kd7 Rxg8 Qxg8 d4 exd4 b6 Qf7","judgment":{"name":"Mistake","comment":"Mistake. Best move was Ra3."}},{"eval":298,"best":"e6g4","variation":"Qg4 Qxg4 hxg4 g3 gxh3 Rc2 Rh7 g4 Rd7 Rc5 Rxd3 Rxe5 Rf3 g5","judgment":{"name":"Mistake","comment":"Mistake. Best move was Qg4."}},{"eval":255},{"eval":439,"best":"b6e6","variation":"Qe6 Qf3","judgment":{"name":"Mistake","comment":"Mistake.Best move was Qe6."}},{"eval":272,"best":"b2a2","variation":"Ra2 Rg5 Qd1 Qd6 Qa4 Kc8 Qa8+ Kd7 Qxb7 Qc5 Qa7 Qxa7 Rxa7 Kd6","judgment":{"name":"Mistake","comment":"Mistake. Best move was Ra2."}},{"eval":322,"best":"d4b4","variation":"Qb4 Qd1 Qxb5 d4 f3 g3 Rh7 dxe5 Qxe5 Rd8+ Rxd8 Qxd8+ Ka7 Rb1","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Qb4."}},{"eval":235,"best":"h5d1","variation":"Qd1 Qa5 Rb2 Qa3 Qc2 f3 g4 Qf8 Rg3 Rh8 Qd2 Qf4 Qxf4 exf4","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Qd1."}},{"eval":234},{"eval":221},{"eval":272,"best":"g7d7","variation":"Rd7 Ra1 Rdg7 Rab1 Qd2 Kh1 Qa2 Rbd1 Qb3 d4 Qxf3 gxf3 Rxg1+ Rxg1","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Rd7."}},{"eval":221},{"eval":344,"best":"g7g5","variation":"Rg5","judgment":{"name":"Mistake","comment":"Mistake. Best move was Rg5."}},{"eval":300},{"eval":340},{"eval":191,"best":"d4d5","variation":"Rd5 Qe7 Rf5 Rg6 e5 Rg5 Rxg5 Rxg5 Rb1 c6 Re1 Rf5 Qe4 Rf8","judgment":{"name":"Mistake","comment":"Mistake. Best move was Rd5."}},{"eval":271,"best":"c7c6","variation":"c6 Rd8+","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was c6."}},{"eval":238},{"eval":300,"best":"c7c6","variation":"c6","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was c6."}},{"eval":182,"best":"d3a3","variation":"Qa3","judgment":{"name":"Mistake","comment":"Mistake. Best move was Qa3."}},{"eval":277,"best":"e5g5","variation":"Qg5 Kh1","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Qg5."}},{"eval":239},{"eval":425,"best":"g7h7","variation":"Rh7","judgment":{"name":"Mistake","comment":"Mistake. Best move wasRh7."}},{"eval":422},{"eval":415},{"eval":400},{"eval":375},{"eval":0,"best":"d8a5","variation":"Qa5+","judgment":{"name":"Blunder","comment":"Blunder. Best move was Qa5+."}},{"eval":0},{"eval":75},{"eval":459,"best":"b7b5","variation":"b5","judgment":{"name":"Blunder","comment":"Blunder. Best move was b5."}},{"eval":-157,"best":"g3g4","variation":"g4","judgment":{"name":"Blunder","comment":"Blunder. Best move was g4."}},{"eval":-158},{"eval":-106},{"eval":-34},{"eval":-308,"best":"h3h4","variation":"h4 c3 Rc1 b4 h5 Rb7 d6 b3 Rxc3 b2 d7 Rxd7 Rb3 Rb7","judgment":{"name":"Mistake","comment":"Mistake. Best move was h4."}},{"eval":-288},{"eval":-380,"best":"d1c1","variation":"Rc1 b4","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Rc1."}},{"eval":-298},{"eval":-631,"best":"d1c1","variation":"Rc1 Kb6","judgment":{"name":"Blunder","comment":"Blunder. Best move was Rc1."}},{"eval":-443},{"eval":-641,"best":"d1a1","variation":"Ra1+ Kb6 Kd3 b3 Kc3 Rxd5 Kxb3 Rd1 Kxc2 Rxa1 h4 Kc5 Kd3 Kd5","judgment":{"name":"Mistake","comment":"Mistake. Best move was Ra1+."}},{"eval":-641},{"eval":-525},{"eval":-347},{"eval":-305},{"eval":9,"best":"d5d1","variation":"Rd1 Rxc2 bxc2 Kxc2 Rh1 Kd3 Kb6 Ke4 Kc6 Ke5 Kd7 Kf6 Rxh3 f4","judgment":{"name":"Blunder","comment":"Blunder. Best move was Rd1."}},{"eval":9},{"eval":27},{"eval":0},{"eval":0},{"eval":-88,"best":"c1e1","variation":"Re1 Rb7 f5Kb4 Re3 Ka4 Re4+ Kb5 Re1","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Re1."}},{"eval":-111},{"mate":-11,"best":"c1f1","variation":"Rf1 Rd5 Rf3 Ka4 Rf4+ Ka5 Rc4 Rd1 Kxb3 c1=Q Rxc1 Rxc1 f6 Rf1","judgment":{"name":"Blunder","comment":"Checkmate is now unavoidable. Best move was Rf1."}},{"mate":-10},{"mate":-5},{"mate":-4},{"mate":-4}],"clock":{"initial":900,"increment":15,"totalTime":1500}}
{"id":"i7LN6liR","rated":true,"variant":"standard","speed":"classical","perf":"classical","createdAt":1547448694135,"lastMoveAt":1547450191465,"status":"resign","players":{"white":{"user":{"id":"mbellm","name":"mbellm"},"rating":1726,"ratingDiff":-11,"analysis":{"inaccuracy":2,"mistake":1,"blunder":1,"acpl":46}},"black":{"user":{"id":"endasan","name":"endasan"},"rating":1719,"ratingDiff":11,"analysis":{"inaccuracy":1,"mistake":2,"blunder":0,"acpl":22}}},"winner":"black","opening":{"eco":"B30","name":"Sicilian Defense: Old Sicilian","ply":4},"moves":"e4 c5Nf3 Nc6 Bc4 e6 Nc3 Nf6 d3 d5 exd5 exd5 Bb3 Be7 O-O O-O a3 Be6 Ng5 Nd4 Nxe6 Nxe6 Qf3 Nd4 Qd1 Nxb3 cxb3 a6 d4 h6 dxc5 Bxc5 Na4 Rc8 Nxc5 Rxc5 Be3 Rc8 Qd4 b5 Rac1 Qd6 Bf4 Qe6 Rfe1 Rxc1 Rxc1 Rc8 h3 Rxc1+ Bxc1 Qe1+","analysis":[{"eval":12},{"eval":18},{"eval":19},{"eval":8},{"eval":-20},{"eval":0},{"eval":-21},{"eval":-23},{"eval":-20},{"eval":-9},{"eval":-53},{"eval":-43},{"eval":-54},{"eval":-31},{"eval":-37},{"eval":-41},{"eval":-57},{"eval":-52},{"eval":-172,"best":"h2h3","variation":"h3","judgment":{"name":"Mistake","comment":"Mistake. Best move was h3."}},{"eval":-68,"best":"e6g4","variation":"Bg4","judgment":{"name":"Mistake","comment":"Mistake. Best move was Bg4."}},{"eval":-71},{"eval":-28},{"eval":-99,"best":"f1e1","variation":"Re1 h6 Ne2 Qc7 c3 Bd6 Ng3 Rfe8 Be3 d4 Bd2 Rac8 Bxe6 fxe6","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was Re1."}},{"eval":-82},{"eval":-91},{"eval":-79},{"eval":-109},{"eval":-10,"best":"d5d4","variation":"d4 Ne2 Qd5 Bd2 a5 Re1 Rfe8 Nf4 Qf5 Qf3 Bd6 Qh3 Qg5 Qf3","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was d4."}},{"eval":-12},{"eval":0},{"eval":-43},{"eval":-36},{"eval":-118,"best":"b3b4","variation":"b4 Ba7","judgment":{"name":"Inaccuracy","comment":"Inaccuracy. Best move was b4."}},{"eval":9,"best":"c5a7","variation":"Ba7","judgment":{"name":"Mistake","comment":"Mistake. Best move was Ba7."}},{"eval":0},{"eval":6},{"eval":0},{"eval":16},{"eval":14},{"eval":31},{"eval":23},{"eval":36},{"eval":20},{"eval":13},{"eval":0},{"eval":3},{"eval":9},{"eval":8},{"eval":-670,"best":"c1c8","variation":"Rxc8+ Qxc8 h3 Qe6 Be3 Nd7 Qa7 Ne5 a4 bxa4 bxa4 Nc4 b3 Nxe3","judgment":{"name":"Blunder","comment":"Blunder. Best move was Rxc8+."}},{"eval":-658},{"eval":-637},{"eval":-621}],"clock":{"initial":900,"increment":15,"totalTime":1500}}

Just a curious question, how does one figure out to use the headers and to use the 'Accept' key? What is the thought process behind this? — ycx, Jan 15 '19 at 17:47
This is a basic [http](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) structure, FYI. — Lau Real, Jan 16 '19 at 02:13

How to specify requests to return data as JSON?

2 Answers2

Linked