How do I use pagination in requests

Question

Every time I run my code I get 250 records from the API. 250 is stored in max_records. The maximum amount of records is 1429. 1429 is stored in total_records. I want to extract the data in pieces of 250 till I reach 1429.

This is my code:

response1 = requests.get('https://***-***-***.***.com/odata/***', headers=headers, params=params1, proxies=proxies)

data = response1.json()

next_link = data['@odata.nextLink']
total_records = data['@odata.count']
max_records = next_link[313:]


while max_records is not total_records:
    n_response = requests.get(next_link, headers=headers, params=params1, proxies=proxies)
    if max_records == total_records:
        break

When I run above code I get the following error: MaxRetryError: HTTPSConnectionPool............

We don't know which API you use, so we can't help much. That being said, I think the issue comes from the API itself and not your code. Try perhaps to search in the API documentation if they support some kind of pagination. — bastantoine, Aug 27 '20 at 14:09
It has nothing to do with the API, its more about the code. I need an example about auto paging — Mediterráneo, Aug 27 '20 at 14:12

dh762 · Answer 1 · 2020-08-27T16:55:00.160

0

This assumes that there is a field @odata.nextLink until the last page.

I suggest to use a generator and a Session to do paging which also might help with the MaxRetryError you're getting.

import requests

session = requests.Session()

def paging():
    url = 'https://***-***-***.***.com/odata/***'
    page = session.get(url, headers=headers, params=params1, proxies=proxies).json()
    yield page

    next_url = page.get('@odata.nextLink')
    while next_url:
         page = session.get(next_url, headers=headers, params=params1, proxies=proxies).json()
         yield page
         next_url = page.get('@odata.nextLink')

for page in paging():
    # process page

You might need to construct the URL again as '@odata.nextLink' is probably relative to the URL, not absolute.

Adapted from this answer also see this answer.

edited Aug 27 '20 at 16:55

answered Aug 27 '20 at 15:53

dh762

2,259
4
25
44

I dont' have a page parameter, or maybe I don't know where to find it. – Mediterráneo Aug 27 '20 at 16:11
I can assure you that I dont have a page parameter. – Mediterráneo Aug 28 '20 at 09:47
what do you mean by `process page` – Mediterráneo Aug 30 '20 at 17:48
I did `for page in paging(): print(page)` and I received the following error: `JSONDecodeError: Expecting value` – Mediterráneo Aug 31 '20 at 07:34
is it possible to chat with each other? – Mediterráneo Aug 31 '20 at 08:14

fiverbox.com · Answer 2 · 2020-08-27T17:16:28.830

You can get the concept from this JavaScript function,

Note that I send the number of records (take=20 , skip=0) parameters to server side and process it to get them from DB,

Then send the next (take=20 , skip=20) , it will give me the next 20 records, and so on.

simple function:

function loadRecords(skip = 0, take = 20) {
                            $.getJSON('api url', {                           
                                skip: skip,
                                take: take
                            }).done(function (data) {
                                console.log(data);//data to be viewed
                            });                                
                          };

more advanced:

 function loadRecords(skip = 0, take = 20) {
                            var d = new $.Deferred(); 
                            $.getJSON('api url', {                           
                                skip: skip,
                                take: take
                            }).done(function (data) {
                                d.resolve(data);
                            });
                            return d.promise();
                          };

It seems very complex to change this JavaScript file to a Python file? — Mediterráneo, Aug 27 '20 at 14:14

How do I use pagination in requests

2 Answers2