Pagination issue in RESTful API design

Question

I am designing a RESTful API for a mobile application I am working on. My problem is with large collections containing many items. I understand that a good practice is to paginate large number of results in a collection.

I have read the Facebook Graph API doc (https://developers.facebook.com/docs/graph-api/using-graph-api/v2.2), Twitter cursors doc (https://dev.twitter.com/overview/api/cursoring), GitHub API doc (https://developer.github.com/v3/) and this post (API pagination best practices).

Consider an example collection /resources in my API that contains 100 items named resource1 to resource100 and sorted descending. This is the response you will get upon a GET request (GET http://api.path.com/resources?limit=5):

{
    "_links": {
        "self": { "href": "/resources?limit=5&page=1" },
        "last": { "href": "/resources?limit=5&page=7" },
        "next": { "href": "/resources?limit=5&page=2" }
    },

    "_embedded": {
        "records": [ 
            { resource 100 },
            { resource 99 },
            { resource 98 },
            { resource 97 },
            { resource 96 }
        ]
    }
}

Now my problem is a scenario like this:

1- I GET /resources with above contents.

2- After that, something is added to the resources collection (say another device adds a new resource for this account). So now I have 101 resources.

3- I GET /resources?limit=5&page=2 as the initial response suggests will contain the next page of my results. The response would be like this:

{
    "_links": {
        "self": { "href": "/history?page=2&limit=5" },
        "last": { "href": "/history?page=7&limit=5" },
        "next": { "href": "/history?page=3&limit=5" }
    },

    "_embedded": {
        "records": [ 
            { resource 96 },
            { resource 95 },
            { resource 94 },
            { resource 93 },
            { resource 92 }
        ]
    }
}

As you can see resource 96 is repeated in both pages (Or similar problem may happen if a resource gets deleted in step 2, in that case one resource will be lost).

Since I want to use this in a mobile app and in one list, I have to append the resources of each API call to the one before it so I can have a complete list. But this is troubling. Please let me know if you have a suggestion. Thank you in advance.

P.S: I have considered timestamp like query strings instead of cursor based pagination, but that will make problems somewhere else for me. (let me know if you need more info about that.)

Why not use both cursor-based pagination and a timestamp? – Jonathan W Jan 26 '15 at 16:58 — Jonathan W, Jan 26 '15 at 16:58

score 3 · Answer 1 · answered Jan 25 '15 at 05:14

3

We just implemented something similar to this for a mobile app via a REST API. The mobile app passed an additional query parameter which represents a timestamp at which elements in the page should be "frozen".

So your first request would look something like GET /resources?limit=5&page=1&from=2015-01-25T05:10:31.000Z and then the second page request (some time later) would increment the page count but keep the same timestamp: GET /resources?limit=5&page=2&from=2015-01-25T05:10:31.000Z

This also gives the mobile app control if it wants to differentiate a "soft" page (preserving the timestamp of the request of page 1) from a "hard refresh" page (resetting the timestamp to the current time).

answered Jan 25 '15 at 05:14

Jonathan W

3,759
19
20

Thank you very much for the reply. Yes I am considering that approach but as I said in my P.S that may cause further problems for me. Anyways if I end up using this solution I will mark your answer as accepted. Thanks again. – Mepla Jan 25 '15 at 08:08
Well I ended up doing both curser based and a before/after timestamp. Add this to your answer and I'll accept it. Thank you very much :) – Mepla Jan 27 '15 at 05:08
Except that I don't understand why you would use both a before and an after timestamp per request, nor did you explain why only using one timestamp would cause you issues. – Jonathan W Jan 27 '15 at 05:25
@JonathanW so when do you update and save the 'from' value then ? – Petar Sep 11 '15 at 15:44
@pe60t0 In the case of the mobile app I was involved with, the app itself would do that when a pull-down refresh was issued. When doing infinite scroll towards the bottom of the list, it would send the same timestamp. Make sense? – Jonathan W Sep 11 '15 at 16:59
@JonathanW yes this makes sense, thanks. Can you summarise how do you handle this on the backend ? I mean how do you know what information to return based on the timestamp ? – Petar Sep 12 '15 at 14:48
Just page based on item creation timestamp in descending order, starting from the time the client passes and working backwards in time. – Jonathan W Sep 13 '15 at 04:48
@Mepla, I'm working on the same problem now too. At first, I used only timestamp+limit (offset is no need with changable timestamp), but I found it inappropriate because of coverage of timestamps. What should I do, if I set limit to 20, but have 21 messages per 1ms (it can be true due to some lag on server side), so I totally lost #21. In that case, I use now only primary db ids of needed entities. In my case, it is okey because of its immutability and ids incrementation (so it can be used as timestamps too). – Nexen Jan 31 '16 at 17:06

score 0 · Answer 2 · answered Jan 24 '15 at 13:01

0

Why not just maintain a set of seen resources?

Then when you process each response you can check whether the resource is already being presented.

answered Jan 24 '15 at 13:01

Stephen Souness

272
1
3

That will only prevent something to repeat in view, but in a lower level, I am wasting time performing API calls that do not get me what I want. (Imaging my step 2 if twenty resources were added) – Mepla Jan 24 '15 at 13:21

Pagination issue in RESTful API design

2 Answers2