RESTful API - what happens if the records' count increases while fetching data with pagination?

Question

Let's assume I need to retrieve all the products from my RESTful API server.

So I call http://myserver/products/?page1 and I retrieve all the products up to http://myserver/products/?pageX.

But, unfortunately :-), during this process, a new product was added, and the products' count increased while I was retrieving my products.

How can I deal with this problem? I guess, depending if the product was added to the first/last pages I may be lucky to retrieve it during the process, but if I would use some kind of metadata with the number of products, like:

  {
  "_metadata": 
  {
      "page": 5,
      "per_page": 20,
      "page_count": 20,
      "total_count": 521,
      "Links": [
        {"self": "/products?page=5&per_page=20"},
        {"first": "/products?page=0&per_page=20"},
        {"previous": "/products?page=4&per_page=20"},
        {"next": "/products?page=6&per_page=20"},
        {"last": "/products?page=26&per_page=20"},
      ]
  },
  "records": [
    {
      "id": 1,
      "name": "Widget #1",
      "uri": "/products/1"
    },
    {
      "id": 2,
      "name": "Widget #2",
      "uri": "/products/2"
    },
    {
      "id": 3,
      "name": "Widget #3",
      "uri": "/products/3"
    }
  ]
}

my application would notice that not all products were retrieved at the end of the process..?

Thank you

score 0 · Answer 1 · edited May 23 '17 at 12:17

Please note that I don't think the gist or your question has a different answer in a restful environment than in a non-restful environment.

If I understand you right you'd like to do a selection of items by certain criterion e.g. "all", page thru these selected items and if the selection would increase or get smaller during the paging you'd like to react on that.

There for your rest-api will have some kind of means to

specify the selection e.g. "all"
specify the current record pointer e.g. "5th record"
handle the information about the total count and page e.g. 15 records on 3 pages current 5th record on page 1 (which should n't have to be part of the restful url in your scenario)

I'd breakdown the problem into three parts:

keeping track of the selection
pagination
pointer to the current record

keeping track of the selection

Your worst case is if you loose the pointer to the current record since it was deleted - so your problem is also linked to the transactional problem which you could solve by

locking the current record
locking all records of the current page
locking the whole selection
locking records selected by the user

where the feasibility of these approaches depends a lot on how high the probability is for changes in the set of locked records. The probabily is obviously lower the smaller the set of locked records is. Even if locking might be technical feasible it must also make sense. E.g. in any kind of reservation system option 4 "locking records selected by the user" is what you'd need to avoid deletion of relevant records.

Added and deleted records are first detected by keeping track of the selection. Here it is a question which data you need to keep track of the changes. The following approaches have different "cost" involved:

just getting the count of all records - for pagination this might be enough but it might happen that the same number of records where deleted and added and the count stays the same but some of the relevant / selected records changed which is annoying to the user.
getting the change data for all modified records (id, sort position, date of modification, date of deletion/addition and title of the record) and keeping track of the current pagination by adding the modifications to the list of to titles to be displayed
getting all data for all modified records and keeping track of the full list by a synchronization mechanism and caching
getting all data fo all records

option 4 is the easiest but most costly approach and not feasible for big selections

pagination

You'll find quite a few question and answers on this topic e.g. Smart pagination algorithm

pointer to the current record

If there are additions and deletions to the selection your "current record" may change which includes the option that it get's invalid/deleted discussed above. Also you current record might end up on another page. At least for the user the experience to see these shifts is not ideal but it is certainly feasible to deal with the situation.

E.g. Let's assume we selected 15 records on 3 pages with 5 entries each. The user was at record 5 of 15 (the last item on page 1) he presses "next" and would expect to end up on page 2 with record 6. In the meantime there are now 17 records. 1 was deleted on page 1 and 2 where added so that there is now a fourth page. The user will stay on page 1 since his record with sorting number 5 still has sorting number 5

RESTful API - what happens if the records' count increases while fetching data with pagination?

1 Answers1