0

We are building this REST API layer for out partners where the architecture is something like this:

  • We will have a bunch of REST API scripts built using PHP sitting on our storage server.
  • A partner will hit our API with a transfer request
  • We will fetch the customer data files related to that transfer request and push them to the partner's server.

The problem we are having is with the data files being requested. One transfer request can can ask for hundreds of data files.

Ideally we would prefer getting a single request from the partner for the transfer. (One request that will have all the data files to be transferred).

The other way is that each data file is a separate request and so one order of a hundred files will have 100 requests. This will have additional to and fro traffic as well as individual notification for each file which can be cumbersome.

So my question is - how to format a request to a REST API to retrieve a collection of over hundred objects? If I ask the partner to POST a txt file with the data file names it will not technically be REST, right? What is the ideal way to tackle this?

Undefined Variable
  • 4,196
  • 10
  • 40
  • 69

2 Answers2

1

Client should send all the image requests with in a single Json, XML or List. That can be parsed within the web API method and provide result to the caller.

Bhasyakarulu Kottakota
  • 823
  • 1
  • 6
  • 10
  • Can I pass JSON string in a CURL get request? Also do you know if there is any restriction in size? The JSON string can easily become pretty big with all the meta data required for the transfer... – Undefined Variable Jul 02 '14 at 14:03
  • Yes size limit is there for Json. We do have several options to increase this limit still that doesn't meet yours need then try to convert the request from get to post. – Bhasyakarulu Kottakota Jul 02 '14 at 14:08
1

It sounds like you will need to handle the actual file transfers separately from the REST response, regardless of how you structure the actual API call. And I would suggest allowing an arbitrary number of files to be included in a single API call, this is, as you mention, less traffic to handle, and makes your API easier to use for the client. This does make for large payloads per request (for example a single call requesting a few thousands files would likely be up to 1MB of data), but you can fiddle with a cap on the maximum number of files per request as needed. So, your API will accept requests for file transfers, and then perhaps insert them in a queue of pending file transfers. This allows you to scale the API and the file transfer systems separately, managing the load on each as needed. For example if there is a slowdown in your transfer system, you can still accept requests via your REST API without overloading the system. A sample JSON structure would look something like this:

{
  files: [ { filepath: /path/to/file, height: 35, width: 100, type: image/jpeg  },
     { filepath: /path/to/file, height: 35, width: 100, type: image/jpeg  }, ]

}
oliakaoil
  • 1,615
  • 2
  • 15
  • 36
  • Like you said, I am planning to use a queue database in order to handle the actual file transfers. I guess my main concern is how do I pass a huge JSON query string using GET? If I use POST, then it is no longer REST compliant... – Undefined Variable Jul 02 '14 at 14:13
  • Sorry, why is using a POST call not REST compliant? – oliakaoil Jul 02 '14 at 14:14
  • We are not making a request to change the STATE of the resource (for which we should POST). If we need to get a resource/collection, we should use a GET request - I think that is what REST specifies. Here I want to transfer resource so the request should be GET to be REST compliant - right? – Undefined Variable Jul 02 '14 at 14:52
  • I see what you're saying, but you could technically argue that you are creating a new file transfer request, i.e. a request to create new resource (http://stackoverflow.com/questions/630453/put-vs-post-in-rest) – oliakaoil Jul 02 '14 at 15:06
  • Sorry, can't edit my comment again, using POST especially makes sense here, from my perspective, because you are not directly returning the files as part of the response to the API call. – oliakaoil Jul 02 '14 at 15:12