Aggregate multiple APIs request results using python

Question

I'm working on an application that will have to use multiple external APIs for information and after processing the data, will output the result to a client. The client uses a web interface to query, once query is send to server, server process send requests to different API providers and after joining the responses from those APIs then return response to client.

All responses are in JSON.

current approach:

import requests
def get_results(city, country, query, type, position):
    #get list of apis with authentication code for this query 
    apis = get_list_of_apis(type, position) 
    results = [ ]
    for api in apis:
        result = requests.get(api)
        #parse json 
        #combine result in uniform format to display 
    return results

Server uses Django to generate response.
Problem with this approach
(i) This may generate huge amounts of data even though client is not interested in all.
(ii) JSON response has to be parsed based on different API specs.

How to do this efficiently?

Note: Queries are being done to serve job listings.

@AlexMorozov (i) Query retrieval (ii) Combining results from different API — sonus21, Jan 27 '16 at 08:04
How about using aiohttp as mentioned here: https://stackoverflow.com/a/53022415/1333610 ? — arun, Jul 02 '20 at 15:28

score 0 · Answer 1 · answered Jan 27 '16 at 07:44

Most APIs of this nature allow for some sort of "paging". You should code your requests to only draw a single page from each provider. You can then consolidate the several pages locally into a single stream.

If we assume you have 3 providers, and page size is fixed at 10, you will get 30 responses. Assuming you only show 10 listings to the client, you will have to discard and re-query 20 listings. A better idea might be to locally cache the query results for a short time (say 15 minutes to an hour) so that you don't have to requery the upstream providers each time your user advances a page in the consolidated list.

As far as the different parsing required for different providers, you will have to handle that internally. Create different classes for each. The list of providers is fixed, and small, so you can code a table of which provider-url gets which class behavior.

You are correct but it increases overhead on creating too many classes to parse data. — sonus21, Jan 28 '16 at 10:26

score 0 · Answer 2 · answered Jan 25 '17 at 18:42

Shameless plug but I wrote a post on how I did exactly this in Durango REST framework here.

I highly recommend using Django REST framework, it makes everything so much easier

Basically, the model on your APIs end is extremely simple and simply contains information on what external API is used and the ID for that API resource. A GenericProvider class then provides an abstract interface to perform CRUD operations on the external source. This GenericProvider uses other providers that you create and determines what provider to use via the provider field on the model. All of the data returned by the GenericProvider is then serialised as usual.

Hope this helps!

Good one, more and less same method written by Austin. But the catch here is most of the REST APIs providers don't provide libraries to do most of the stuff like, so in that case only way to go is parsed retrieved JSON and send the result back to client. — sonus21, Jan 26 '17 at 06:07

Aggregate multiple APIs request results using python

2 Answers2