1

My frontend web app is calling my python Flask API on an endpoint that is cached and returns a JSON that is about 80,000 lines long and 1.7 megabytes.

It takes my UI about 7.5 seconds to download all of it. ui_loading_time

It takes Chrome when calling the path directly about 6.5 seconds. enter image description here

I know that I can split up this endpoint for performance gains, but out of curiosity, what are some other great options to improve the download speed of all this content?

Options I can think of so far:

1) compressing the content. But then I would have to decompress it on the frontend

2) Use something like gRPC

Further info: My flask server is using WSGIServer from gevent and the endpoint code is below. PROJECT_DATA_CACHE is the already Jsonified data that is returned:

@blueprint_2.route("/projects")
def getInitialProjectsData():
    global PROJECT_DATA_CACHE
    if PROJECT_DATA_CACHE:
        return PROJECT_DATA_CACHE
    else:
        LOGGER.debug('No cache available for GET /projects')
        updateProjectsCache()
        return PROJECT_DATA_CACHE
NateW
  • 2,856
  • 5
  • 26
  • 46

2 Answers2

1

Maybe you could stream the file? I cannot see any way to transfer a file 80,000 lines long without some kind of download or wait.

This would be an opportunity to compress and decompress it, like you suggested. Definitely make sure that the JSON is minified.

  1. One way to minify a JSON: https://www.npmjs.com/package/json-minify
  2. Streaming a file: https://blog.al4.co.nz/2016/01/streaming-json-with-flask/

It also really depends on the project, maybe you could get the users to download it completely?

davidism
  • 121,510
  • 29
  • 395
  • 339
Coolq B
  • 344
  • 5
  • 10
1

The best way to do this is to break your JSON into chunks and stream it by passing a generator to the Response. You can then render the data as you receive it or show a progress bar displaying the percentage that is done. I have an example of how to stream data as a file is being downloaded from AWS s3 here. That should point you in the right direction.

Allie Fitter
  • 1,689
  • 14
  • 18