2

I have a .csv file with 400 million lines of data. I was wondering if I were to convert it into an data API which returns into JSON format, will there be any limitations if consumers were to call and GET the data API. Would it show the full content of the data API and would it take long for the API to produce an output when being called.

timothy
  • 23
  • 2

1 Answers1

0

If you convert this as GET API call then you might run into following issues:

  1. You might run into max size limit issue i.e maximum size of data you can transfer over a GET request although it will depend on you server and clients device you can refer this answer of details
  2. Latency will depend on the physical location of your server and the clients, you can potentially reduce this by cache your information if your data is not changing frequently.

Hope that helps

Community
  • 1
  • 1
Harshal Bulsara
  • 7,898
  • 4
  • 43
  • 61
  • I have currently pushed the .csv into Datastore on CKAN but everytime when i try use a GET request, it gives me a TimeOut error. – timothy Apr 07 '16 at 06:16