GET request with chunks in Python

Question

I'm getting the data from API using requests library doing like this:

import requests

url = "Some String"

headers = {

'Authorization':"Some Token"}

response = requests.request("GET", url, headers=headers)

But the file that I'm trying to get is very large so I receive the time exceptions error. How can I get it using chunks in request?

Thanks!

Possible duplicate of [How to download large file in python with requests.py?](https://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py) — RoadRunner, Jun 22 '18 at 12:45
have you seen this [SO post](https://stackoverflow.com/questions/47586741/how-to-get-large-size-json-files-using-requests-module-in-python)? i think using `stream=True` argument might help. — gyx-hh, Jun 22 '18 at 13:55

score 0 · Accepted Answer · answered Jun 27 '18 at 11:41

import requests    
import datetime
import pandas as pd


url = "some URL"
headers = {

'Authorization':"Some Token"}

start_date = datetime.datetime(2018, 6, 1)
end_date = datetime.datetime.now()
temp_end_date = start_date + datetime.timedelta(days=7)

output = dict()
while temp_end_date <= end_date:
    temp_url = url % (start_date.timestamp()*1000, temp_end_date.timestamp()*1000)
    response = requests.get(temp_url, headers=headers)
    temp_data = response.json()
    for key, value in temp_data.items():
        output_arr = output.get(key, [])
        output_arr.extend(value)
        output[key] = output_arr
    start_date = temp_end_date + datetime.timedelta(seconds=1)
    temp_end_date += datetime.timedelta(days=7)

data=output
df=pd.DataFrame(data)

df.head()

GET request with chunks in Python

1 Answers1