0

So, im using GNIP Decahose. BACKGROUND: Once you log into this URL you get a stream of data in a json format that goes on continuously unless you specify how long you want it to go on for

I want to be able to access the Decahose URL and enter my credentials. Once I enter my credentials I would like to only access the URL for about 30 sec and save the json data file to data frame or something.

import requests
import urllib.request as urllib2
from requests.auth import HTTPBasicAuth

#credentials
url = "my url"
un= "my username"
pw = "my password"

runtime = 30 #SECONDS
chunksize = 2000 #how much data i want to access
newline = "\r\n"



h = { 'Accept': 'application/json','Connection': 'Keep-Alive','Accept-Encoding' : 'gzip','Authorization' : 'Basic %s' % base64.encodestring('%s:%s' % (un, pw))  }


req = urllib2.Request(decaurl, headers=h)
response = urllib2.urlopen(req, timeout=1+gnip)
decompressor = zlib.decompressobj(16+zlib.MAX_WBITS)
remainder = ''

while True:
  tmp = ''
  try:
    tmp = decompressor.decompress(response.read(CHUNKSIZE))
  except httplib.IncompleteRead as e:
    with err_lock:
      sys.stderr.write("Incomplete Read: %s"%(str(e))) 
     tmp=''
Tim Roberts
  • 48,973
  • 4
  • 21
  • 30
Mjude
  • 1

0 Answers0