0

I am fetching data from a URL and saving it as a JSON file with the following code:

import json
import urllib2
urllink=urllib2.urlopen("link/api/phpfile?auth=key&sql=select * from table").read()
print urllink
print len(urllink)
#decoding
result=json.loads(urllink)['result']
print result

with open('Fact_CMCharges.json','w') as outfile:
    json.dump(result,outfile)

This code was working fine for moderately sized data. The problem arises when I read a big JSON file through urlopen().

I am going to read around 2800 thousand data from the URL, but for such big data I am not getting the proper response.

I have the idea to read the JSON data partially with rowid and save it to a file, but I am confused about how to save all partial read data to a single existing file and make sure it is valid JSON. For example, if the original structure is [{ some data }], and I edit the locally saved JSON file by adding partial JSON data then I think it will be saved as [{ some data }][{ new data}], but I want all data in the form [{ some data + more data }].

My sample of JSON data is:

[{"TxnDate": "Jan  1 2009 12:00:00:000AM", "TxnDateInt": "20090101", "LeaseID": "CANAGEO1", "FCWorkBook": "ZZZ None", "FASBEndDate": "", "FCLeaseNo": "0", "GLPostedStatus": "Posted", "SourceCode": "CH", "PaidStatus": "Paid", "FASBStartDate": "", "ActualProjected": "Charges", "Frequency": "M", "ID": "1", "FASBChargeEndDate": "", "CurrencyCode": "", "FASBChargeEffDate": "", "Dept": "None", "FASBOccDays": "", "SuiteID": "0301    ", "FCSuiteID": "ZZZ None", "EntityID": "495KINGW", "GLClosedStatus": "Closed", "RetroPD": "0", "FASBWeightedAmt": "", "Period": "200901", "IncomeCat": "41000 ", "OpenAmt": "0", "ChargeAmt": "2600", "Invoice": "200901    ", "BldgID": "495KINGW"}, {"TxnDate": "Jan  1 2009 12:00:00:000AM", "TxnDateInt": "20090101", "LeaseID": "CANAGEO1", "FCWorkBook": "ZZZ None", "FASBEndDate": "", "FCLeaseNo": "0", "GLPostedStatus": "Posted", "SourceCode": "CH", "PaidStatus": "Paid", "FASBStartDate": "", "ActualProjected": "Charges", "Frequency": "O", "ID": "2", "FASBChargeEndDate": "", "CurrencyCode": "", "FASBChargeEffDate": "", "Dept": "None", "FASBOccDays": "", "SuiteID": "0301    ", "FCSuiteID": "ZZZ None", "EntityID": "495KINGW", "GLClosedStatus": "Closed", "RetroPD": "0", "FASBWeightedAmt": "", "Period": "200901", "IncomeCat": "GST   ", "OpenAmt": "0", "ChargeAmt": "130", "Invoice": "200901    ", "BldgID": "495KINGW"}
]

Is there any other good way to solve this problem?

bouteillebleu
  • 2,456
  • 23
  • 32
Kalyan
  • 1,880
  • 11
  • 35
  • 62
  • Possible duplicate of http://stackoverflow.com/questions/39339044/how-to-write-large-json-data – Rajesh Yogeshwar Jan 26 '17 at 07:37
  • Does the server return JSON? Then just dump it to a file without decoding. Otherwise, decode it after you retrieved all the data. – MKesper Jan 26 '17 at 07:39
  • the server return data roughly 100000 not all data. i have checked it using top (limit range)* from table. and that's rings me for doing partial load and append new json data with previous json data and getting complete one – Kalyan Jan 26 '17 at 07:42
  • So you first need to partition your request such that following requests don't return overlapping results. – MKesper Jan 26 '17 at 07:45
  • yes that's ok i could try partition it by rowid but how about saving these partitioned loaded data in file in single json array.i am beginner in python so i am confused about the format , is not will be formed [{}][{}]? but i want [{},{}...] . how could i get that – Kalyan Jan 26 '17 at 07:48

0 Answers0