Python conversion of JSON object to pandas from an API streaming data

Question

I have a function in Python, defined in an API from my broker, "getCalendar" which recieves a list of news announcements and expected impact on market. how can I transform this list which arrives as a JSON object to a pandas dataframe, so I can analyze it?

P.S.: the API is a connection to a server, which is first established and only then can data be streamed from there, so using url address and converting that to pandas dataframe is not possible. Thanks in advance for your help.

This is currently way too broad: have you been able to connect to the source and retrieve the data yet? — Jon Clements, Sep 09 '13 at 11:49
Yes, I have managed to connect, and I can get the data in the JSON format but I do not know how to convert it because if I try assigning it to calendar <- getCalendar for example, it simply prints the JSON data without assigning it. — Cristi Popescu, Sep 09 '13 at 11:59
Well then, that'd be good to add to your question... it shows how far you've got, show some example data, and then what you've tried with Pandas and what error you're getting... then we've almost got a proper question... — Jon Clements, Sep 09 '13 at 12:41
Can you give an example of the data (json), how are you intending to convert it (have you tried read_json?). Also `calendar <- getCalender` doesn't look like valid python to me (is that R?)... — Andy Hayden, Sep 09 '13 at 13:00

score 0 · Answer 1 · edited May 23 '17 at 12:20

Seems your getCalendar is writing stuff to stdout and you first need to capture that into a string variable. Use this post solution to capture: Can I redirect the stdout in python into some sort of string buffer? (e.g. write a wrapper like getCalendarStdout() used below)

Once you got the json output into a variable (say calendar), try this:

calendar = getCalendarStdout()

import json
import pandas as pd

data=json.loads(calendar)
dafr = pd.DataFrame(data,columns=['col1','col2'])

Here we are trying to get only certain fields from the json output into the dafr DataFrame. If you can paste the calendar json data (part of it) people can help you get desired dataframe.

Python conversion of JSON object to pandas from an API streaming data

1 Answers1