0

I am recently working on twitter analysis and want to save the results from twitter search locally into .json files.

I am doing both historical search using "searchtweets" and streaming using "tweepy". The result I get are objects which are slightly different but I think both of the results are compatible with .json.

Here are my code(sample, I have a logged in to API already):

new_tweets = API.user_timeline(screen_name='huliangw20', count=20, 
tweet_mode='extended')

with open('test_tweets_v2.json', 'w',encoding='utf-8') as outfile:
    outfile.write(new_tweets)

It does not work. In general, each query gives me a Resultset of many Tweet objects. And my goal is to save those objects into .json files using python.

Would really appreciate if you could give me some advice on this.

Liang Hu
  • 3
  • 1

1 Answers1

2

You can use the json standard library from python.

Reading a string as json:

>>> import json
>>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
'["foo", {"bar": ["baz", null, 1.0, 2]}]'

Transforming json as string:

>>> import json
>>print(json.dumps({"c": 0, "b": 0, "a": 0}, sort_keys=True))
{"a": 0, "b": 0, "c": 0}

With this function you can save on disk:

def writeToJSONFile(path, fileName, data):
    try:
        filePathNameWExt = path + fileName
        with open(filePathNameWExt, 'w') as fp:
            json.dump(data, fp)
    except Exception as e:
        if(not quiet):
            print "writeToJSONFile exception"
            print e

And finally generating a json on "hand":

>>> jsonToSave = {}
>>> jsonToSave['id'] = id
>>> metaJson = {}
>>> metaJson['version'] = VERSION
>>> jsonToSave['metadata'] = metaJson

More info: https://docs.python.org/3/library/json.html More info: https://docs.python.org/2/library/json.html

Lucas Araújo
  • 429
  • 5
  • 17
  • Sorry I am quite confused. So the result I have is the variable "new_tweets", it is a set of "Tweet object" (each one of them contains all the information about a tweet) . How can I save them locally into one json file? – Liang Hu Oct 31 '18 at 14:56
  • You can construct the json optimal for your application, per example, you can create a jsonToSave = {} and write just some parts of your "Tweet object" like: jsonToSave["Text"] = "bla bla", jsonToSave["Date"] = xx/yy/zzz so on. – Lucas Araújo Oct 31 '18 at 15:04
  • Thanks Lucas. I already know how to save a particular field into a file. The thing is I want to save the whole object for future use and there are like 100+ attributes. Just wondering if there is any quite way to save the whole thing? – Liang Hu Oct 31 '18 at 15:41
  • Ohh now I got what u need, this format of json that I'm saying to you is a dict, you can cast objects to dict, look this answer https://stackoverflow.com/questions/45834577/turn-python-object-into-json-output That's help you? – Lucas Araújo Oct 31 '18 at 15:43