Twitter Python Json to CSV

Question

I am new to Python and programing in general.

I wrote this script and it runs with out error but is doesn't print any content in the .csv even though I know there is content to print. I have been stuck for a day or 2 and need some help.

import sys
import json
import urllib
import oauth2 as oauth
import requests
import time
import csv


CONSUMER_KEY = ""
CONSUMER_SECRET = ""
ACCESS_KEY = ""
ACCESS_SECRET = ""

consumer = oauth.Consumer(key=CONSUMER_KEY, secret=CONSUMER_SECRET)
access_token = oauth.Token(key=ACCESS_KEY, secret=ACCESS_SECRET)
client = oauth.Client(consumer, access_token)

html ="https://api.twitter.com/1.1/search/tweets.json?q=@gmail.com"
response, data = client.request(html)

f = open("twitter_gmail.csv", 'a')

handle_tweet =json.loads(data)
def handle_tweet(self, data):
    search_terms = ['@gmail.com']
text = message.get('text')
words = text.split()
matches = []
for term in search_terms:
    match = [word for word in words if term in word]
    matches.append(match)
f.write('%s,%s,%s,%s\n' % (message.get('created_at'), message.get('text'), message.get('user').get('id'),matches))

when I change it "data.get" I still get nothing but no errors either — user2748540, Dec 06 '13 at 02:35

score 1 · Answer 1 · edited May 23 '17 at 12:13

It looks like you are importing the csv module without even using it. There is probably something funky with your f.write statement, but things will be much easier for you if you try to write to your file using csv.writer. The csv.writer can easily take in a list, and spit out a comma separated line of the values in the list. I'd recommend reading its documentation in order to implement it.

You're pretty close already. I'm not exactly sure what you want on each row, but the following might be close to your intended goal.

f = open("twitter_gmail.csv", 'a')
# This next line makes the csv writer.
writer = csv.writer(f)
handle_tweet =json.loads(data)
def handle_tweet(self, data):
    search_terms = ['@gmail.com']
text = message.get('text')
words = text.split()
matches = []
for term in search_terms:
    match = [word for word in words if term in word]
    matches.append(match)
# The next line is how you write with a csv writer. 
writer.writerows(matches)

If I'm reading your code correctly, matches is a list of lists. I've used csv.writer's writerows method which will put each list on its own line (notice how it's plural, as opposed to writerow, which expects a single list and writes it to a single line).

Hugh Bothwell · Accepted Answer · 2013-12-06T16:42:41.210

I don't have Twitter auth keys, so can't test, but should work based on Twitter's API documentation.

If you are doing serious work (as opposed to figuring out how it is done), you should probably use a tested module like twitter or python-twitter.

import csv
import json
import oauth2 as oauth
import urllib

# I don't if any of these are actually needed?
import sys
import requests
import time

CONSUMER_KEY    = ''
CONSUMER_SECRET = ''
ACCESS_KEY      = ''
ACCESS_SECRET   = ''

class TwitterSearch:
    def __init__(self,
        ckey    = CONSUMER_KEY,
        csecret = CONSUMER_SECRET,
        akey    = ACCESS_KEY,
        asecret = ACCESS_SECRET,
        query   = 'https://api.twitter.com/1.1/search/tweets.{mode}?{query}'
    ):
        consumer     = oauth.Consumer(key=ckey, secret=csecret)
        access_token = oauth.Token(key=akey, secret=asecret)
        self.client  = oauth.Client(consumer, access_token)
        self.query   = query

    def search(self, q, mode='json', **queryargs):
        queryargs['q'] = q
        query = urllib.urlencode(queryargs)
        return self.client.request(self.query.format(query=query, mode=mode))

def write_csv(fname, rows, header=None, append=False, **kwargs):
    filemode = 'ab' if append else 'wb'
    with open(fname, filemode) as outf:
        out_csv = csv.writer(outf, **kwargs)
        if header:
            out_csv.writerow(header)
        out_csv.writerows(rows)

def main():
    ts = TwitterSearch()
    response, data = ts.search('@gmail.com', result_type='recent')
    js = json.loads(data)

    # This _should_ work, based on sample data from
    # https://dev.twitter.com/docs/api/1.1/get/search/tweets
    messages = ([msg['created_at'], msg['txt'], msg['user']['id']] for msg in js.get('statuses', []))

    write_csv('twitter_gmail.csv', messages, append=True)

if __name__ == "__main__":
    main()

The json looks like this sample "{"statuses":[{"metadata":{"result_type":"recent","iso_language_code":"en"},"created_at":"Fri Dec 06 06:00:58 +0000 2013","id":408838455058505728,"id_str":"408838455058505728"," — user2748540, Dec 06 '13 at 06:02

Twitter Python Json to CSV

2 Answers2

Linked