1

Sorry, I am new in coding in Python, I would need to save a json file generated in a for loop as csv for each iteration of the loop.

I wrote a code that works fine to generate the first csv file but then it is overwritten and I did not find a solution yet. Can anyone help me? many thanks

from twarc.client2 import Twarc2
import itertools
import pandas as pd
import csv
import json
import numpy as np

# Your bearer token here
t = Twarc2(bearer_token="AAAAAAAAAAAAAAAAAAAAA....WTW")

# Get a bunch of user handles you want to check:

list_of_names = np.loadtxt("usernames.txt",dtype="str")
# Get the `data` part of every request only, as one list
def get_data(results):
    return list(itertools.chain(*[result['data'] for result in results]))

user_objects = get_data(t.user_lookup(users=list_of_names, usernames=True))

for user in user_objects:

    following = get_data(t.following(user['id']))

    # Do something with the lists
   
    print(f"User: {user['username']} Follows {len(following)} -2")

    json_string = json.dumps(following)
    df = pd.read_json(json_string)
    df.to_csv('output_file.csv')
Camilla Ancona
  • 13
  • 3

2 Answers2

0

You need to add a sequence number or some other unique identifier to the filename. The clearest example would be to keep track of a counter, or use a GUID. Below I've used a counter that is initialized before your loop, and is incremented in each iteration. This will produce a list of files like output_file_1.csv, output_file_2.csv, output_file_3.csv and so on.

counter = 0

for user in user_objects:
    following = get_data(t.following(user['id']))
    # Do something with the lists
    print(f"User: {user['username']} Follows {len(following)} -2")
    json_string = json.dumps(following)
    df = pd.read_json(json_string)
    df.to_csv('output_file_' + str(counter) + '.csv')
    counter += 1

We convert the integer to a string, and paste it inbetween the name of your file and its extension.

Zimano
  • 1,870
  • 2
  • 23
  • 41
-1
from twarc.client2 import Twarc2
import itertools
import pandas as pd
import csv
import json
import numpy as np

# Your bearer token here
t = Twarc2(bearer_token="AAAAAAAAAAAAAAAAAAAAA....WTW")

# Get a bunch of user handles you want to check:

list_of_names = np.loadtxt("usernames.txt",dtype="str")
# Get the `data` part of every request only, as one list
def get_data(results):
    return list(itertools.chain(*[result['data'] for result in results]))

user_objects = get_data(t.user_lookup(users=list_of_names, usernames=True))

for idx, user in enumerate(user_objects):

    following = get_data(t.following(user['id']))

    # Do something with the lists
   
    print(f"User: {user['username']} Follows {len(following)} -2")

    json_string = json.dumps(following)
    df = pd.read_json(json_string)
    df.to_csv(f'output_file{str(idx)}.csv')
Mukul Kirti Verma
  • 350
  • 3
  • 9