0

I wrote a script that goes through API and collect data, it was slow and i've tried to do multi-threading and ended up with this:

import requests
from collections import defaultdict
import csv
import time
from queue import Queue
from threading import Thread

def dmg(ID):
    server='countryball'
    my_dict = defaultdict(lambda: {'dmg':0})
    time.sleep(0.3)
    r = requests.get('https://'+server+'.e-sim.org/apiFights.html?battleId='+str(ID)).json()
    for hit in r:
        try:my_dict[hit['citizenId']]['dmg'] += hit['damage']
        except:my_dict[hit['citizenId']]['dmg'] += hit['damage']
    my_dict={k:dict(v) for k,v in my_dict.items()}
    if my_dict:
        with open('dmg.csv','a',newline='') as f:
            writer = csv.writer(f)
            writer.writerow([my_dict])

battle_ids=list(range(10317,10900))
NUM_WORKERS = 6
task_queue = Queue()
def worker():
    while True:
        ID = task_queue.get()
        time.sleep(0.5)
        dmg(ID)
        task_queue.task_done()
threads = [Thread(target=worker) for _ in range(NUM_WORKERS)]
[task_queue.put(item) for item in battle_ids]
[thread.start() for thread in threads]
task_queue.join()

a small part of output is such as below:

"{2174: {'dmg': 377156}, 469: {'dmg': 785358}, 535: {'dmg': 63542}}"
{226: {'dmg': 688205}}
{1330: {'dmg': 59370}}
"{37068: {'dmg': 184975}, 1164: {'dmg': 239162}, 1244: {'dmg': 734525}}"
"{1852: {'dmg': 306785}, 12218: {'dmg': 112228}, 9481: {'dmg': 22745}}"
{933: {'dmg': 21747}}
{2702: {'dmg': 375177}}
{1796: {'dmg': 21845}}
"{36683: {'dmg': 241816}, 41129: {'dmg': 200211}, 303: {'dmg': 1601865}}"
"{1546: {'dmg': 434303}, 1158: {'dmg': 1138277}, 9834: {'dmg': 37585}, 1750: {'dmg': 347775}}"
{362: {'dmg': 880661}}
"{24132: {'dmg': 459637}, 7566: {'dmg': 100760}}"
{1049: {'dmg': 185024}}

since there are multiple keys with different values and i have to sum up value of those keys, this output is not valuable to me, hence i need help with 2 possible solutions,

  1. how can i store output of dmg, which is my_dict in a list and sum up data and benefit from multi-threading(currently not possible with multi-threading .)
  2. reading csv file,storing data in a list like this : data = [[key(ex:2174):value(ex:{'dmg': 377156})]]

-for the second part, i've tried multiple approaches but i cant read the dictionary part({'dmg': 377156}) as a dictionary, its always str.

If anyone is kind enough, help me with this, im new to coding...

Abolfazl
  • 23
  • 4

1 Answers1

0

fixed my problem,

buffer=[]
def dmg(ID):
    my_dict = defaultdict(lambda: {'dmg':0})
    time.sleep(0.3)
    r = requests.get('https://'+server+'.e-sim.org/apiFights.html?battleId='+str(ID)).json()
    for hit in r:
        try:my_dict[hit['citizenId']]['dmg'] += hit['damage']
        except:my_dict[hit['citizenId']]['dmg'] += hit['damage']
    my_dict={k:dict(v) for k,v in my_dict.items()}
    if my_dict:
        buffer.append(my_dict)

i've tried using other lists as a storage but it didnt work, for some reason, using buffer is allowed.

if anyone knows the reason, let me know please...

my_dict = defaultdict(lambda: {'dmg':0})
for item in buffer:
    a = item
    for k,v in a.items():
        my_dict[k]['dmg'] += v['dmg']
my_dict={k:dict(v) for k,v in my_dict.items()}
Abolfazl
  • 23
  • 4
  • What is the try/except for in the first code snippet? – AMC Feb 17 '20 at 04:59
  • @AMC its for anti request spamming, sometimes it gives error when fetching data... – Abolfazl Feb 17 '20 at 17:59
  • Right, but won't the except run practically instantly after the try? They're doing the same thing, too. There has to be a simpler way to check if you're being rate limited. Using bare except statemements is bad practice, by the way (see, for example: https://stackoverflow.com/questions/14797375/should-i-always-specify-an-exception-type-in-except-statements). – AMC Feb 17 '20 at 20:58