0

Basically I have a function that returns an API response with a huge amount of dictionaries, simplified to their keys, I then have another function, called getPlayerData which sends an api call to the same api to get information about the specific player, instead of all of them, the problem is that alone, getPlayerData is fast, but in this scenario, it is way more than unusable.

Is there a way i can speed up this? getPlayerData is not required, I can just make a request too.

The dictionary search

    residents = []
    for resident in getListData("resident"):
        if getPlayerData(resident)["town"] == town:
            residents.append(resident)
    print(residents)

getPlayerData()

    def getPlayerData(player):
    r = requests.get("http://srv.earthpol.com/api/json/residents.php?name=" + player)
    j = r.json()
    
    player = player.lower()
    global emptyresult
    emptyresult = False
    if str(j) == "{}":
        emptyresult = True
    else:
        result = {"town": j[player]["town"],
                    "town-rank": j[player]["townRank"],
                    "nation-ranks": j[player]["nationRanks"],
                    "lastOnline:": j[player]["lastOnline"],
                    "registered": j[player]["registered"],
                    "town-title": j[player]["title"],
                    "nation-title": j[player]["surname"],
                    "friends": j[player]["friends"],
                    "uuid": j[player]["uuid"],
                    "avatar": "https://crafatar.com/avatars/"+ j[player]["uuid"]}
        return result
BoomBoxBoy
  • 1,770
  • 1
  • 5
  • 23
  • So you want to filter by `town`? There's no way you decrease the number of iterations, without any prior knowledge about the `resident`s. I mean you need to query each single one to check whether it matches the given town. – a_guest Dec 21 '20 at 17:24
  • @a_guest would checking the town only help? cause getplayerdata is really cluttered with things that can slow it down –  Dec 21 '20 at 17:28
  • Reorganize your data structure in a way which makes more sense. Dictionaries are very very fast whey you are searching for a key. If you want to search by town name, make a dictionary with the town name as the key. – zvone Dec 21 '20 at 17:29
  • @zvone basically i am using an api which should have the residents of a town in it, it doesnt, so im trying to circumvent this by getting all the residents and seeing who is in that town, if i did that, wouldn't it essentially not let me know which resident is in what town? Or did you mean something more like playertown = {townname:town} –  Dec 21 '20 at 17:33
  • @kz-n Perhaps there is a way you can directly query for town? Something like `http://srv.earthpol.com/api/json/residents.php?town=...`. Reducing the number of requests would definitely help. But if the only thing you can do is query by name, well, then you got to query every name. – a_guest Dec 21 '20 at 17:33
  • @a_guest yeah, that isn't possible :C –  Dec 21 '20 at 17:34
  • @kz-n So what does `getListData("resident")` do? – a_guest Dec 21 '20 at 17:35
  • 4
    The bottleneck is nothing to do with the dictionaries, it is because you are making a lot of requests one at a time; currently your code has to wait for each request to be served before it makes the next request. If there is really no way to make a single API request for the result you want, try making multiple requests in parallel. – kaya3 Dec 21 '20 at 17:35
  • What you are currently doing is going through the total amount of data to find one thing. What you need to do is to somehow find that one thing without looking at everything. You can do e.g. by using a different api (if that exists), or by maintaining a local copy of data structured so that the key is what you are looking for and the value is the result, or something else... depending on what you can or cannot do with that api. – zvone Dec 21 '20 at 17:36
  • This Q&A might be helpful: https://stackoverflow.com/questions/22190403/how-could-i-use-requests-in-asyncio – kaya3 Dec 21 '20 at 17:37
  • @kaya3 how would i handle that? –  Dec 21 '20 at 17:39
  • @kz-n Just to make sure, if you are in control of `srv.earthpol.com` (or have any influence on the API), the correct way would be to modify that API so you can query for town directly. I mean this data must be stored in a database somewhere, so making an isolated request for every single row is highly inefficient. – a_guest Dec 21 '20 at 17:39
  • @zvone, i've tried caching every minute or so but i have been having the same problem of the speed of the requests –  Dec 21 '20 at 17:39
  • @a_guest okay, i will let the api maintainer know about this –  Dec 21 '20 at 17:40
  • 3
    @kz-n By the way, it appears that `http://srv.earthpol.com/api/json/residents.php` gives you a dictionary of all players, mapping their corresponding attributes. So you can just get all the data once, and then filter by town: `r = requests.get("http://srv.earthpol.com/api/json/residents.php"); j = r.json(); by_town = defaultdict(list); for name, data in j.items(): by_town[data["town"]].append(name)`. – a_guest Dec 21 '20 at 17:43

0 Answers0