For Loop 60 items 10 per 10

Question

I'm working with an api that gives me 61 items that I include in a discord embed in a for loop.

As all of this is planned to be included into a discord bot using pagination from DiscordUtils, I need to make it so it male an embed for each 10 entry to avoid a too long message / 2000 character message.

Currently what I use to do my loop is here: https://api.nepmia.fr/spc/ (I recomend the usage of a parsing extention for your browser or it will be a bit hard to read it)

But what I want to create is something that will look like that : https://api.nepmia.fr/spc/formated/

So I can iterate each range in a different embed and then use pagination.

I use TinyDB to generate the JSON files I shown before with this script:

import urllib.request, json
from shutil import copyfile
from termcolor import colored
from tinydb import TinyDB, Query

db = TinyDB("/home/nepmia/Myazu/db/db.json")

def api_get():
    print(colored("[Myazu]","cyan"), colored("Fetching WynncraftAPI...", "white"))
    try:
        with urllib.request.urlopen("https://api.wynncraft.com/public_api.php?action=guildStats&command=Spectral%20Cabbage") as u1:
            api_1 = json.loads(u1.read().decode())
            count = 0
            if members := api_1.get("members"):
                print(colored("[Myazu]","cyan"),
                      colored("Got expecteded answer, starting saving process.", "white"))
                for member in members:
                    nick = member.get("name")
                    ur2 = f"https://api.wynncraft.com/v2/player/{nick}/stats"
                    u2 = urllib.request.urlopen(ur2)
                    api_2 = json.loads(u2.read().decode())
                    data = api_2.get("data")
                    for item in data:
                            meta = item.get("meta")
                            playtime = meta.get("playtime")
                            print(colored("[Myazu]","cyan"),
                                  colored("Saving playtime for player", "white"),
                                  colored(f"{nick}...","green"))
                            db.insert({"username": nick, "playtime": playtime})
                            count += 1
            else: 
                print(colored("[Myazu]","cyan"), 
                      colored("Unexpected answer from WynncraftAPI [ERROR 1]", "white"))
    except:
        print(colored("[Myazu]","cyan"), 
              colored("Unhandled error in saving process [ERROR 2]", "white"))
    finally:
        print(colored("[Myazu]","cyan"),
              colored(f"Finished saving data for", "white"),
              colored(f"{count}", "green"), 
              colored("players.", "white"))

but this will only create a range like this : https://api.nepmia.fr/spc/

what I would like is something like this : https://api.nepmia.fr/spc/formated/

Thanks for your help!

PS: Sorry for your eyes I'm still new to Python so I know I don't do stuff really properly :s

Answer 1

To follow up from the comments, you shouldn't store items in your database in a format that is specific to how you want to return results from the database to a different API, as it will make it more difficult to query in other contexts, among other reasons.

If you want to paginate items from a database it's better to do that when you query it.

According to the docs, you can iterate over all documents in a TinyDB database just by iterating directly over the DB like:

for doc in db:
    ...

For any iterable you can use the enumerate function to associate an index to each item like:

for idx, doc in enumerate(db):
    ...

If you want the indices to start with 1 as in your examples you would just use idx + 1.

Finally, to paginate the results, you need some function that can return items from an iterable in fixed-sized batches, such as one of the many solutions on this question or elsewhere. E.g. given a function chunked(iter, size) you could do:

pages = enumerate(chunked(enumerate(db), 10))

Then list(pages) gives a list of lists of tuples like [(page_num, [(player_num, player), ...].

The only difference between a list of lists and what you want is you seem to want a dictionary structure like

{'range1': {'1': {...}, '2': {...}, ...}, 'range2': {'11': {...}, ...}}

This is no different from a list of lists; the only difference is you're using dictionary keys to give numerical indices to each item in a collection, rather than the indices being implict in the list structure. There's many ways you can go from a list of lists to this. The easiest I think is using a (nested) dict comprehension:

{f'range{page_num + 1}': {str(player_num + 1): player for player_num, player in page} 
 for page_num, page in pages}

This will give output in exactly the format you want.

Answer 2

Thanks @Iguananaut for your precious help.

In the end I made something similar from your solution using a generator.

def chunker(seq, size):
    for i in range(0, len(seq), size):
        yield seq[i:i+size]

def embed_creator(embeds):
    pages = []
    current_page = None
    for i, chunk in enumerate(chunker(embeds, 10)):
        current_page = discord.Embed(
            title=f'**SPC** Last week online time',
            color=3903947)
        for elt in chunk:
            current_page.add_field(
                name=elt.get("username"), 
                value=elt.get("play_output"), 
                inline=False)
        current_page.set_footer(
            icon_url="https://cdn.discordapp.com/icons/513160124219523086/a_3dc65aae06b2cf7bddcb3c33d7a5ecef.gif?size=128", 
            text=f"{i + 1} / {ceil(len(embeds) / 10)}"
            )
        pages.append(current_page)
        current_page = None
    return pages

Using embed_creator I generate a list named pages that I can simply use with DiscordUtils paginator.