Build a dict and populate it in a for loop

Question

I'm trying to build a dictionary that should contain data about a coin, date and its close price on the given date.

The data I'm looping through, which I am using as the base out of which I want to build my dict, look like this (it's a Python list):

[('2018-08-31', 'BSV', 0.0), ('2018-08-31', 'EOS', 6.4050002), ('2018-08-31', 'ETC', 12.7299995), ('2018-08-31', 'BNB', 10.96), ('2018-08-31', 'LTC', 62.1300011), ('2018-08-31', 'XMR', 116.1399994), ('2018-08-31', 'BTT', 0.0), ('2018-08-31', 'BCH', 542.4199829), ('2018-08-31', 'ADA', 0.1039), ('2018-08-31', 'DASH', 194.25), ('2018-08-31', 'BTC', 7014.6000977), ('2018-08-31', 'ZEC', 149.6300049), ('2018-08-31', 'XLM', 0.2216), ('2018-08-31', 'XRP', 0.335), ('2018-08-31', 'QTUM', 4.7399998), ('2018-08-31', 'NEO', 20.1100006), ('2018-08-31', 'ETH', 281.6300049), ('2018-08-31', 'TRX', 0.02532), ('2018-09-01', 'BSV', 0.0), ('2018-09-01', 'EOS', 6.6620002), ('2018-09-01', 'ETC', 13.1899996), ('2018-09-01', 'BNB', 11.4499998), ('2018-09-01', 'LTC', 66.5), ('2018-09-01', 'XMR', 121.0299988), ('2018-09-01', 'BTT', 0.0), ('2018-09-01', 'BCH', 618.8699951), ('2018-09-01', 'ADA', 0.1098)]

It's a list of tuples and each tuple has this value in it: (date, coin, close_price).

My expected JSON at the end should look like this:

{
    "coins": [{
        "BTC": [{
            "2018-08-31": 7014.6000977
        }, {
            "2018-09-01": 7197.3999023
        }],
        "EOS": [{
            "2018-08-31": 6.4050002
        }, {
            "2018-09-01": 6.6620002
        }]
    }]
}

My current code to build such a JSON looks like this:

prices_json = {"coins": []}
for date, _coin, close_price in prices: #prices is the above-mentioned list of tuples
    if date:
        prices_json["coins"].append(
            {
                f"{_coin}": []
            }
        )
        prices_json["coins"][0][f"{_coin}"].append(
            {
                f"{date}": f"{close_price}"
            }
        )

The reason I want to build a JSON like this from the provided list of tuples is, that I want to access the close_price data in this way:

my_json_with_prices[f"{coin}"][f"{date}"]

This should result in the close price number, for example 6.4050002

The values of f"{coin}" and "f{date}" will be provided in a for loop later on in the script.

If you have any other idea how to access the close_price values from the list of tuples based on the date and the coin, I am open to all of the suggestions.

You seem to be asking about how to build a dictionary, not about producing JSON. I have edited accordingly, but left in the JSON terminology in a couple of places. Please review. — tripleee, Apr 25 '21 at 07:36
`f"{_coin}"` is a really roundabout way to say `str(_coin)`, or just `_coin` if you know it's already a string. — tripleee, Apr 25 '21 at 07:38
Why would you store the endprices as list of dicts with one date->value each? If you only have one closing value per date per coin a dict of date->value should suffice. If you have more then 1 closing value, you're a time traveler... — Patrick Artner, Apr 25 '21 at 07:38
The root dict with just `"coins"` as its sole key seems completely redundant here. — tripleee, Apr 25 '21 at 07:39
Why do you want the innermost information to be a list of dictionaries with a single key in each? Assuming the dates will always be unique (because how would you otherwise interpret the data? A coin can't possibly have two correct values at a given time) it makes sense for the per-coin information to be just a dictionary. (I guess @Patrick was trying to say something similar.) — tripleee, Apr 25 '21 at 07:43
You are both right - for every single coin and every single day there is only 1 price. @tripleee What do you mean exactly with the 'per-coin information to be just a dictionary'? — neisor, Apr 25 '21 at 07:45
As mentioned in my question, if someone has a better approach/structure to achieve my described goal, please share it - I'm open to all of the suggestions. Thank you — neisor, Apr 25 '21 at 07:51

tripleee · Answer 1 · 2021-04-25T08:09:34.007

The structure you are building has several levels of nesting. Let's think about them inside out.

Each coin's data is a mapping from date to value. It makes sense for this to be a single dict rather than a list of singleton dictionaries. To add a new mapping, just set coindict[date] = value

The structure above this is a dictionary of coin names, which are strings, with a coindict as described above as the value of each.

Python's defaultdict provides a good way to avoid having to go through

if key not in dic:
    dic[key] = dict()
# now safe to add something to dic[key]

So with that, let's traverse your input structure.

from collections import defaultdict

# see https://stackoverflow.com/questions/5029934/defaultdict-of-defaultdict
close_price = defaultdict(lambda: defaultdict(dict))

for date, coin, price in prices:
    close_price[coin][date] = price

Demo: https://ideone.com/Zulvwe

Now, close_price[coin][date] produces the close value of coin on date, assuming of course that date exists as a key in close_price[coin]. To loop over the available dates for a particular coin, simply for date in close_price[coin]:

If you want to put close_price inside another dictionary, that's easy, too, but seems redundant:

unnecessary = {"coins": [close_price] }

If you want to save this as JSON, try

import json
with open('coins.json', 'w') as jfile:
    json.dump(unnecessary, jfile)

score 0 · Answer 2 · answered Apr 25 '21 at 08:04

use pandas -

import pandas as pd
df  = pd.DataFrame(l1,columns= ['date','coin', 'value'])
temp_list = df.groupby('coin',as_index=False).agg(list).to_dict(orient='records')
result = {
    item['coin']: (dict(zip(item['date'], item['value'])))
    for item in temp_list
}
result = {'coins': result}
print(result)

output -

{'coins': {'ADA': {'2018-08-31': 0.1039, '2018-09-01': 0.1098},
  'BCH': {'2018-08-31': 542.4199829, '2018-09-01': 618.8699951},
  'BNB': {'2018-08-31': 10.96, '2018-09-01': 11.4499998},
  'BSV': {'2018-08-31': 0.0, '2018-09-01': 0.0},
  'BTC': {'2018-08-31': 7014.6000977},
  'BTT': {'2018-08-31': 0.0, '2018-09-01': 0.0},
  'DASH': {'2018-08-31': 194.25},
  'EOS': {'2018-08-31': 6.4050002, '2018-09-01': 6.6620002},
  'ETC': {'2018-08-31': 12.7299995, '2018-09-01': 13.1899996},
  'ETH': {'2018-08-31': 281.6300049},
  'LTC': {'2018-08-31': 62.1300011, '2018-09-01': 66.5},
  'NEO': {'2018-08-31': 20.1100006},
  'QTUM': {'2018-08-31': 4.7399998},
  'TRX': {'2018-08-31': 0.02532},
  'XLM': {'2018-08-31': 0.2216},
  'XMR': {'2018-08-31': 116.1399994, '2018-09-01': 121.0299988},
  'XRP': {'2018-08-31': 0.335},
  'ZEC': {'2018-08-31': 149.6300049}}}

Pandas is severe overkill for this simple task. The code is much too dense for a beginner question. — tripleee, Apr 25 '21 at 08:06
Yup. It’s just another way to solve the issue. Plus, you can do more manipulation very easily, if required :) — Nk03, Apr 25 '21 at 08:09

score 0 · Accepted Answer · answered Apr 25 '21 at 08:21

You are creating unnecessary nesting inside the dictionary. You may represent the data as follows,

    prices_json = {
        currency1 : {
            date1 : price1,
            date2 : price2
        }
        
        currency2 : {
            date1 : price1,
            date2 : price2
        }
    }

Given the list of the tuples,

prices = [('2018-08-31', 'BSV', 0.0), ('2018-08-31', 'EOS', 6.4050002), ('2018-08-31', 'ETC', 12.7299995), ('2018-08-31', 'BNB', 10.96), ('2018-08-31', 'LTC', 62.1300011), ('2018-08-31', 'XMR', 116.1399994), ('2018-08-31', 'BTT', 0.0), ('2018-08-31', 'BCH', 542.4199829), ('2018-08-31', 'ADA', 0.1039), ('2018-08-31', 'DASH', 194.25), ('2018-08-31', 'BTC', 7014.6000977), ('2018-08-31', 'ZEC', 149.6300049), ('2018-08-31', 'XLM', 0.2216), ('2018-08-31', 'XRP', 0.335), ('2018-08-31', 'QTUM', 4.7399998), ('2018-08-31', 'NEO', 20.1100006), ('2018-08-31', 'ETH', 281.6300049), ('2018-08-31', 'TRX', 0.02532), ('2018-09-01', 'BSV', 0.0), ('2018-09-01', 'EOS', 6.6620002), ('2018-09-01', 'ETC', 13.1899996), ('2018-09-01', 'BNB', 11.4499998), ('2018-09-01', 'LTC', 66.5), ('2018-09-01', 'XMR', 121.0299988), ('2018-09-01', 'BTT', 0.0), ('2018-09-01', 'BCH', 618.8699951), ('2018-09-01', 'ADA', 0.1098)

The code to load it into the prices_json may look something like this,

prices_json = {}

for date, currency, price in prices:
  
    prices_json[currency] = prices_json.get(currency, {})
    prices_json[currency][date] = price

After running the code you would be able to fetch the price of a specific currency for a given date in this way,

prices_json[currency][date]

score 0 · Answer 4 · answered Apr 25 '21 at 08:23

Your question says:

The reason I want to build a JSON like this from the provided list of tuples is, that I want to access the close_price data in this way:
my_json_with_prices[f"{coin}"][f"{date}"]

You don't need a JSON structure for that. JSON is an interchange format. Python's native dicts will do exactly what you want. Even this:

 my_json_with_prices[f"{coin}"][f"{date}"]

is overcomplicated. The Python equivalent is

 python_equivalent[coin][date]

You can build a structure that does this as follows:

# Start with the data
data = [('2018-08-31', 'BSV', 0.0), ('2018-08-31', 'EOS', 6.4050002), ('2018-08-31', 'ETC', 12.7299995), ('2018-08-31', 'BNB', 10.96), ('2018-08-31', 'LTC', 62.1300011), ('2018-08-31', 'XMR', 116.1399994), ('2018-08-31', 'BTT', 0.0), ('2018-08-31', 'BCH', 542.4199829), ('2018-08-31', 'ADA', 0.1039), ('2018-08-31', 'DASH', 194.25), ('2018-08-31', 'BTC', 7014.6000977), ('2018-08-31', 'ZEC', 149.6300049), ('2018-08-31', 'XLM', 0.2216), ('2018-08-31', 'XRP', 0.335), ('2018-08-31', 'QTUM', 4.7399998), ('2018-08-31', 'NEO', 20.1100006), ('2018-08-31', 'ETH', 281.6300049), ('2018-08-31', 'TRX', 0.02532), ('2018-09-01', 'BSV', 0.0), ('2018-09-01', 'EOS', 6.6620002), ('2018-09-01', 'ETC', 13.1899996), ('2018-09-01', 'BNB', 11.4499998), ('2018-09-01', 'LTC', 66.5), ('2018-09-01', 'XMR', 121.0299988), ('2018-09-01', 'BTT', 0.0), ('2018-09-01', 'BCH', 618.8699951), ('2018-09-01', 'ADA', 0.1098

# Get the coins
coins = set(d[1] for d in data)

# Build an empty lookup table for each coin
lookup = {}
for c in coins:
    lookup[c] = {}

# Now populate it
for date, c, price in data:
    lookup[c][date] = price

(That can be done in fewer steps.) This will allow you do do

required_value = lookup[coin][date]

If you really want a JSON structure for this, though I can't imagine why unless you are passing it to some other program, then

my_json_with_prices = json.dumps(lookup)

Thank you @BoarGules - I didn't end up using your code but thank you again for your time and effort :) — neisor, Apr 25 '21 at 13:23

Build a dict and populate it in a for loop

4 Answers4