0

I wonder what's the most efficient way to convert a very large dictionary to matrix, for example:

{'0': [1, 2, 3....99999999, 100000000], '1': [1, 2, 3....99999999, 100000000], '2': [1, 2, 3....99999999, 100000000], '3': [1, 2, 3....99999999, 100000000]......'999': [1, 2, 3....99999999, 100000000], '1000': [1, 2, 3....99999999, 100000000]}

to this

list([[1, 1, 1, 1.....1, 1, 1, 1, 1], [2, 2, 2, 2....2, 2, 2, 2, 2], ..... [99999999, 99999999, 99999999, 99999999....99999999, 99999999, 99999999, 99999999], [100000000, 100000000, 100000000, 100000000...100000000, 100000000, 100000000, 100000000]])

user1527840
  • 69
  • 6
  • 1
    Not exactly a matrix in the sense that you are looking for, but this almost screams for pandas ;-) You can construct a dataframe from a dictionary with ease there. See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_dict.html – C Hecht Mar 08 '22 at 20:24
  • that is unreasonably large and is unlikely to fit in memory. How big is the actual dictionary you want to convert? Also, is is full or sparse? – anon01 Mar 08 '22 at 20:56
  • related: https://stackoverflow.com/questions/37862139/convert-dictionary-to-sparse-matrix – anon01 Mar 08 '22 at 20:58
  • @anon01 the whole dataset is around 500*1 billion and you are absolutely right about my concern that it wont fit in memory, without sacrificing response time by loading bits of the dataset one at the time... – user1527840 Mar 08 '22 at 22:33

0 Answers0