How can I load my data in one line using Python and Jupyter

Question

How can I load or create a loop to load my data composed by the same name but with different numbers {a}dhs{b}... eg., (1dhs2 or 3dhs4 ) using NumPy or pygeostat and then extract the value of the variable to use it (i have 10.000 files). Thanks :)

df1_1 = gs.DataFile('1dhs1.out')
df1_2 = gs.DataFile('1dhs2.out')
...
df3_2 = gs.DataFile('3dhs2.out')

value1_1 = df1_1['value']
value1_2 = df1_2['value']
value1_3 = df1_3['value']
value1_4 = df1_4['value']
...
value4_3 = df1_3['value']
value4_4 = df1_4['value']

Does this answer your question? [How do I create variable variables?](https://stackoverflow.com/questions/1373164/how-do-i-create-variable-variables) — G. Anderson, Jan 07 '21 at 17:24

score 0 · Accepted Answer · answered Jan 07 '21 at 17:38

0

Something like this?

import os
import re

# Regular expression for the expected file name: <first number>dhs<second number>.out 
FILENAME_REGEX = re.compile(r'(\d+)dhs(\d+).out')

folder_path = 'some path'  # path of the folder containing the data files
values = {}  # dictionary to collect extracted values

# Loop over all the files in the folder_path
for f in next(os.walk(folder_path))[2]:
    # if the name of the file does not match the template, go to the next one
    m = FILENAME_REGEX.match(f)
    if m is None:
        continue
    # extract relevant numbers from the file name
    first_int, second_int = list(map(int, m.groups()))

    # store the values in a two-level dictionary, using the extracted numbers as keys
    values.setdefault(first_int, {})[second_int] = gs.DataFile(os.path.join(folder_path, f))['value']

# Print values extracted from 3dhs2.out
print(values[3][2])

answered Jan 07 '21 at 17:38

PieCot

3,564
1
12
20

Exactly what I was looking for. The idea of how to loop over the folder with all the files. Thank you very much – Karim Mokdad Jan 07 '21 at 17:58
One last question, please , how can i organize all the imported data in one big array i simply tried this for i from 1 to 4:for j in range (1,4): for i in range (1,4): X_train = np.array([values[i][j]]) but it returns X_train.shape --> (1, 200) which is wrong because it should be (16,200) – Karim Mokdad Jan 07 '21 at 18:18
Maybe ` X_train = np.row_stack([values[k][l] for k in range(1, 5) for l in range(1, 5)])`? – PieCot Jan 07 '21 at 18:56
it worked !! extremely helpful ... Many thanks :) – Karim Mokdad Jan 07 '21 at 19:02

How can I load my data in one line using Python and Jupyter

1 Answers1