Call Pandas Dataframe Functions from a string

Question

For context, I am trying to build a script that will output information about the DataFrames I have in memory.

I have a DataFrame with all the DataFrames listed (dflist). I would like to get the Name of the Dataframe out to call a function. df1.shape for example. I am running into an issue with the system seeing df1 as a string.

dflist
|Name|
|df1 |
|df2 |

Thank You.

Could you share the code with which you construct the dataframe of dataframes.. — Mohamed Ali JAMAOUI, Sep 13 '17 at 13:58
Why don't you keep the dataframes in the list, as opposed to their names? — Nils Werner, Sep 13 '17 at 14:22
I didn't know I could do that, can you share? Wouldn't that start to consume more memory? I am looking at some pretty large tables. Processing isn't a problem. — RMichalowski, Sep 13 '17 at 14:24
Just do `dflist = [df1, df2, df3]` and do `[x.shape for x in dflist]` to get all shapes all at once. — Nils Werner, Sep 13 '17 at 14:25

score 3 · Answer 1 · answered Sep 13 '17 at 14:22

3

You can use locals(), which returns a dict of all local variables, i.e.

df1

is the same as

locals()['df1']

so you can do

[locals()[x].shape for x in list_of_names]

answered Sep 13 '17 at 14:22

Nils Werner

34,832
7
76
98

score 1 · Accepted Answer · answered Sep 13 '17 at 14:10

1

I assume you meant you have a list of DataFrame names corresponding to the dataframes in memory. In any case, the eval function is what you are looking for.

import pandas as pd

df1 = pd.DataFrame([0, 1])
df2 = pd.DataFrame([2, 3])
df3 = pd.DataFrame([4, 5])

list_of_names = ['df1', 'df2', 'df3']

[print(eval(x).shape) for x in list_of_names]

answered Sep 13 '17 at 14:10

PejoPhylo

469
2
11

1

Using `eval()` is a bad idea. – Nils Werner Sep 13 '17 at 14:23
@NilsWerner, could you elaborate? – PejoPhylo Sep 13 '17 at 14:26
It is insecure as it lets you execute arbitrary code and doesn't express what you are actually trying to do: `eval(x)` could be anything, while `locals()[x]` is the local variable with the name `x`. – Nils Werner Sep 13 '17 at 14:27
@NilsWerner Agree and deleted mine ~ thank you for clarify – BENY Sep 13 '17 at 14:30
I see, that makes sense. Your answer should be the accepted one. – PejoPhylo Sep 13 '17 at 14:33
This is a terrible idea. – cs95 Sep 13 '17 at 20:11
While I see how eval can be dangerous, I wouldn't call it a 'terrible idea' in this case, no? – PejoPhylo Sep 14 '17 at 06:53

score 0 · Answer 3 · answered Sep 13 '17 at 15:04

Since using eval is a bad idea (Why is using 'eval' a bad practice?), I suggest that you use a dictionary:

import pandas as pd
d = {} 
d['df1'] = pd.DataFrame([0, 1])
d['df2'] = pd.DataFrame([2, 3])
d['df3'] = pd.DataFrame([4, 5])

Then, you will be able to iterate through the dataframes to compute the shape or apply any function as follows:

In [7]: d.keys() 
Out[7]: ['df1', 'df3', 'df2']

In [8]: [d[x].shape for x in d.keys()]  
Out[8]: [(2, 1), (2, 1), (2, 1)]

The extra advantage is that you can easily access any dataframe d['df1'] with no extra memory or speed loss:

In [10]: timeit d['df1']
10000000 loops, best of 3: 29.2 ns per loop

Call Pandas Dataframe Functions from a string

3 Answers3