50

I have a list, with each entry being a company name

companies = ['AA', 'AAPL', 'BA', ....., 'YHOO']

I want to create a new dataframe for each entry in the list.

Something like

(pseudocode)

for c in companies:
     c = pd.DataFrame()

I have searched for a way to do this but can't find it. Any ideas?

Alexander
  • 105,104
  • 32
  • 201
  • 196

6 Answers6

156

Just to underline my comment to @maxymoo's answer, it's almost invariably a bad idea ("code smell") to add names dynamically to a Python namespace. There are a number of reasons, the most salient being:

  1. Created names might easily conflict with variables already used by your logic.

  2. Since the names are dynamically created, you typically also end up using dynamic techniques to retrieve the data.

This is why dicts were included in the language. The correct way to proceed is:

d = {}
for name in companies:
    d[name] = pd.DataFrame()

Nowadays you can write a single dict comprehension expression to do the same thing, but some people find it less readable:

d = {name: pd.DataFrame() for name in companies}

Once d is created the DataFrame for company x can be retrieved as d[x], so you can look up a specific company quite easily. To operate on all companies you would typically use a loop like:

for name, df in d.items():
    # operate on DataFrame 'df' for company 'name'

In Python 2 you were better writing

for name, df in d.iteritems():

because this avoids instantiating the list of (name, df) tuples that .items() creates in the older version. That's now largely of historical interest, though there will of course be Python 2 applications still extant and requiring (hopefully occasional) maintenance.

holdenweb
  • 33,305
  • 7
  • 57
  • 77
  • 6
    Good point, I hadn't thought of that, but you're absolutely right. – maxymoo Jun 05 '15 at 00:03
  • 11
    This answer taught me a lot. – Moondra Feb 09 '17 at 00:38
  • 4
    I don't understand why the other answer was accepted while this one is clearly better. – Bowen Liu Mar 01 '19 at 21:39
  • 5
    The original questioner has a reputation score of 67, so probably has the answer they wanted (perhaps it went into production somewhere!) and doesn't use Stackoverflow any more. It's possibly unfortunate that the accepted answer uses `exec`, but in the larger scheme of things it's a small concern - though thanks for saying this one is better. Stackoverflow is not a competition for me, but rather a way of providing information for which there is a visible need. – holdenweb Mar 01 '19 at 22:15
23

You can do this (although obviously use exec with extreme caution if this is going to be public-facing code)

for c in companies:
     exec('{} = pd.DataFrame()'.format(c))
maxymoo
  • 35,286
  • 11
  • 92
  • 119
  • In the ipython notebook I get File "", line 1 S.1 = pd.DataFrame() ^ SyntaxError: invalid syntax – Luis Ibáñez Herrera Jun 04 '15 at 05:05
  • It does work if I dont use a loop and just execute the exec statatement with a randmo c value, like format('test') – Luis Ibáñez Herrera Jun 04 '15 at 05:11
  • 1
    The error message is saying that "S.1" is not a valid variable name since a variable can't contain punctuation. You could try and fix this by changing the code to `format(c.replace('.',''))`. – maxymoo Jun 04 '15 at 05:14
  • Yes, I have some company names with '.' in them. Now it works!, thanks :) – Luis Ibáñez Herrera Jun 04 '15 at 05:16
  • 6
    Dynamically creating names in a Python namespace is almost invariably a bad idea. It would be much more sensible to use a dict `d` and write `d[c] = pd.DataFrame()`. Read [this answer](http://stackoverflow.com/questions/5036700/how-can-you-dynamically-create-variables-in-python-via-a-while-loop), for example, to start to understand why it's a bad idea. – holdenweb Jun 04 '15 at 08:19
  • Note that the canonical response to questions like this is currently [given here](https://stackoverflow.com/questions/1373164/how-do-i-create-a-variable-number-of-variables). – holdenweb Jul 12 '20 at 10:24
6

Adding to the above great answers. The above will work flawless if you need to create empty data frames but if you need to create multiple dataframe based on some filtering:

Suppose the list you got is a column of some dataframe and you want to make multiple data frames for each unique companies fro the bigger data frame:-

  1. First take the unique names of the companies:-

    compuniquenames = df.company.unique()
    
  2. Create a data frame dictionary to store your data frames

    companydict = {elem : pd.DataFrame() for elem in compuniquenames}
    

The above two are already in the post:

for key in DataFrameDict.keys():
    DataFrameDict[key] = df[:][df.company == key]

The above will give you a data frame for all the unique companies with matching record.

holdenweb
  • 33,305
  • 7
  • 57
  • 77
ak3191
  • 583
  • 5
  • 14
  • Thanks for editing @zx485. Can you help me with one question:- How i can split the dictionary back into multiple dataframes based on all the unique names of the comapny? – ak3191 Jul 25 '18 at 20:39
  • I'm sorry, but I'm no _Python_ guy. – zx485 Jul 25 '18 at 20:41
  • 2
    I think something is wrong in your code. Last part of code should be: `for key in companydict.keys():` ` companydict[key] = df[:][df.company == key]` But in any case I do not see exactly what is the output of this – pink.slash Apr 07 '19 at 23:00
  • @pink.slash for me the exact code worked but if there another use case i would be happy to have a look. – ak3191 Sep 05 '19 at 15:57
6

Below is the code for dynamically creating data frames in loop:

companies = ['AA', 'AAPL', 'BA', ....., 'YHOO']

for eachCompany in companies:
    #Dynamically create Data frames
    vars()[eachCompany] = pd.DataFrame()

For difference between vars(),locals() and globals() refer to the below link:

What's the difference between globals(), locals(), and vars()?

Chandan
  • 704
  • 7
  • 20
1

you can do this way:

for xxx in yyy:
   globals()[f'dataframe_{xxx}'] = pd.Dataframe(xxx)
Dave2e
  • 22,192
  • 18
  • 42
  • 50
0

The following is reproducable -> so lets say you have a list with the df/company names:

companies = ['AA', 'AAPL', 'BA', 'YHOO']

you probably also have data, presumably also a list? (or rather list of lists) like:

 content_of_lists = [
 [['a', '1'], ['b', '2']],
 [['c', '3'], ['d', '4']],
 [['e', '5'], ['f', '6']],
 [['g', '7'], ['h', '8']]
]

in this special example the df´s should probably look very much alike, so this does not need to be very complicated:

dic={}
for n,m in zip(companies, range(len(content_of_lists))):
   dic["df_{}".format(n)] = pd.DataFrame(content_of_lists[m]).rename(columns = {0: "col_1", 1:"col_2"}) 

Here you would have to use dic["df_AA"] to get to the dataframe inside the dictionary. But Should you require more "distinct" naming of the dataframes I think you would have to use for example if-conditions, like:

dic={}
    for n,m in zip(companies, range(len(content_of_lists))):
if n == 'AA':
    special_naming_1 = pd.DataFrame(content_of_lists[m]).rename(columns = {0:     
    "col_1", 1:"col_2"})
elif n == 'AAPL':
    special_naming_2 ...

It is a little more effort but it allows you to grab the dataframe object in a more conventional way by just writing special_naming_1 instead of dic['df_AA'] and gives you more controll over the dataframes names and column names if that´s important.

Aku
  • 660
  • 6
  • 9