Use TQDM Progress Bar with Pandas

Question

Is it possible to use TQDM progress bar when importing and indexing large datasets using Pandas?

Here is an example of of some 5-minute data I am importing, indexing, and using to_datetime. It takes a while and it would be nice to see a progress bar.

#Import csv files into a Pandas dataframes and convert to Pandas datetime and set to index

eurusd_ask = pd.read_csv('EURUSD_Candlestick_5_m_ASK_01.01.2012-05.08.2017.csv')
eurusd_ask.index = pd.to_datetime(eurusd_ask.pop('Gmt time'))

Arjun Kava · Answer 1 · 2018-10-15T04:49:14.443

226

Find length by getting shape

for index, row in tqdm(df.iterrows(), total=df.shape[0]):
   print("index",index)
   print("row",row)

edited Oct 15 '18 at 04:49

answered Sep 13 '18 at 07:51

Arjun Kava

5,303
3
20
20

4

I think this is a better solution than from @sonance207, as this does not convert the iterator to a list, but accesses a given property. – guerda Nov 13 '18 at 12:53
1

this solution doesn't seem to work with df.itertuples() ? – Giacomo Apr 26 '19 at 11:27
no, but he is asking while reading...if you have a dataframe already are you assuming the file has been read once? – mithunpaul Oct 25 '19 at 03:57
2

doesn't iterrows slow down processing by a lot? – Matthew May 14 '20 at 17:00
Basically, the example shows that you can use `df.shape[0]` to get a length of the list. Otherwise, you can replace `df.iterrows()` any other compatible method. – Arjun Kava Jun 09 '20 at 15:08
3

@Giacomo: At least for me and tqdm version '4.42.0' it also works for itertuples. `for idx_row, col1, col2, ... in tqdm.tqdm(df.itertuples(), total=len(df)):` – gebbissimo Sep 04 '20 at 12:01
1

holds also for tqdm_notebook – Ofer Rahat Aug 31 '22 at 12:25

score 22 · Answer 2 · edited Oct 14 '20 at 05:47

22

with tqdm(total=Df.shape[0]) as pbar:    
    for index, row in Df.iterrows():
        pbar.update(1)
        ...

edited Oct 14 '20 at 05:47

Community

1
1

answered Jan 19 '18 at 23:14

jmcgrath207

1,317
2
19
31

Add pbar.close(); at the end, else the loop will keep running (not exit), so will not move to the next code block. – Abhilash Singh Chauhan May 11 '21 at 16:31
1

the `with` statement does close the object for you – meduz Jul 07 '21 at 13:17

score 3 · Answer 3 · edited Jun 30 '21 at 00:28

There is a workaround for tqdm > 4.24. As per https://github.com/tqdm/tqdm#pandas-integration:

from tqdm import tqdm
        
# Register `pandas.progress_apply` and `pandas.Series.map_apply` with `tqdm`
# (can use `tqdm_gui`, `tqdm_notebook`, optional kwargs, etc.)
tqdm.pandas(desc="my bar!")
eurusd_ask['t_stamp'] = eurusd_ask['Gmt time'].progress_apply(lambda x: pd.Timestamp)
eurusd_ask.set_index(['t_stamp'], inplace=True)

score 1 · Answer 4 · answered Nov 09 '17 at 20:17

1

You could fill a pandas data frame in line by line by reading the file normally and simply add each new line as a new row to the dataframe, though this would be a fair bit slower than just using Pandas own reading methods.

answered Nov 09 '17 at 20:17

ZeerakW

323
1
3
9

85nd · Answer 5 · 2022-12-19T14:34:09.847

1

I find it very easy to implement. You only need to add the total argument.

import pandas as pd
df = pd.read_excel(PATH_TO_FILE)


for index, row in tqdm(df.iterrows(),  total=df.shape[0], desc=f'Reading DF'):
        print(row(['df_colum'])

edited Dec 19 '22 at 14:34

answered Dec 19 '22 at 14:31

85nd

11
5

Use TQDM Progress Bar with Pandas

5 Answers5