How to see progress of Dask compute task?

Question

I would like to see a progress bar on Jupyter notebook while I'm running a compute task using Dask, I'm counting all values of id column from a large csv file +4GB, so any ideas?

import dask.dataframe as dd

df = dd.read_csv('data/train.csv')
df.id.count().compute()

Also https://stackoverflow.com/questions/44483950/combining-tqdm-with-delayed-execution-with-dask-in-python — jtlz2, Apr 05 '22 at 08:43

score 43 · Accepted Answer · answered Feb 28 '18 at 22:38

43

If you're using the single machine scheduler then do this:

from dask.diagnostics import ProgressBar
ProgressBar().register()

http://dask.pydata.org/en/latest/diagnostics-local.html

If you're using the distributed scheduler then do this:

from dask.distributed import progress

result = df.id.count.persist()
progress(result)

Or just use the dashboard

http://dask.pydata.org/en/latest/diagnostics-distributed.html

answered Feb 28 '18 at 22:38

MRocklin

55,641
23
163
235

Is there any chance to see total time to complete a task on the Dashboard? – ambigus9 Feb 28 '18 at 23:10
An individual function/task? No. Tasks contain arbirary Python code and so behave in unpredictable ways. – MRocklin Mar 01 '18 at 13:16
when running the .register where would one see the progress bar? – AZhao Jul 16 '18 at 13:49
When I use dask with the progress bar, it just freezes on zero while generating enough heat and CPU usage that I presume it's doing something. How does the progress bar get updated? – ifly6 Apr 08 '19 at 17:44
1

This is great for running Dask on Kaggle, which does not seem to support the dashboard (see https://www.kaggle.com/questions-and-answers/54405) – Nick Fernandez May 26 '19 at 04:50
Thank you, this is so very helpful! – rjurney Jun 10 '20 at 22:36

score 0 · Answer 2 · answered Oct 05 '21 at 08:55

0

This resource provides full-code examples for both cases (local and distributed) and more detailed information about using the Dask Dashboard.

Note that when working in Jupyter notebooks you may have to separate the ProgressBar().register() call and the computation call you want to track (e.g. df.set_index('id').persist()) into two separate cells for the progress bar to actually appear.

DO:

DON'T DO:

answered Oct 05 '21 at 08:55

rrpelgrim

342
2
13

1

Link is dead.... – jtlz2 Jun 11 '23 at 20:20

How to see progress of Dask compute task?

2 Answers2

Linked

Related