4

I'm coding with Python and Spacy. I want to track the progress of the execution of nlp.pipe(sentences) because it lasts a lot. How to do that?

nlp = spacy.load('en_core_web_sm')
sentences = [...]
docs = nlp.pipe(sentences, n_process=8)
Marco
  • 41
  • 5

1 Answers1

3

Use tqdm.

from tqdm import tqdm

nlp = spacy.load('en_core_web_sm')
sentences = [...]
for doc in tqdm(nlp.pipe(sentences, n_process=8)):
    ... do stuff ...
polm23
  • 14,456
  • 7
  • 35
  • 59
  • 1
    `7465it [00:37, 197.51it/s] ` I don't have progress bar. Do you know why? Do you tested it with `nlp.pipe`? – Marco Jun 28 '21 at 10:09
  • 2
    Please read the tqdm documentation. `nlp.pipe` returns a generator, so tqdm doesn't know how long it is. You can do `tqdm(nlp.pipe(sentences), total=len(sentences))`. https://stackoverflow.com/questions/41985993/tqdm-show-progress-for-a-generator-i-know-the-length-of – polm23 Jun 29 '21 at 04:44