0

I have the following code running on a large ubuntu instance:

    log.info('ArtworkURL: %s/%s' % (num, num_chunks))
    t1 = time.time()
    pool = Pool(20)
    t2 = time.time()

Doing t2-t1 alone takes almost 30 seconds!!

28.9083909988 Pool |

What could be causing this? This function usually takes less than a half second, even on my local (much less powerful) machine. Python 2.7.

David542
  • 104,438
  • 178
  • 489
  • 842
  • Where are you executing code using the pool? Have you tried profiling? Is the instance you are running on already under heavy load? – jordanm Feb 19 '16 at 18:54
  • @jordanm actually the code that runs in parallel is quite fast. It is only the `Pool(n)` that takes all the time: `28.6946499348 Pool | 1.24954819679 | 5.96046447754e-06 | 1.43729496002`. – David542 Feb 19 '16 at 19:19
  • @jordanm this had to do with memory consumption. It seems doing Pool.close() didn't immediately free up the memory, so when I tried creating a new pool, it would lag for quite some time. – David542 Feb 19 '16 at 19:31
  • if your task require working on some large variable (array, long list ?), it has to be copied in each task of the pool, which can cause memory overflow and slowing down the whole pool. Is it the case ? – CoMartel Feb 20 '16 at 23:03

0 Answers0