-2

I've written a python script with heavy computational costs. After running the code in a remote linux server(via putty), it takes one week to get finished.

Our remote server has 32 cpu cores and 64 GB of Ram memory. I need to run my script in a shorter time by using more numbers of cpu cores but I don't know exactly how to increase using more numbers of cpu cores for my script.

I would like to know if there is any python code to add at the first or end part of my python script for adding numbers of cpu cores, memory usage and etc to be able to run the script in a shorter time. I also need to return to the default state after getting my script result.

Is there any way to achieve this in Python?

Sara
  • 163
  • 2
  • 10

2 Answers2

2

You can take a look on multiprocessing lib (https://docs.python.org/2/library/multiprocessing.html) but you will have to manage data to share between processes.

You can also take a look on threading lib (https://docs.python.org/2/library/threading.html) which may be efficient if you have lot of I/O.

Finally you use asyncio lib (python 3.4+ https://docs.python.org/3/library/asyncio.html) which is, in my opninion, more complicated to implement but give you more ability to manage explicitly the behavior of your code.

Edit: This is a large question, you may find lot of documentation on this issue. It really depends on what your code really do, you should better give a snippet of your code.

srjjio
  • 930
  • 1
  • 8
  • 16
1

Rewrite your script to do things in parallel. Since threads don't use more than one CPU in CPython, you should use https://docs.python.org/2/library/multiprocessing.html.

See also: Does python support multiprocessor/multicore programming?

Thiago F. Pappacena
  • 326
  • 1
  • 5