Python "range" resource consumption

Question

I wrote the following script

Basically, I'm just learning Python for Machine Learning and wanted to check how really computationally intensive tasks would perform. I observe that for 10**8 iterations, Python takes up a lot of RAM (around 3.8 GB) and also a lot of CPU time (just froze my system)

I want to know if there is any way to limit the time/memory consumption either through code or some global settings

Script -

initial_start = time.clock()
for i in range(9):
 start = time.clock()
 for j in range(10**i):
  pass
 stop = time.clock()
 print 'Looping exp(',i,') times takes', stop - start, 'seconds'
final_stop = time.clock()
print 'Overall program time is',final_stop - initial_start,'seconds'

Perform something computationally intensive, then. Did iterating a long list give you any insights? — Michael Foukarakis, Aug 15 '12 at 19:45
You are surprised that something computationally intensive uses a lot of CPU? — John La Rooy, Aug 15 '12 at 20:47
Hi Everybody. Thanks a lot for your help. This was my first question on Stackoverflow. But I haven't got the answer I was looking for. My question is - 'How do I limit the time/memory consumption of a Python program? If I end up running some really heavy code (bad programming/huge data etc.), I don't want it to take my system down' — Sammy25, Aug 16 '12 at 00:32

score 8 · Answer 1 · edited May 23 '17 at 10:24

8

In Python 2, range creates a list. Use xrange instead. For a more detailed explanation see Should you always favor xrange() over range()?

Note that a no-op for loop is a very poor benchmark that tells you pretty much nothing about Python.

Also note, as per gnibbler's comment, Python 3's range is works like Python 2's xrange.

edited May 23 '17 at 10:24

Community

1
1

answered Aug 15 '12 at 19:36

Steven Rumbalski

44,786
9
89
119

2

note that `range()` in Python3 works like `xrange()` from Python2 – John La Rooy Aug 15 '12 at 19:47

score 2 · Answer 2 · answered Aug 15 '12 at 19:38

2

Python takes RAM because you're creating a very large list of 10 ** 8 length with range function. That's where iterators become useful.

Use xrange instead of range.

It will work the same way as range do but instead of creating that large list in memory, xrange will just calculate inner index (incrementing it's value by 1 each iteration).

answered Aug 15 '12 at 19:38

Rostyslav Dzinko

39,424
5
49
62

Hi Rostyslav. Thanks a lot for your help. This was my first question on Stackoverflow. But I haven't got the answer I was looking for. My question is - 'How do I limit the time/memory consumption of a Python program? If I end up running some really heavy code (bad programming/huge data etc.), I don't want it to take my system down' – Sammy25 Aug 16 '12 at 02:20

score 2 · Accepted Answer · edited May 23 '17 at 10:34

look at this question: How to limit the heap size?

To address your script, the timeit module measures the time it takes to perform an action more accurately

>>> import timeit
>>> for i in range(9):
...     print timeit.timeit(stmt='pass', number=10**i)
...
0.0
0.0
0.0
0.0
0.0
0.015625
0.0625
0.468752861023
2.98439407349

Your example is taking most of its time dealing with the gigantic lists of numbers you're putting it memory. xrange instead of range will help fix that issue but you're still using a terrible benchmark. the loop is going to execute over and over and not actually do anything, so the cpu is busy checking the condition and entering the loop.

As you can see, creating these lists is taking the majority of the time here

>>> timeit.timeit(stmt='range(10**7)', number=1)
0.71875405311584473
>>> timeit.timeit(stmt='for i in range(10**7): pass', number=1)
1.093757152557373

score 1 · Answer 4 · edited Mar 27 '23 at 18:00

If you're considering Python for machine learning, take a look at numpy. Its philosophy is to implement all "inner loops" (matrix operations, linear algebra) in optimized C, and to use Python to manipulate input and output and to manage high-level algorithms - sort of like Matlab that uses Python. That gives you the best of both worlds: ease and readability of Python, and speed of C.

To get back to your question, benchmarking numpy operations will give you a more realistic assessment of Python's performances for machine learning.

score 0 · Answer 5 · answered Aug 15 '12 at 19:57

0

As regards cpu, you have a for loop running for billions of iterations without any sort of sleep or pause inbetween, so no wonder the process hogs the cpu completely ( at least on a single core computer).

answered Aug 15 '12 at 19:57

Sachin

915
6
10

Python "range" resource consumption

5 Answers5