0

About a decade ago, when I played around with C++ in my high school, I learnt about pointers, and memory overflows. In C++, sometimes, using a pointer to expand (or contract) an array in place can cause it to run over (or under) the memory allotted, and cause weird behaviour. I am interpreting slice assignment in Python to be something similar to assignment by pointers:

a[:] = list(range(10000))  # Similar to using pointers in C++,
# because memory location remains unchanged.

So how does Python avoid overflows (or underflows)?

Of course, in C++, we initialize each array to a specified size, and pointer-based assignments violating that size are horrible programming practice.

Kartik
  • 8,347
  • 39
  • 73
  • *"I am interpreting slice assignment in Python to be something similar to assignment by pointers"* It isn't. More like "pointers with guard rails that will not let you overrun/underrun". Like in Java. You can only pass a legal index, not any arbitrary positive or negative index. – smci Apr 04 '18 at 20:34
  • 1
    Python manages its memory behind the scenes to prevent memory overflows. You may get an out of memory exception but that's still well defined behavior. – Mark Ransom Apr 04 '18 at 20:34
  • 3
    It is a mistake to assume that just because you are modifying an existing list, your new storage must go into the same space in memory. – khelwood Apr 04 '18 at 20:35
  • Incidentally, the things you're saying about C++ indicate that you need to stop using so many arrays and get used to [vectors](http://en.cppreference.com/w/cpp/container/vector), which also resize automatically. – user2357112 Apr 04 '18 at 20:36
  • In (pure) Python you don't need to worry about memory _at all_. At least to a point where you run out of said memory, but that's another problem. – ForceBru Apr 04 '18 at 20:36
  • In pythons, a list object is an intermediate structure that knows how to expand and contract its internal storage as needed. As long as you don't blow out of available memory completely and the python programmers are any good, it won't blow up. – tdelaney Apr 04 '18 at 20:44
  • Thanks everyone! @smci, not quite what I meant. I know one cannot pass an arbitrary positive or negative index. I meant a different length of array. @user2357112, Honestly, I have not used much C/C++ since I graduated high school. I know that what I learnt back then was just scratching the surface. @tdelaney, @ForceBru, @MarkRansom, How does it do that? I tried initializing a small list, and replacing it with a gigantic one, yet the `id(.)` function gives the same memory location. @khelwood, please explain. Thanks. – Kartik Apr 04 '18 at 21:00
  • @smci: That's exactly what I said. You can't access a list with an arbitrary index, and you can't hoax or redeclare its length in order to do that, unlike C/C++ – smci Apr 04 '18 at 21:19
  • Some clarifications about C++: Stack-allocated C-arrays or std::arrays can not be resized. Heap-allocated arrays can be resized with `realloc` (if allocated with `malloc`) or `delete[]` + `new[]` (if allocated with `new[]`). `std::vector`s can be resized using member functions such as `resize` or `push_back`. None of those options can cause buffer overflows. A buffer over-/underflow would happen if you write to an invalid index (i.e. `arr[i] = something;` where `i` is greater than the length of `arr` or less than 0). In Python that would always cause an exception. – sepp2k Apr 04 '18 at 21:20
  • How it works: a `list` is an object which *contains* a buffer of references to other objects. If it needs to be resized, a new buffer is created which replaces the old one. The `list` object itself doesn't get replaced so it retains its ID. – Mark Ransom Apr 09 '18 at 19:14

1 Answers1

1

Python lists are resizable arrays. Most of the time, there is extra space to allow for growth (and shrinkage) without reallocation. When more space is needed, or too much is wasted, the internal space is reallocated. This can result in a MemoryError if more space is needed and not available. The details depend on the implementation, OS, and OS settings.

Terry Jan Reedy
  • 18,414
  • 3
  • 40
  • 52
  • Can you link to a source or two where I can read and learn more? Thanks. – Kartik Apr 04 '18 at 21:01
  • https://stackoverflow.com/questions/3917574/how-is-pythons-list-implemented; https://www.laurentluce.com/posts/python-list-implementation/ For more, search SO for `[python] list implementation` or web for `python list implementation`, which is what I did. – Terry Jan Reedy Apr 06 '18 at 01:30