1

When a list is sliced, are the references to its contents copied from the original list? I can imagine that this may not be necessary, but I read the opposite (mentioned in passing).

This question matters for instance for the following idiom, in the case of a very long my_list:

for (first_elmt, second_elmt) in itertools.izip(my_list[:-1], my_list[1:]):
    …

A copy would use up both memory and, presumably, some time. I compared to looping over the index of first_elmt with xrange(), on a list of 100 million integers. The slicing approach is actually 20% faster, but does seem to copy references (the system time is longer). Is this indeed the case?

PS: I now realize that it is quite natural that slices copy the references: if the original list is modified, the slice does not change, so it is easier to have the implementation of the slice copy the references of the original list. A pointer to the CPython implementation would be interesting, though.

Community
  • 1
  • 1
Eric O. Lebigot
  • 91,433
  • 48
  • 218
  • 260
  • 2
    `does seem to copy data` - Nope, it will never do that. In fact, there will never be any copying of data unless you explicitly ask for it. – thefourtheye Jun 14 '15 at 08:37
  • I agree. I meant "copy the references", as I was writing in the title and at the beginning of the question. – Eric O. Lebigot Jun 14 '15 at 11:42

2 Answers2

4

Slicing will copy the references. If you have a list of 100 million things:

l = [object() for i in xrange(100000000)]

and you make a slice:

l2 = l[:-1]

l2 will have its own backing array of 99,999,999 pointers, rather than sharing l's array. However, the objects those pointers refer to are not copied:

>>> l2[0] is l[0]
True

If you want to iterate over overlapping pairs of elements of a list without making a copy, you can zip the list with an iterator that has been advanced one position:

second_items = iter(l)
next(second_items, None) # Avoid exception on empty input
for thing1, thing2 in itertools.izip(l, second_items):
    whatever()

This takes advantage of the fact that zip stops when any input iterator stops. This can be extended to cases where you're already working with an iterator using itertools.tee

i1, i2 = itertools.tee(iterator)
next(i2, None)
for thing1, thing2 in itertools.izip(i1, i2):
    whatever()
user2357112
  • 260,549
  • 28
  • 431
  • 505
  • Do you have a reference about the copy of the references? Now that I think of it, it makes sense to copy them just in case the original list is modified, though. Another point: instead of `next(…, None)`, I find it clearer to directly use `itertools.islice(…, 1, None)`, since it explicitly is a slice (and there is no need for an arbitrary default value like `None`, as in the case of `next()`). – Eric O. Lebigot Jun 14 '15 at 11:46
  • 1
    @EOL: From the tutorial: "All slice operations return a new list containing the requested elements. This means that the following slice returns a new (shallow) copy of the list: `>>> squares[:]`". You can also look at the [source code](https://hg.python.org/cpython/file/a985b6455fde/Objects/listobject.c#l466). – user2357112 Jun 14 '15 at 21:50
  • Thanks! That's something I did not remember from reading the tutorial 10 years ago. :D – Eric O. Lebigot Jun 15 '15 at 01:50
2

Yes, slicing DO copy references, in fact, it is an idiom to make copy of a list this way: newlst = lst[:].

NeoWang
  • 17,361
  • 24
  • 78
  • 126
  • Good point. This is indeed yet another reason to expect that slices copy references (in addition to the fact that a slice does not change even if the original list does). – Eric O. Lebigot Jun 15 '15 at 13:06