Fastest way to make a list-like object containing integers in python

Question

What is the fastest way to make a list-like object containing integers/floats (very simple datatypes) in python?

What do I mean by "list-like"?

This means I want to have an object that supports the two (very) basic operations of a list: getting an object in a certain index (1) and changing its value (2).

What posts did I come across before posting this and why didn't they solve my problem?

I came across these two: [1] [2]

They didn't solve my problem because all the solutions to them were simply too slow: in my PC array.array('i',(0,)*10 ** 8) resulted in an error (lol); [0 for _ in range(10**8)] took about 15 seconds (wow!); [0] * 10 ** 8 took 2.3 seconds; [None] * 10 ** 8 took 1.8 seconds; (1.8sec could be faster...)

What did I try to do?

I tried using the ctypes module

from ctypes import c_int
array = (c_int * 10 ** 8)()

The code above took only 0.7 seconds ... but is there a way to make it faster? Besides being fast it has some disadvantages:

1. As it uses the c/c++ variables' skeleton, the integers in it will be in a "not as unlimited as python" integer value range
2. You can't have more than one datatype in the list
3. You have to import a module to use it

Is it really possible to do what I'm asking? Is there a faster way rather than using the ctypes module? If so make sure that you are using a 'built-in' / 'pre-installed' module.

Edit:

Why can't I simply install some module, like numpy?

I'm using python for competitive programming and most interpreters/judges just won't allow external libraries.

Can we have custom objects stored with array.array?

I can see many of the answers use the array function of the array module. They all use 'i' to specify we want to store integers. Is it possible to make a class and create an `array.array' containing it? For example:

class Point:
 def __init__(self, x, y):
  self.x = x
  self.y = y

# make array.array object with all indexes containing a Point with atributes x and y with value 0
# an example with a list of what I want to do is this:
# l = [Point(0, 0) for _ in range(10**3)]

Answer 1

array.array('i',(0,) * 10**8) resulted in an error (lol)

You didn't specify what error you got - that works for me, although it's not very fast because it builds an intermediate tuple and immediately discards it. Using Python's built-in types, array.array will probably result in best performance, provided you avoid the tuple:

a = array.array('i', (0,)) * 10**8

The code above took only 0.7 seconds ... but is there a way to make it faster?

It will be hard to beat array.array if you're not allowed to create or import C extensions. On my several years old machine the above takes 0.6 seconds. You can further optimize it by increasing the size of the initial array. For example, this produces the same result, but is almost 3x faster (!):

# 0.22 s
a = array.array('i', (0,) * 10) * 10**7

On my machine the following version works best:

# 0.19 s
a = array.array('i', (0,) * 100) * 10**6

Further increasing the initial array size doesn't help and soon starts degrading performance.

To get better efficiency, consider alternative approaches, such as a lazy list or an altogether different data structure tailored for your use case. Given the context of a competition, that might be what is actually being sought.

Be aware, however, that each solution will have different tradeoffs. For example, a lazy array such as one provided by @KonstantinNikitin will be extremely efficient to construct, but its __getitem__ and __setitem__, implemented in pure Python, will be several orders of magnitude slower than those of list or array.array. Which is better for you boils down to what operations are more frequent in your program, and that is up to you to find out.

Answer 2

I would just use the numpy module, which supports fast array operations.

For example making an array with numbers 0 to 10**8:

import numpy as np
import time

b = time.time()
a = np.linspace(0, 10**8, 10**8)
c = time.time()
print(c-b)
>>>0.5000154972076416

Or making an array of 0s that is 10**8 long:

b = time.time()
a = np.zeros(shape=(10**8,))
c = time.time()
print(c-b)
>>>0.0

The main reason why numpy is this fast is because it is implemented in C.

EDIT: If you want to do it with only pre-installed packages, you can try using the array package:

import array
import time
r = time.time()
a = array.array('i', [0]) * (10**8)
print(time.time()-r)
>>>0.15627217292785645

Answer 3

I'd say that you can try different approaches:

1) numpy. It's really a standard for arrays. It comes with a cost of crossing Python <-> C boundary for each operation, but it really depends on your task.

x = numpy.array(10 ** 8)

timeit.timeit('x = numpy.array(10 ** 8)', 'import numpy', number=1)
4.195800283923745e-05

2) lazy initialization (like JavaScript arrays).

class LazyArray:
    def __init__(self, size):
        self.storage = {}
        self.size = size

    def check(self, i):
        if i < 0 or i >= self.size:
            raise RuntimeError() 

    def __getitem__(self, i):
        self.check(i)
        return self.storage.get(i, 0)

    def __setitem__(self, i, value):
        self.check(i)
        self.storage[i] = value

x = LazyArray(10 ** 8)
x[10]
>> 0
x[10] = 5
x[10]
>> 0

Answer 4

If you really only want these two properties:

getting an object in a certain index (1) and changing its value (2)

then you can just use a collections.defaultdict:

import collections
my_list = collections.defaultdict(lambda: 0)

which is rather fast (~0.4 μs):

$ python3 -m timeit -s 'import collections' 'collections.defaultdict(lambda: 0)' 
1000000 loops, best of 3: 0.417 usec per loop

however, actually using it will probably be quite a bit slower than any of the types suggested in other answers.

Answer 5

For cases where you only need integers from 0 to 255, bytearray objects are quite fast to create:

>>> timeit.timeit('bytearray(100000)', number=1000)
0.005567271093696036
>>> timeit.timeit('array.array("B", [0])*100000', 'import array', number=1000)
0.36631167401839093
>>> timeit.timeit('array.array("i", [0])*100000', 'import array', number=1000)
0.56494557472422

Unlike array.array, this zeros the allocation directly instead of copying from an object initialized with zeros.