0

Initially, I could create a large matrix like 10,000 by 10,000 in less than a second but now it takes around 12 seconds to create a matrix of 1,000 by 1,000 and gives the error: can't start new threads when I try to create a matrix of 10,000 by 10,000.

I restarted my Macbook Air m1, yet the issue remains (I just restarted my laptop not shutdown).

I want it will run at the same pace as before.

Code:

import numpy as np
import threading

row,column = 1,1000
matrix = np.zeros((row,column))
test = np.zeros((row,column))

def inmatrix(matrix):
    global row,column,test
    for i in range(row):
        for j in range(column):
            matrix[i][j] = np.random.choice([0,1,2,3])
    test = np.concatenate((test,matrix),axis=0)


threads = list()
for index in range(1000):
    x = threading.Thread(target=inmatrix, args=(matrix,))
    threads.append(x)
    x.start()

for index, thread in enumerate(threads):
    thread.join()

print(test)
Akito
  • 1
  • 1

1 Answers1

0

yes, there is a limit

as you can see in this StackOverflow Question Maximum limit on number of threads in python

in my Macbook Pro it is 4096 threads

about threads in general

you are misusing threads, you can have faster results without using threads

example bellow is 4 times faster as a single thread, than your variant 1 000 x 1 000

for i in range(row):
    for j in range(column):
        matrix[i][j] = np.random.choice([0, 1, 2, 3])

there is an overhead (with threads and merging results), so threads should be used carefully

Michal Miky Jankovský
  • 3,089
  • 1
  • 35
  • 36