0

I want to create a vector with the size 10^15 with numpy and fill it with random numbers, but I get the following error:

Maximum allowed dimension exceeded.

Can it help if i use MPI?

Thank you

Thorte
  • 31
  • 5
  • SO answer here: https://stackoverflow.com/questions/14351255/techniques-for-working-with-large-numpy-arrays – pink spikyhairman May 05 '20 at 10:54
  • Do you have enough RAM to hold a quadrillion numbers? That would take something like 8000 TB = 8 million GB – John Coleman May 05 '20 at 10:55
  • 1
    Does this answer your question? [Techniques for working with large Numpy arrays?](https://stackoverflow.com/questions/14351255/techniques-for-working-with-large-numpy-arrays) – Joe May 05 '20 at 11:35

1 Answers1

0

The Message Passing Interface (MPI) is mainly used to do parallel computations across multiple machines (nodes). Large arrays can be split into smaller arrays and stored on different machines. However, while it's of course possible to distribute the data to different nodes, you should carefully think about the necessity of doing this for your particular task. Additionally, if you are able to split your array, you could also do this on one machine. If performance is not an issue, avoid parallel computing.

Joscha Fregin
  • 204
  • 6
  • 13