Why Python shared memory requires a copy?

Question

I was trying to build a simple example of new Python SharedMemory class by amending a shared numpy array. I`ve seen a few examples on the web here:

Shared memory in multiprocessing

Is shared readonly data copied to different processes for multiprocessing?

However all of them change the shared array inside the __main__ block for some reason and then just print it. However, I would like to return a shared array from the function and reuse it later. Here is my try:

from multiprocessing import shared_memory
import numpy as np


def create_shared_array():
    # instantiate array
    np_array = np.ones((5, 2, 2))
    # instantiate shared array
    shm = shared_memory.SharedMemory(size=np_array.nbytes, create=True)
    # copy data from original array into shared array
    array_shared = np.ndarray(np_array.shape, dtype=np_array.dtype, buffer=shm.buf)
    array_shared[:] = np_array[:]
    return shm


def change_and_return(name):
    existing_shm = shared_memory.SharedMemory(name=name, create=False)
    array_shared = np.ndarray((5, 2, 2), dtype=np.float64, buffer=existing_shm.buf)
    # change shared array
    array_shared[0] = 888
    return array_shared


shm = create_shared_array()
result = change_and_return(shm.name)
print(result)

For some reason this code returns "The instruction at <memory address> referenced memory at <another address>. The memory could not be read." on my Windows machine.

However, if I just replace this

return array_shared

with this

return array_shared.copy()

Then it works just fine.

My thoughts:

Since change_and_return does not return existing_shm then it gets garbage collected and array_shared is destroyed as well. However, I do make sure not to kill shm so the reference should still exist ?

Question: why an additional copy is required and what is the right way here to avoid making an additional copy ?

UPDATE: let me be clear here - I would like to change array values in shared memory but after that I would like to use it in a non-shared fashion. Just like a regular numpy array. So the question is basically if it is possible to turn a shared array back into non-shared one without copying ?

"Then it works just fine." - no it doesn't. The copy isn't backed by shared memory. — user2357112, Feb 02 '21 at 03:26
The code works for me in this case - does it break for you ? — Kreol, Feb 02 '21 at 03:39
It runs without raising an exception, but it doesn't actually work - it fails silently, doing something completely different from what you intended. Again, the copy isn't backed by shared memory. — user2357112, Feb 02 '21 at 03:42

score 1 · Accepted Answer · answered Feb 02 '21 at 03:46

1

The copy is wrong. It makes an array backed by non-shared memory, completely defeating the point of what you're doing.

You say that

However, I do make sure not to kill shm

but that's not true. You do nothing to keep shm alive. It dies, and its __del__ method calls close, and the memory is no longer accessible.

You need to retain a reference to shm to make sure the memory is still accessible. Additionally, for proper cleanup, each process should call shm.close() after it no longer needs to use the shared memory, and a single process should call shm.unlink() after all processes have called close. Otherwise, you risk leaking the memory even after your program has terminated. (On Unix, Python spawns a server process to try to protect against this, but it's not completely reliable, and even when it works, it'll perform cleanup late.)

answered Feb 02 '21 at 03:46

user2357112

260,549
28
431
505

Sorry but if I do shm.close(), existing_shm.close() after I call both functions then memory error will happen as well. What is the right way to use array_shared and not pair it every time with the corresponding SharedMemory class instance ? – Kreol Feb 02 '21 at 03:50
@Kreol: You need to call it **after you no longer need the shared memory**. You're calling it while you still need the memory. – user2357112 Feb 02 '21 at 03:51
"What is the right way to use array_shared and not pair it every time with the corresponding SharedMemory class instance ?" - there is no right way to do that. You need to keep the SharedMemory object around. – user2357112 Feb 02 '21 at 03:52
I do not need array_shared to be "shared" after I have done transforms to it in other processes. I just would like to keep using this numpy array in my main process. I guess your point is that in this case I have to make another copy ? – Kreol Feb 02 '21 at 03:56
@Kreol: Then stop using shared memory! Why are you using shared memory if you don't want it to be shared? – user2357112 Feb 02 '21 at 03:57
As I said, I just would like to make changes to the array using shared memory. But I do not need shared memory AFTER I made changes. Just the array itself – Kreol Feb 02 '21 at 03:58
2

@Kreol: Then go ahead and make a non-shared copy, but remember to clean up the shared memory. – user2357112 Feb 02 '21 at 03:59

Why Python shared memory requires a copy?

1 Answers1