0

What is the best way to replicate the below behavior in a cython (without having to interact with Python)? Assuming that the function which will be passed into the new process is a cdef function.

import time
from multiprocessing import Process

def func1(n):
    while True:
        # do some work (different from func2)
        time.sleep(n)

def func2(n):
    while True:
        # do some other work (different from func1)
        time.sleep(n)


p1 = Process(target=func1, args=(1,))
p1.start()
p2 = Process(target=func2, args=(1,))
p2.start()

How to run a function in another process using Cython (without interacting with Python)?

Greg
  • 8,175
  • 16
  • 72
  • 125
  • What do you mean by "not interacting with Python"? The only way I see is to compile with a C compiler the output of Cython and start a process that way, but it's not clear wether this is what you seek – Adonis Apr 17 '18 at 08:04
  • @Adonis By "not interacting with Python" I mean similar behavior to `cdef` when compared to `def` or `cpdef` when defining functions. Where cdef is only available through cython and as such has the least Python overhead. That would be an ideal solution. – Greg Apr 17 '18 at 08:09
  • You're going to have to use the multi threading implementation of OpenMP with Cython, see [Using Parallelism](http://cython.readthedocs.io/en/latest/src/userguide/parallelism.html) as well as [this post](https://stackoverflow.com/q/24132506/4121573) – Adonis Apr 17 '18 at 11:23
  • 1
    @Adonis I'm looking at spawning threads to perform unique tasks instead of splitting data amongst threads to perform the same task. The documentation and post seem to refer to the latter use case which is not the solution needed. – Greg Apr 17 '18 at 11:58
  • Then please post a [mcve] as what you are seeking is absolutely not clear. In your example, you refer to multiprocessing, in your last comment to multi threading which are not the same concept. – Adonis Apr 17 '18 at 12:05
  • @Adonis Hopefully the example is clearer now. In terms of threading / multiprocessing, I am after multiprocessing (I referred to it as threading as in c++ that is what is meant by multiprocessing). – Greg Apr 17 '18 at 12:16
  • It isn't obvious what you gain by avoiding Python? I'd think that it's reasonably slow to start a new process so the Python overhead would be negligible. Given that the multiprocessing module does provide useful features (e.g. data serialization) why not just use it? – DavidW Apr 17 '18 at 16:04
  • @DavidW I am looking to give the processes read-only access to memory of another process without copying the memory each time, it's related to this question: https://stackoverflow.com/questions/49872179/how-to-grant-read-only-non-copy-access-to-an-object-in-another-process-in-cython – Greg Apr 17 '18 at 23:41
  • 1
    There is no such thing as shared memory among processes, except memory explicitly allocated as such by `multiprocessing.Value` et al. So what you are after is not suitable for your program. Read up on multi threading - threads within the same process do share memory. If the data is separate, then it is by definition not shared. So why is 'shared memory' needed in the first place? Can't the data be generated by the process which is consuming it? In any case, to avoid python you'd have to write all the threading/multi processing code in C/C++. – danny Apr 18 '18 at 16:17

0 Answers0