0

Ray allows for parallel processing and I am trying to use it with scipy module. I just setup ray and I am not sure if the behavior is expected. But anyway here is the script and the output.


    import math
    import cmath as cm
    import numpy as np
    from scipy.integrate import dblquad
    import ray

    ray.init(num_cpus=6) # initializing ray here 
    #-----------------------------------------------------------

    # DEFINE FUNCTIONS (I am not sure if i should use decorator here)

    def wrap_real(r,  rho, phi):
        res = cm.cos(phi)*cm.exp(-1j*2*math.pi*cm.sqrt(rho**2 +\
                    r**2))/cm.sqrt(rho**2 + r**2)
        return res.real

    def wrap_imag(r,  rho, phi):
        res = cm.cos(phi)*cm.exp(-1j*2*math.pi*cm.sqrt(rho**2 + \
                    r**2))/cm.sqrt(rho**2 + r**2)
        return res.imag

    rMax = 5

    # Functions called by ray
    @ray.remote
    def I_real (value ):
        return dblquad(lambda rho, phi: wrap_real (value ,  rho, phi) \
                       , 0, rMax, lambda x: 0, lambda x: 2*math.pi)
    @ray.remote
    def I_imag (value ):
        return dblquad(lambda rho, phi: wrap_imag ( value,  rho, phi) ,\
                       0, rMax, lambda x: 0, lambda x: 2*math.pi)
    #-----------------------------------------------------------
    #-----------------------------------------------------------
    # TEST INTEGRATION

    print("\n-----------COMPLEX INTEGRATION RESULT ----------")

    run1 = I_real.remote(6)
    run2 = I_imag.remote(6)

    run3 = I_real.remote(6)
    run4 = I_imag.remote(7)

    run5 = I_real.remote(8)
    run6 = I_imag.remote(10)

    resultp1=ray.get([run1, run2])
    resultp2=ray.get([run3, run4])
    resultp3=ray.get([run5, run6])

    print(resultp1)
    print(resultp2)
    print(resultp3)


OUTPUT

2019-06-01 19:07:55,609 INFO node.py:497 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-06-01_19-07-55_608930_25145/logs.
2019-06-01 19:07:55,713 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:32983 to respond...
2019-06-01 19:07:55,846 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:18737 to respond...
2019-06-01 19:07:55,847 INFO services.py:806 -- Starting Redis shard with 10.0 GB max memory.
2019-06-01 19:07:55,882 INFO node.py:511 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-06-01_19-07-55_608930_25145/logs.
2019-06-01 19:07:55,883 WARNING services.py:1293 -- Warning: Capping object memory store to 20.0GB. To increase this further, specify `object_store_memory` when calling ray.init() or ray start.
2019-06-01 19:07:55,884 INFO services.py:1441 -- Starting the Plasma object store with 20.0 GB memory using /dev/shm.

-----------COMPLEX INTEGRATION RESULT ----------
2019-06-01 19:07:56,446 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,467 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,467 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,467 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,548 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,560 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,571 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,572 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,604 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,606 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,628 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,629 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 59, in <module>
    resultp1=ray.get([run1, run2])
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 2197, in get
    raise value
ray.exceptions.RayTaskError: ray_worker:ques_modpy_asked_SO_modifying2.I_real() (pid=25201, host=varsh)
TypeError: f() takes 0 positional arguments but 1 was given

>>> 2019-06-01 19:07:56,911     ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,911 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,932 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,933 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,934 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,945 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,967 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,968 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,979 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,979 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:56,990 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_real' with function ID ca83a4149613ad1164353ce7183f9a21dd1da53f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.

2019-06-01 19:07:57,000 ERROR worker.py:1678 -- Failed to unpickle the remote function 'ques_modpy_asked_SO_modifying2.I_imag' with function ID b09f533dace5def97fe6c388da2e8b0be6c7d06f. Traceback:
Traceback (most recent call last):
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/function_manager.py", line 424, in fetch_and_register_remote_function
    function = pickle.loads(serialized_function)
  File "/home/ankit/Documents/confocal_vol_python/ques_modpy_asked_SO_modifying2.py", line 13, in <module>
    ray.init(num_cpus=6)
  File "/home/ankit/.local/lib/python3.6/site-packages/ray/worker.py", line 1351, in init
    raise Exception("Perhaps you called ray.init twice by accident? "
Exception: Perhaps you called ray.init twice by accident? This error can be suppressed by passing in 'ignore_reinit_error=True' or by calling 'ray.shutdown()' prior to 'ray.init()'.


Any suggestions regarding the output.

ankit7540
  • 315
  • 5
  • 20
  • Probably related to this question, https://stackoverflow.com/questions/39777299/weird-behavior-of-multiprocessing-scipy-optimization-inside-of-a-function?rq=1 But i do not know how to bypass the limitation. – ankit7540 Jun 01 '19 at 11:23
  • 1
    code working, don't get any error related to ray.init – Zaraki Kenpachi Jun 01 '19 at 12:08
  • Oh, i see. I will check my installation. – ankit7540 Jun 01 '19 at 12:34
  • @Zaraki Kenpachi : If possible could you post the output received. (maybe use pastebin or something) – ankit7540 Jun 01 '19 at 13:39
  • 1
    [(-0.11678688737646281, 1.7636954379590675e-09), (0.1479416501795927, 3.1158250007878863e-09)], [(-0.11678688737646281, 1.7636954379590675e-09), (0.147397575183974, 3.0855094109057124e-10)], [(-0.14107337021052904, 3.9931410135159824e-11), (0.09913984173910342, 1.4866783904780216e-12)] – Zaraki Kenpachi Jun 01 '19 at 13:46
  • @Zaraki Kenpachi : Thank you. :-) – ankit7540 Jun 01 '19 at 14:02
  • @Zaraki Kenpachi : I was able to get it working after a new installation of ray and updating numpy on my system. – ankit7540 Jun 01 '19 at 16:28
  • 1
    The code you posted works for me as well. However, based on the error, my guess is that what is happening is when the remote function `ques_modpy_asked_SO_modifying2.I_real` (from the error message) is pickled and shipped to the worker processes, the worker processes try to unpickle it, and the process of unpickling it requires the module `ques_modpy_asked_SO_modifying2.py` to be imported, and that runs `ray.init()`, which shouldn't be run on the workers. If the issue occurs again, it should be fixable by moving the `ray.init()` into a `if __name__ == '__main__':` block. – Robert Nishihara Jun 02 '19 at 02:13
  • @Robert Nishihara : Thank you. I was able to use my code after correct installation of Ray. :-) – ankit7540 Jun 02 '19 at 03:50

1 Answers1

1

Found that the problem was in building Ray from source and presumably it was not installed correctly.

An installation using pip install -U ray did it.

As mentioned in the comments by Robert, moving ray.init() inside if __name__ == '__main__': block allows for proper execution on a Windows machine.

Ray Documentation

ankit7540
  • 315
  • 5
  • 20