15

I use the logging module in the main function/process, it works well, but it seems can't work in Actor process/subprocess. How to make it work? In the sample below code, logging.info work in the main process but failed in the worker process. Thanks.

import logging
import ray

@ray.remote
class Worker(object):
   ...

   def train(self):
       logging.info("fail print")


...

worker = Worker.remote()

ray.get(worker.train.remote())

logging.info("successful print")
Han Zheng
  • 309
  • 2
  • 8

3 Answers3

17

There are a couple things to be careful about.

  • First, you should create a new logger inside of the worker because the worker runs on a different Python process. If you try to use a logger that you created outside of the worker within the worker, then Ray will try to pickle the logger and send it to the worker process, and Python loggers typically do not behave correctly when pickled and unpickled.
  • Second, you have to make sure the logging level is set correctly. I'm using logger.warning instead of logger.info because the Python logging level is set to `warning by default.

Here is a working example:

import logging
import ray

logger = logging.getLogger(__name__)

@ray.remote
class Worker(object):
    def __init__(self):
        self.logger = logging.getLogger(__name__)
    def train(self):
        self.logger.warning("print from inside worker")


ray.init()

worker = Worker.remote()

ray.get(worker.train.remote())

logger.warning("print from outside worker")
Robert Nishihara
  • 3,276
  • 16
  • 17
  • Thanks. If I want to write the log info into the same debug file, which way is good? creat two loggers and set the same basicConfig? `logging.basicConfig(...,filename="the same filepath")` Can I create a shared Config? – Han Zheng Mar 24 '19 at 10:48
  • I haven't tried it, so I'm not sure, but I think that having two workers log to the same file would be undesirable because they may overwrite each other's logs or may intersperse their logs making the file difficult to read. – Robert Nishihara Mar 25 '19 at 22:13
  • loggers are thread-safe. just make sure to make the file handler use append mode: `your_logger.addHandler(logging.FileHandler(outpath, mode="a+"))` – crypdick Jul 31 '20 at 17:59
  • 2
    I am creating the logger inside the ray remote function and also asking it to write to same log file but it is logging the main info not from the remote function. Any pointers will be helpful. logger = logging.getLogger(log_name) logger.addHandler(logging.FileHandler(lfn, mode="a+")) – user3341078 Jul 24 '21 at 02:30
1

I've had trouble with Ray suppressing console output from loggers. To get around that, add a StreamHandler:

your_logger.addHandler(logging.StreamHandler())
crypdick
  • 16,152
  • 7
  • 51
  • 74
  • i am facing a similar issue, where `import ray` is overriding my custom logging formatter. i posted a question https://stackoverflow.com/questions/71084138/doing-import-ray-breaks-my-custom-logging-formatter if you have any ideas or experience please assist! – Kots Feb 11 '22 at 18:53
0

A little late to the game here, but to get around this I do something like the following:

import ray
import os
import importlib
import logging

@ray.remote()
def my_remote_func_w_custom_logger():

    logging.shutdown()
    importlib.reload(logging)

    pid = os.getpid()

    logging.basicConfig(format= f"%(asctime)s - %(levelname)s - %(funcName)s - PID {pid} - %(message)s", 
                        level= logging.INFO,
                        datefmt= '%m/%d/%Y %I:%M:%S %p',
                        filename= 'remote_func.log')
   
    logger = logging.getLogger(__name__)

    logger.info("This is the remote function.")

So essentially, reload the logging module after you import ray and then specify your logging config. I don't know if this is the best solution, but it seems to work.

I've run into this problem with other packages as well, not just ray, but I have not done enough research to understand why some packages have this behavior while others do not.

KG3
  • 1