0

I am unable to see the logs when I configure the google cloud dataflow eclipse plugin as documented which is setting workerLogLevelOverrides to {\"com.mycompany.testdataflow.StarterPipeline\":\"TRACE\"}

The strange bit is doing this configuration for ERROR level seems to show the logs but anything lower doesnt.

Could anyone have come across this?

Here is the code segment I am using to try and output the logs

private static final Logger LOG = LoggerFactory.getLogger(StarterPipeline.class);

public static void main(String[] args) {
    Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());

    for (String string : args) {
        // System.out.println("########## "+string);
        LOG.trace("Doing the trace for pipeline ##### Parameter is #### " + string);

        LOG.trace("Doing the trace for pipeline ##### Parameter is #### ");
    }

    LOG.trace("Doing the trace for pipeline");
    p.apply(Create.of("Hello", "World")).apply(ParDo.of(new DoFn<String, String>() {
        @Override
        public void processElement(ProcessContext c) {
            LOG.trace("Doing the trace for pipeline");
            c.output(c.element().toUpperCase());
        }
    })).apply(ParDo.of(new DoFn<String, Void>() {
        @Override
        public void processElement(ProcessContext c) {
            LOG.trace("Doing the trace for pipeline");
            LOG.info(c.element());
        }
    }));

This is from the dataflow example except for the logging that I added to see if I am able to see any logs.

Japheth Odonya
  • 333
  • 4
  • 15
  • What PipelineRunner are you using? Where are you looking for the logs? Which log in specific are you looking for? Have you looked at the [DebuggingWordCount example](https://cloud.google.com/dataflow/examples/wordcount-example#debugging-wordcount-example) which talks about logging? – Ben Chambers Jun 01 '16 at 17:22
  • @BenChambers Thanks for your quick response. I am using the BlockingDataflowPipelineRunner , I am looking for the logs on the console (eclipse console) for the logs like LOG.trace("Doing the trace for pipeline ##### Parameter is #### "); which is a log line that I just added to see that something is being written on the console. But even when I look in the google cloud console under logs I am not able to see the logs I am looking for, infact in the log levels there is no TRACE – Japheth Odonya Jun 01 '16 at 20:47

1 Answers1

1

When running a pipeline using BlockingDataflowPipelineRunner or DataflowPipelineRunner there are two parts to execution. First, your main method is run locally on your machine to produce the pipeline. When you call p.run() the pipeline is shipped to the Dataflow service where it runs on (possibly many) virtual machines.

The log statements that are directly in the main method (such as "Doing the trace for pipeline ...") happen on your machine, and would show up in the Eclipse console. The log statements within the DoFns happen on the virtual machines, and only show up in Cloud Logging, as described in DebuggingWordCount example.

Specifying workerLogLevelOverrides affects the log level of the worker virtual machines. So, changing that value will affect what gets logged by the DoFns to Cloud Logging.

To adjust the log level of code running on your machine (so the main method) you need to adjust the SLF4J or java.util.logging log levels that the main program is run with. For example, these instructions on Stack Overflow suggest using a logging config file.

Community
  • 1
  • 1
Ben Chambers
  • 6,070
  • 11
  • 16
  • This is it, I was looking at the wrong place, I can actually see the logs in google cloud logs. One issue though is that there is no trace in the log levels in the filter section, I saw the trace logs showing up as DEBUG severity looks like that is google's intention but it works for now. Thanks. – Japheth Odonya Jun 02 '16 at 07:57