8

We can generate storm jar using particular parameter. However, if we need to debug this project (actually a far) locally as well as remotely?

If it is simple jar, that we can debug. However, here we are deploying jar using following command: storm jar project.jar main_class_name

Not sure how can we deploy storm topology, so that we can do storm project in debugging mode?

Please find updated yaml file as below:

 

    # Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    ########### These MUST be filled in for a storm configuration
    # storm.zookeeper.servers:
    #     - "server1"
    #     - "server2"
    # 
    # nimbus.host: "nimbus"
    # 
    # 
    # ##### These may optionally be filled in:
    #    
    ## List of custom serializations
    # topology.kryo.register:
    #     - org.mycompany.MyType
    #     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
    #
    ## List of custom kryo decorators
    # topology.kryo.decorators:
    #     - org.mycompany.MyDecorator
    #
    ## Locations of the drpc servers
    # drpc.servers:
    #     - "server1"
    #     - "server2"

    ## Metrics Consumers
    # topology.metrics.consumer.register:
    #   - class: "backtype.storm.metric.LoggingMetricsConsumer"
    #     parallelism.hint: 1
    #   - class: "org.mycompany.MyMetricsConsumer"
    #     parallelism.hint: 1
    #     argument:
    #       - endpoint: "metrics-collector.mycompany.org"

    worker.childopts:"-agentlib:jdwp=transport=dt_socket,server=y,address=8999,suspend=n"
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Jagrut Dalwadi
  • 295
  • 1
  • 4
  • 17

2 Answers2

10

If you want to remote debug, you need to enable debugging in worker JVMs. See here for the correct Java flags: Remote debugging a Java application

Add this flags in all/shared supervisor storm.yaml to entry worker.childopts (it might be necessary to restart Storm cluster). You need to make sure, that only a single worker is stared per host! Otherwise, two JVMs want to open the same port and one will fail of course.

In Eclipse, choose Run -> Debug Configuration and create a new Remote Java Application. Specify the host and port (port must be the same as specified in worker.childopts) of the running worker JVM you want to debug.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • Thank you. Let me try with this. – Jagrut Dalwadi Aug 27 '15 at 05:44
  • I am using default storm.yaml file coming into installation. For adding required parameters into this, I have added below parameter: worker.childopts:"java -agentlib:jdwp=transport=dt_socket,server=y,address=8999,suspend=n -jar application.jar" – Jagrut Dalwadi Aug 27 '15 at 07:45
  • This will lead to following error: >Exception in thread "main" ja va.lang.ExceptionInInitializerError Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java .util.Map at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:141) at backtype.storm.utils.Utils.readStormConfig(Utils.java:188) at backtype.storm.utils.Utils.(Utils.java:71) ... 57 more Could not find the main class: backtype.storm.daemon.nimbus. Program will exit. – Jagrut Dalwadi Aug 27 '15 at 07:49
  • I think I need to apply any other settings as well. – Jagrut Dalwadi Aug 27 '15 at 07:50
  • You only need to include the options without "java" and "-jar application.jar": `worker.childopts:"-agentlib:jdwp=transport=dt_socket,server=y,address=8999,suspend=n"` – Matthias J. Sax Aug 27 '15 at 08:01
  • I have tried with suggestion, same error is comming. – Jagrut Dalwadi Aug 27 '15 at 10:34
  • Not sure any other parameter do I need to set or not? – Jagrut Dalwadi Aug 27 '15 at 11:17
  • Can you post your yaml file (edit your question and add it there). It seems there is some formatting problem. – Matthias J. Sax Aug 27 '15 at 12:19
  • Please find yaml file. – Jagrut Dalwadi Aug 27 '15 at 12:41
  • It might sound stupid, but maybe you need a blank after the colon... worker.childopts: "-agentlib..." Otherwise, I cannot spot any problem in your yaml file. But the error shows, that the yaml cannot be parsed correctly. Thus, it's not a semantical problem of other missing parameters... – Matthias J. Sax Aug 27 '15 at 14:10
  • Can you maybe show the whole stack trace of the error. I am also wondering how the shown file can be the correct yaml file for the supervisor... It does not specify enough parameters for the supervisor to start correctly (for example, "nimbus.host" is put into comment). Where is this yaml file located? – Matthias J. Sax Aug 27 '15 at 14:15
  • That problem was with spaces. After that I have tried to debug, eclipse debug keeps waiting. But debug is coming in eclipse. I have put yaml file under location: apache-storm-0.9.3\conf. – Jagrut Dalwadi Aug 28 '15 at 11:05
  • "But debug is coming in eclipse" -> do you mean it does still not work? Is the yaml file only located at you local machine? You need to put it on all machines that run supervisors. – Matthias J. Sax Aug 28 '15 at 11:14
  • 1
    Now...Yeah. I am able to dubug it via eclipse. It seems two problems:1. nimbus, supervisor, ui were not running properly, as firing command one after another. I have kept 15seconds delay in execution. 2. when toplogy has deployed and after that if I am not starting debugging instantly, connection refused error was coming. I am able to resolve it. Thank you very much Matthias for helping in this. Appreciated!!!! – Jagrut Dalwadi Aug 28 '15 at 13:28
  • @MatthiasJ.Sax I have a similar question, If you have any ideas, can you please advise? http://stackoverflow.com/questions/32405136/how-to-debug-apache-storm-in-local-cluster-mode-through-eclipse – Anil Sep 04 '15 at 19:25
  • For me it shows connectionRefused error . Jagrut..if possible can u elaborate the steps in bit more detail. I am deploying the jar and then starting the application in debug mode using the debug configurations specified above . But it always show ConnectionRefused Error – Ankur Garg Oct 19 '15 at 17:51
  • Eclipse shows connection refused? Sounds like a firewall problem on the debug port. – Matthias J. Sax Oct 19 '15 at 17:59
  • No Firewall issue ..still :( ..do we have to give location of jar file as well? – Ankur Garg Oct 19 '15 at 18:09
  • Can you check the worker node if the worker JVM is running and if the port got opened... – Matthias J. Sax Oct 19 '15 at 18:13
  • Can anyone put the detailed steps here. I am a newbie and facing issues in understanding the steps to debug. – DebD Nov 01 '16 at 06:54
  • Did you read the "Remote Debugging" link from the answer? Do you have problem with remote debugging general, or just in combination with Storm? What did you try? What does not work? Maybe it's worth starting a new question? – Matthias J. Sax Nov 01 '16 at 07:00
5

To enable debugging across the various supervisor VMs, all you have to do is edit the storm.yaml file and update the worker.childopts entry to something like:

worker.childopts: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5%ID%"

If you had your supervisors on ports 6700,6701,6702 etc., this would create a debug port on 56700,56701,56702 etc.

Jason K.
  • 790
  • 1
  • 14
  • 22