I am trying to deploy a storm topology to a remote machine using the storm jar command on Windows
storm jar jarName.jar mainClass
Here is what I have in my storm.yaml file:
storm.zookeeper.servers:
- "HostName"
nimbus.host: "HostName"
nimbus.thrift.port: 6627
But whenever I try to deploy, it always tries to deploy to my local machine.
EDIT
I tried putting the following configurations but it does not work:
conf.put(Config.NIMBUS_HOST, "HostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"HostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
I also tried the -c option on the command line as:
storm -c nimbus.host=HostName jar jarName.jar mainClass
But it does not work; I get:
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I tried placing the -c part at different places but nothing seems to work.
Also, I get
The field Config.NIMBUS_HOST is deprecated
when I set:
conf.put(Config.NIMBUS_HOST, "HostName");
I am using Storm version 0.10.0-beta
EDIT2
Submitting works perfectly fine from IDE after I use
System.setProperty("storm.jar", "path to jar");
however, it does not work from command line.
Edit3
Thanks Matt for all your help. I was able to deploy the topology using the java command that eclipse executes. I am still unable to deploy using the storm jar command however. It complains that the topology already exists even though i dont see anything on the Storm UI. Also, when i deploy using Eclipse or the Java command from command line, the topology deploys successfully. How should i compare the difference between executing the java command and what the storm jar command is doing? i tried to echo the variables in this section of the cmd file
if %storm-command% == jar (
set STORM_OPTS=%STORM_CLIENT_OPTS% %STORM_OPTS% -Dstorm.jar=%2
echo %STORM_OPTS%
set CLASSPATH=%CLASSPATH%;%2
echo %CLASSPATH%
set CLASS=%3
set args=%4
goto start
:start
shift
if [%4] == [] goto done
set args=%args% %4
echo args: %args%
goto start
:done
echo %args%
set storm-command-arguments=%args%
)
but i dont see anything unusual. Also, i see the following exception in the stacktrace
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
notice
topologies:[]
and then it complains that the topology name exists
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at mainClass.submitTopology(mainClass.java:127)
at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
finally, here is the entire stacktrace.
X:\abc\xyz\target>ERROR StatusLogger Invalid URL C:/asd/ewt/apache-storm 0.10.0-beta/log4j2/cluster.xml java.net.MalformedURLException: unknown protocol: c
at java.net.URL.<init>(URL.java:593)
at java.net.URL.<init>(URL.java:483)
at java.net.URL.<init>(URL.java:432)
at java.net.URI.toURL(URI.java:1089)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromUri(ConfigurationFactory.java:296)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:395)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:254)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:419)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
at org.apache.log4j.Category.<init>(Category.java:56)
at org.apache.log4j.Logger.<init>(Logger.java:35)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:59)
at org.apache.log4j.Logger.getLogger(Logger.java:39)
at mainClass.<clinit>(mainClass.java:32)
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field nimbus_uptime_secs' is unset!Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at mainClass.submitTopology(mainClass.java:127)
at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
and by the way what is that weird logging exception/error at the top of the stacktrace? could that have something to do with the weird behavior of storm jar command?
if i do
java -cp jarName.jar;C:\abc\asdasd\apache-storm-0.10.0-beta\lib\* mainClass
i get
513 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
582 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
590 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD
5-digest: -6100635753051151148:-8346710213172614614
591 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds []
599 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
626 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protoc
ol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at com.rbccm.th2015.storm.SampleTopologyX.submitTopology(SampleTopologyX.java:127)
at com.rbccm.th2015.storm.SampleTopologyX.main(SampleTopologyX.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName,uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
so its complaining about the same thing but via a different route. could this have something to do with my YAML? and what is this business about nimbus_uptime_secs?
Final Edit
ok, time to put this to rest. For now at least, it does not look like the sotrm jar command is suitable for windows. I can clearly see a difference in the command that storm.py tries to execute in Windows and the one that it tries to execute in Linux. Basically, the paths get screwed up in windows and that must be what's causing all of these weird exceptions.
Thanks Matt for all of your help and support. I really learned a lot.