Apache Storm : storm-kafka-monitor script throws exception

Question

I have set up an Apache Storm cluster locally and submitted a topology as well which is visible in the Storm UI. Once everything is up and running I am getting the following exception in ui.log file. The error is somewhat related to storm-kafka-monitor which is a bash script and I am running this on windows.
Any help will be highly appreciated. Thanks in advance :)

2018-06-05 11:26:20.293 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception thrown while getting lag for spout id: kafka-spout and spout class: org.apache.storm.kafka.KafkaSpout
2018-06-05 11:26:20.294 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception message:Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
java.io.IOException: Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_162]
    at org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:180) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.getLagResultForKafka(TopologySpoutLag.java:163) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.getLagResultForOldKafkaSpout(TopologySpoutLag.java:193) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.lag(TopologySpoutLag.java:59) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$topology_lag.invoke(core.clj:805) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$fn__9586.invoke(core.clj:1165) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$make_route$fn__5979.invoke(core.clj:100) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$if_route$fn__5967.invoke(core.clj:46) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$if_method$fn__5960.invoke(core.clj:31) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$routing$fn__5985.invoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
    at clojure.core$some.invoke(core.clj:2570) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
    at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[clojure-1.7.0.jar:?]
    at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.shade.compojure.core$routes$fn__5989.invoke(core.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__8894.invoke(cors.clj:149) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8841.invoke(json.clj:56) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7904.invoke(reload.clj:22) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.helpers$requests_middleware$fn__6874.invoke(helpers.clj:52) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$catch_errors$fn__9761.invoke(core.clj:1428) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__6541.invoke(keyword_params.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__6584.invoke(nested_params.clj:84) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__6513.invoke(params.clj:64) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__6836.invoke(flash.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__6822.invoke(session.clj:98) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__6371.invoke(servlet.clj:127) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet$servlet$fn__6375.invoke(servlet.clj:136) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
    at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__6967.invoke(helpers.clj:192) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) ~[storm-core-1.2.1.jar:1.2.1]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
    at java.lang.ProcessImpl.create(Native Method) ~[?:1.8.0_162]
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) ~[?:1.8.0_162]
    at java.lang.ProcessImpl.start(ProcessImpl.java:137) ~[?:1.8.0_162]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_162]
    ... 68 more

I think there is a mismatch with Microsoft OS 32/64-bit version and storm--kafka-monitor 32/64-bit version: https://stackoverflow.com/a/12637373/2987755 — dkb, Jun 05 '18 at 07:36
Nope, no workaround, just check whether there is any mismatch with platform compatibility — dkb, Jun 05 '18 at 09:15
But how? The script is a bash file and i am running windows 10 — Abdul Samad, Jun 05 '18 at 09:23

score 1 · Accepted Answer · answered Jun 05 '18 at 15:14

1

As you note the script is a bash file. The feature (showing how far behind the log end offset the spout is in Storm UI) doesn't work on Windows currently. It's a little inconvenient, but it shouldn't prevent your topologies from working correctly.

If you want to know how far behind the log end offset the spout is, you can use the kafka-consumer-groups.bat script on one of your Kafka machines instead. If you're using the storm-kafka-client spout, the spout lag is also part of the metrics provided by that component.

I'm hoping we can get rid of storm-kafka-monitor at some point (it has some other issues https://issues.apache.org/jira/browse/STORM-2600), but until then, if you'd like to try providing Windows compatibility, you'd need to port https://github.com/apache/storm/blob/master/bin/storm-kafka-monitor to cmd/powershell, and modify https://github.com/apache/storm/blob/4137328b75c06771f84414c3c2113e2d1c757c08/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L146 so it calls either the Windows or bash script depending on OS.

answered Jun 05 '18 at 15:14

Stig Rohde Døssing

3,621
2
7
7

Thankyou @stig for your answer. Does it cause any problems on linux? – Abdul Samad Jun 06 '18 at 06:02
and what is better to use `storm-kafka` or `storm-kafka-client`? I am currently using `storm-kafka 1.2.1`. thanks – Abdul Samad Jun 06 '18 at 07:31
It doesn't cause problems on linux, beyond those listed in the issue I linked. `storm-kafka-client` is based on a newer Kafka client library, you should migrate to that one if you can. – Stig Rohde Døssing Jun 06 '18 at 14:32
thank you very much :). I have one more thing to ask how can we restart our remote zookeeper instance? I am having connection timeout issues. /Zknodes dir needs to be removed i guess. Any help will be appreciated – Abdul Samad Jun 07 '18 at 06:24
1

I'm not entirely clear on what you're asking. Consider opening a new question so other people can find it (and so you can elaborate). If you're asking how to reset Zookeeper (delete the contents), look at https://stackoverflow.com/questions/35154147/zookeeper-data-directory-cleanup. This is not reversible! – Stig Rohde Døssing Jun 07 '18 at 17:24

Apache Storm : storm-kafka-monitor script throws exception

1 Answers1