1

Setup:

  • Storm 0.10.0
  • Streamparse 2.1.4
  • Centos 6.5
  • Python 2.7 (Streamparse needs it)

(Yes i know they are outdated, however i couldnt get anything working with Storm 1.0, its just broken with streamparse 3)

When I attempt to launch a "streamparse submit" from either my nimbus server, or another server in my topology I get the following error:

"IOError: Local port: 6627 already in use, unable to open ssh tunnel to nimbus.server.local:6627."

But ofcourse 6627 is in use on my nimbus server? Its the Thrify port. So i tried moving the Thrifty port to 6637 and restarting Nimbus. But I get the same error back from the client submitting it:

IOError: Local port: 6627 already in use, unable to open ssh tunnel to nimbus.server.local:6627.

Even a netstat tuanp shows that 6627 shows that nothing is listening on that port on nimbus or the box executing the submit.

I have a feeling something to do with SSHD config and allowing tunneling, and that isn't being handled properly by Nimbus and giving an incorrect error when trying to establish the tunnel.

Has anyone else experienced this?

Adam Bradbury
  • 93
  • 2
  • 9
  • Yes, I am seeing the same issue when running streamparse on the host where I set up a local Storm cluster. Before that I spent an entire day trying to deploy storm topology, which I already tested with "sparse run", to a remote Storm cluster without success. What I learned is that every time you run sparse command it spawns two sparse processes. I am assuming both of them are using port 6627 causing "6627 already in use" error to be thrown by one of the processes. – alecswan May 01 '16 at 18:09

2 Answers2

0

This is what I ended up doing to deploy streamparse Storm topology in the local Storm cluster:

> sparse quickstart quickstart-2.1.4
> cd quickstart-2.1.4
> sparse jar
> storm jar _build/quickstart-2.1.4-0.0.1-SNAPSHOT-standalone.jar streamparse.commands.submit_topology topologies/wordcount.clj

This worked with streamparse 2.1.4 and Storm 0.9.5

alecswan
  • 3,670
  • 5
  • 25
  • 35
0

I got the same error when running storm topology.

I made following changes, then it worked fine, Added the following property

In config.json,

        "use_ssh_for_nimbus": false,
        "use_virtualenv": false,

In fabfile.py,

        from fabric.api import env
        env.use_ssh_config = False
        env.password = '****'
        from streamparse.ext.fabric import *

And submitted as "sparse submit" Please let me know, if it worked, or share the config file

Nirmal Ram
  • 1,180
  • 2
  • 9
  • 18