executing kafka sh script remotely from java program

Question

I am trying to execute a kafka script to retrieve the topic, consumergroups and lag info. I keep getting the error and searching through this and other forums reveals conflicting information. Some say it is not possible to execute a remote script on Unix from windows while others give some advice on what to try to correct this error. I am able to connect and run a simple ping command and able to retrieve the response. Perhaps I am missing a simple overlooked error here.

Here is the code:

         try {
            jsch.setKnownHosts("C:\\Users\\.ssh\\ssh_host_rsa_key.pub");
            Session session = jsch.getSession(uname, host, 22);

            Properties config = new Properties();
            config.put("StrictHostKeyChecking", "no");

            session.setConfig(config);
            session.setPassword(pword);
            session.connect();

            Process p = Runtime.getRuntime().exec("./usr/kafka/bin/
             kafka-consumer-groups.sh --bootstrap-server 192.xxx.xx.xxx:9092 
                  --describe -group OPSII");

            InputStream scriptStdout = p.getInputStream();
            BufferedReader scriptReader= new BufferedReader(new 
                InputStreamReader(scriptStdout));
            String scriptOutput = scriptReader.readLine();
            StringBuilder sb = new StringBuilder();

            while ((scriptOutput = scriptReader.readLine())!= null) {
                sb.append(scriptOutput + "\n");
            }

            scriptStdout.close();

The error:

Exception in thread "main" java.io.IOException: Cannot run program 
 "./usr/kafka/bin/kafka-consumer-groups.sh": CreateProcess 
   error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)

I have confirmed that the script works on the remote unix machine and that the directory is correct. Is it perhaps the format and should it be '//' instead of '/'? What exactly would cause this error? Please this is not a duplicate question as none of the other proposed solutions worked.

Try a `cd` to that directory as a means of debugging. After that try running your script. — Nicholas K, Sep 24 '18 at 11:43
check if your user has execute permission on the directory... — Nishu Tayal, Sep 24 '18 at 11:52
when I try cd I get the following: Cannot run program "cd": ....why would 'ping www.google.com' work then? — vbNewbie, Sep 24 '18 at 11:53
`cd` was for changing the directory. I've added an answer, please try to see if that works. — Nicholas K, Sep 24 '18 at 12:24
Why do you need a remote script at all?? That consumer group command can be written **in Java**. All the Kafka shell scripts are actually Java classes themselves — OneCricketeer, Sep 24 '18 at 13:36
@cricket_007 - I have no idea how to implement that. I want to obtain lag information and tried accessing jmx metrics but had too many errors with that. this seemed like the easiest option. Do you have any links to examples? — vbNewbie, Sep 24 '18 at 14:10
You want me to link to the source code for the `kafka-consumer-groups` shell script? Because that's the only example I have... Also, lag metrics can be gathered by tools like Burrow by LinkedIn or Confluent Control Center. It doesn't need to be done "by hand", and neither of those tools use JMX to extract that data — OneCricketeer, Sep 24 '18 at 14:24
@cricket_007 I know about Burrow but am required to build our own for reasons I cannot fathom. But you said the script or consumer group command can be called directly in java & that's what I am asking for guidance with is that? Are you saying I should look at the source of these scripts and this can be run in java? either way I would appreciate some eg or advice on how to do that. Thanks for your input here. — vbNewbie, Sep 24 '18 at 14:43
@cricket_007; what I am doing now is executing the script, getting the output and parsing it, then creating json with the data I need and sending that to influx where this data is readily viewable. I need to first get the list of topics and consumer groups on each server the script will connect to. — vbNewbie, Sep 24 '18 at 14:44
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/180677/discussion-between-vbnewbie-and-cricket-007). — vbNewbie, Sep 24 '18 at 14:58
"*on each server*" doesn't make sense because this command doesn't need to run on a specific server. If you point at any Kafka broker or Zookeeper server, you will get the exact same information. [We've already linked this to you before](https://stackoverflow.com/a/52278961/2308683)... [`AdminClient#listConsumerGroupOffsets`](http://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/AdminClient.html#listConsumerGroupOffsets-java.lang.String-org.apache.kafka.clients.admin.ListConsumerGroupOffsetsOptions-), and I already linked you the source code in other question — OneCricketeer, Sep 24 '18 at 18:40

Nicholas K · Accepted Answer · 2018-09-24T13:27:45.593

1

You can use the following code to run a script via JSch

    // open a channel
    channel = session.openChannel("exec");
    // type in your command
    String command = "./path/to/your_script.sh --add_params \n";
    //Below command will execute the data you set in the previous line
    ((ChannelExec) channel).setCommand(command);
    channel.connect();

Note this is using the JSch library only.

EDIT: As per your comment you want to fetch the output from the console as well for that use :

InputStream in = channel.getInputStream();

edited Sep 24 '18 at 13:27

answered Sep 24 '18 at 12:16

Nicholas K

15,148
7
31
57

Thanks Nicholas. Although this did not produce an error, after the connect statement I have the following: channel.getInputStream() to get the console result but it returns null. could you please advise how you would do that. – vbNewbie Sep 24 '18 at 12:36
Did the script run? Add this line `InputStream in = channel.getInputStream();` before `channel.connect();` and try – Nicholas K Sep 24 '18 at 12:44

executing kafka sh script remotely from java program

1 Answers1