Run a shell script from my startup script

Question

I'm developing an app that uses Kafka and I need a little help in setting up an aws server where I want a script to run when I boot the machine. I've created a script here:

#!/bin/sh
sh /home/ec2-user/kafka_2.13-3.4.0/bin/zookeeper-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/zookeeper.properties &
sh /home/ec2-user/kafka_2.13-3.4.0/bin/kafka-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/server0.properties &
sh /home/ec2-user/kafka_2.13-3.4.0/bin/kafka-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/server1.properties &

cd /home/ec2-user/reimagined-guide/server/auth
/usr/bin/npm run pm2
cd /home/ec2-user/reimagined-guide/server/task
/usr/bin/npm run pm2
cd /home/ec2-user/reimagined-guide/server/bidding
/usr/bin/npm run pm2
cd /home/ec2-user/reimagined-guide/server/file-server
/usr/bin/npm run pm2

sudo systemctl start nginx

Everything works except the lines 2-4, which are related to Kafka (and which are all shell scripts). I need some help to configure this properly so my zookeeper and Kafka servers start when the machine turns on. Running these commands in the terminal works, but it doesn't work through the script.

What I've tried: Made all the files in the Kafka folder executable

sh /home/ec2-user/kafka_2.13-3.4.0/bin/zookeeper-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/zookeeper.properties

cd /home/ec2-user/kafka_2.13-3.4.0/
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

/bin/bash/ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

None of these have worked though. Like I mentioned before, I can run these on the terminal and they work fine, but not in my startup shell script. All these three were either from ChatGPT or from this websites posts.

I mean that my Kafka service, which runs when I input lines 2-4 manually on my shell, work. But when I try to implement it with a shell script, the service is not running. I can check this - when I run the Kafka-server-stop, it says that there's nothing running to stop. — flimsy, Apr 10 '23 at 13:32
Ideally I want my Kafka and zookeeper services up and running for the nodejs apps, but that doesn't happen with the script. — flimsy, Apr 10 '23 at 13:33
BTW, `sh /path/to/something.sh` is actually a bad practice: it ignores that shell script's first line, which might specify something like `#!/bin/bash` or `#!/bin/zsh` or `#!/bin/ksh` instead of `/bin/sh`, and _forces_ `sh` to be used instead of the shell the script was actually written for. (For the same reason, putting `.sh` extensions on shell scripts is _also_ a bad practice, because it encourages people to use `sh` as an interpreter even for shell scripts that aren't actually written for `sh`). — Charles Duffy, Apr 10 '23 at 14:31
To enable trace logging, run `sh -x yourscript`. If you compare the trace log emitted by the original/broken version of your script and the version adjusted per your answer, the difference is likely to explain _why_ the quotes were needed and thus why that answer works. — Charles Duffy, Apr 10 '23 at 14:34
(BTW, earlier I said "sh /path/to/script.sh" is bad practice, but I didn't explain what to do instead: Just run `/path/to/script.sh` with no `sh` before it; that does require your file permissions to be right, but if they aren't, it's better to fix that directly than to work around it in a way that causes other bugs). — Charles Duffy, Apr 10 '23 at 14:36

flimsy · Answer 1 · 2023-04-10T14:26:51.307

0

I don't really understand how this works but it does:

sh $HOME/kafka_2.13-3.4.0/bin/zookeeper-server-start.sh "-daemon" "./$HOME/kafka_2.13-3.4.0/config/zookeeper.properties" &
sh $HOME/kafka_2.13-3.4.0/bin/kafka-server-start.sh "-daemon" "./$HOME/kafka_2.13-3.4.0/config/server0.properties" &
sh $HOME/kafka_2.13-3.4.0/bin/kafka-server-start.sh "-daemon" "./$HOME/kafka_2.13-3.4.0/config/server1.properties" &

Passing the arguments in quotations did the trick

edited Apr 10 '23 at 14:26

answered Apr 10 '23 at 14:25

flimsy

1
1

Any chance your `HOME` directory name contains whitespace? That would explain why this made a difference (mostly; but if that was needed you'd need the first `$HOME` to be in quotes too) – Charles Duffy Apr 10 '23 at 14:32
My $HOME dir doesn't have any spaces, just check it. I was using it by the absolute path typed out rather than $HOME before. I saw this post: https://stackoverflow.com/questions/16988427/calling-one-bash-script-from-another-script-passing-it-arguments-with-quotes-and – flimsy Apr 10 '23 at 14:34
1

Hmm. `./$HOME` is actually a bit interesting: normally it would just be `$HOME/` without any leading `./` to be correct. Unless this "startup script" is executing in a context where HOME is empty... this is a place where `set -x` logs might be interesting. – Charles Duffy Apr 10 '23 at 14:40

Marcelo Guedes · Answer 2 · 2023-04-12T01:08:20.530

let's consider that you use a Linux system with systemd services mananger.
You can run "systemctl --no-pager" to make sure.

1º) locate the service name of kafka and zookeeper:

systemctl --no-pager | grep -Ei "kafka|zookeeper"

2º) get the name of kafka service, and locate the service file of kafka:

systemctl show -p FragmentPath <KAFKA-SERVICE-NAME>

3º) get the "FragmentPath" value and do a copy with another name:

cd /usr/lib/systemd/system/
cp /usr/lib/systemd/system/<KAFKA-SERVICE> /usr/lib/systemd/system/YOUR-SERVICE-NAME.service

4º) edit your new service. Clear all content, and now remember that correctly configure the After, Wants and Requires sessions, it will ensure that your script will run after this services and require that they are up:

[Unit]
Description=My powerful script
After=<KAFKA-SERVICE-NAME> <ZOOKEEPER-SERVICE-NAME>
Wants=<KAFKA-SERVICE-NAME> <ZOOKEEPER-SERVICE-NAME>
Requires=<KAFKA-SERVICE-NAME> <ZOOKEEPER-SERVICE-NAME>

[Service]
Type=oneshot
ExecStart=/usr/bin/YOUR-SCRIPT.sh -s -q --timeout=10
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

OBS:
after: start order
wants: desire start order
Requires: Need to start

6º) now, put your script in /usr/bin/YOUR-SCRIPT.sh (for example), and change perms:

chmod 740 /usr/bin/YOUR-SCRIPT.sh

7º) enable your new service:

systemctl enable YOUR-SERVICE-NAME.service

You will see a message like:

Created symlink from /etc/systemd/system/multi-user.target.wants/YOUR-SERVICE-NAME.service to /etc/systemd/system/YOUR-SERVICE-NAME.service

8º) restart the system, if u can, and verify now.

this answer does not follow the best practices or the best parameters, it is simple and functional.

score -1 · Answer 3 · edited Apr 11 '23 at 00:32

Use the following script.

#!/bin/sh

# Start ZooKeeper and Kafka servers in the background
sh /home/ec2-user/kafka_2.13-3.4.0/bin/zookeeper-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/zookeeper.properties &&
sh /home/ec2-user/kafka_2.13-3.4.0/bin/kafka-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/server0.properties &&
sh /home/ec2-user/kafka_2.13-3.4.0/bin/kafka-server-start.sh -daemon /home/ec2-user/kafka_2.13-3.4.0/config/server1.properties &&
echo "ZooKeeper and Kafka servers started successfully"

# Define an array of Node.js application directories
apps=(
  "/home/ec2-user/reimagined-guide/server/auth"
  "/home/ec2-user/reimagined-guide/server/task"
  "/home/ec2-user/reimagined-guide/server/bidding"
  "/home/ec2-user/reimagined-guide/server/file-server"
)

# Loop through the application directories and start each application with PM2
for app in "${apps[@]}"
do
  cd "$app" && /usr/bin/npm run pm2
  if [ $? -ne 0 ]
  then
    echo "Failed to start $app with PM2"
    exit 1
  fi
done

# Start Nginx server
sudo systemctl start nginx
if [ $? -ne 0 ]
then
  echo "Failed to start Nginx server"
  exit 1
fi

echo "All services started successfully"

I tried your code, I'm getting a log that the services for Kafka have been started, but when I check if they're running, they are not. Also I got an error in the later lines. — flimsy, Apr 10 '23 at 12:44
As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). — Community, Apr 17 '23 at 18:11

Run a shell script from my startup script

3 Answers3