How to start the upstart process for golang application?

Question

I am using upstart to start my golang application. I have my application folder structure like this,

   web-app/
         /app
             main.go

I built the application as below,

$cd /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app/
$go build ./...

It generated the binary app

And placed the web-app.conf in /etc/init/ folder. Here is the web-app.conf content,

#Web app upstart script
description "start and stop web app"

start on (net-device-up
and local-filesystems
and runlevel [2345])

stop on runlevel [016]

respawn
respawn limit 5 30

console output

script
    chdir /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app
    exec ./app
end script

When I tried sudo initctl list, it lists the process as stop/waiting. And I tried to start the process

$sudo initctl start web-app

It shows the process as start/running. But it is not started.

I checked the /var/log/messages logs. It shows,

init: web-app main process (18740) terminated with status 127

I couldn't start the process. I think there is some issue with the chdir. I tried different options for past two days. And I am fairly new to upstart but no luck. Could someone help me with this?

What are you trying to execute exactly? app is a directory isn't it? take a look at this question http://stackoverflow.com/questions/14692843/running-node-js-server-using-upstart-causes-terminated-with-status-127-on-ubu — Lital Kolog, Jul 04 '16 at 06:04
ah that was a typo. Thanks for finding that. I fixed that and tried to start. But still facing the same issue. — Dany, Jul 04 '16 at 06:05
@Lital - When I build the application it generates the binaries in the name app. [I think this is specific to golang] — Dany, Jul 04 '16 at 06:14
@Dany make sure the user who runs the upstart script has permissions to execute app. — Lital Kolog, Jul 04 '16 at 06:17
@Lital - I run as root. I am starting the process from ec2 user data. — Dany, Jul 04 '16 at 06:19
will it run with full path in the exec command (in addition to chdir)? — Mark, Jul 04 '16 at 06:26
@Mark - Running outside of the directory with full path is not working. For example, `$exec /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app/app` the above statement will throw error. But the following will work `$cd /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app $exec app` — Dany, Jul 04 '16 at 07:02
Did you solve this? `status 127` means the binary wasn't found: either doesn't exist, was misspelt, or was invoked with `app` rather than `./app`. The `chdir` is ok otherwise you'd get a different error. — Mark, Jul 04 '16 at 22:25
Note (on ubuntu) if `console output` is changed to `console log` in the upstart script, the stdout and stderr of the script and application will be logged to `/var/log/upstart/web-app.log`. This log may help explain why it's failing. — Mark, Jul 04 '16 at 22:31
@Mark - I tried `chdir /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app exec app` But didn't help. I am not using ubuntu. I am using amazon linux in which I dont find `/var/log/upstart` logs. — Dany, Jul 05 '16 at 00:45
Try removing the `script` and `end script` (currently, ec2 uses an old version of upstart, and it might not handle the nesting). — Mark, Jul 05 '16 at 02:31
@Mark - Thanks for helping me. I tried it with the following, `export GOPATH="/home/ec2-user/go" export GOROOT="/home/ec2-user/go" export APP_STAGE="prod" chdir /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app exec go run server.go` also I tried `exec app` In both cases application failed to start. — Dany, Jul 05 '16 at 03:46
GOPATH and GOROOT are only needed for compiling. Did you try removing the `script` and `end script` lines? I spun up a new aws linux instance with your web-app.conf (identical except script/end script removed), and started a small go web server, and it's still up and running. With `script/end script` the logs were showing status 127 errors, as you saw. — Mark, Jul 05 '16 at 04:07
@Mark - I really appreciate your help. I removed script and end script again and ran. When I used `exec app` it showed `initctl: Job failed to start`. I changed to `exec ./app` it shows `process (18247) terminated with status 1` in the logs. :( — Dany, Jul 05 '16 at 04:55
`terminated with status 1` suggests upstart is running your app, good! The app is crashing though. To verify the app is now the problem, temporarily swap `exec ./app` with a basic program you know to be working, eg, `exec sleep 2`. Restarting the job should result in /var/log/messages with several `web-app main process ended, respawning` lines. — Mark, Jul 05 '16 at 05:40
@Mark - Great catch. Thank you so much. I found the root cause with the technique you suggested. I use an environment variable APP_STAGE in web-app app server. Upstart is not taking this environment variable from console. So I updated the script `env APP_STAGE="prod" chdir /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app exec ./app` Finally it worked. I really appreciate your help. If you could add this as answer, I will accept. — Dany, Jul 05 '16 at 09:07

score 2 · Accepted Answer · answered Jul 06 '16 at 00:26

OP eventually solved after fixing a few issues. See comments, notably:

upstart might not recognise environment variables
Amazon Linux Image currently uses an old version of init (upstart 0.6.5), lacking newer features such as console log & nested script tags
status 127 can occur if exec can't find the binary
status 1 can occur if binary runs but fails
substituting a simple program in an upstart script can help diagnose errors

How to start the upstart process for golang application?

1 Answers1