2

I have successfully upgraded Gitlab to 12.1.6 on Ubuntu 16.04 and checked that all was working. After making sure Ubuntu was fully up to date I checked again: Gitlab worked. I then used the do-release-upgrade command to update to Ubuntu 18.04. After restart, everything seems to work ok, but Gitlab refuses to start.

I get the following errors:

fail: alertmanager: runsv not running
fail: gitaly: runsv not running
fail: gitlab-exporter: runsv not running
fail: gitlab-workhorse: runsv not running
fail: grafana: runsv not running
fail: logrotate: runsv not running
fail: nginx: runsv not running
fail: node-exporter: runsv not running
fail: postgres-exporter: runsv not running
fail: postgresql: runsv not running
fail: prometheus: runsv not running
fail: redis: runsv not running
fail: redis-exporter: runsv not running
fail: sidekiq: runsv not running
fail: unicorn: runsv not running

I tried:

gitlab-ctl reconfigure --> runs with success

I installed runit with success, rebooted the machine but the errors remain

I found a similar issue here: on Stackoverflow, followed the instructions (yum --> apt), still no success

and here on Gitlab. This advised to run

sudo systemctl restart gitlab-runsvdir
sudo gitlab-ctl restart

But the first command never finishes

I found this: on Gitlab which states to run 

sudo gitlab-rake gitlab:env:info --trace

Output:

** Invoke gitlab:env:info (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:env:info

System information
System:     Ubuntu 18.04
Current User:   git
Using RVM:  no
Ruby Version:   2.6.3p62
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  3.2.12
Git Version:    2.24.1
Sidekiq Version:5.2.7
Go Version: unknown
rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.1.4/lib/pg.rb:56:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.1.4/lib/pg.rb:56:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/pg-1.1.4/lib/pg.rb:56:in `connect'

which suggests that the Postgresql server is not running. I have no idea how to start it. Any ideas?

hepabolu
  • 1,214
  • 4
  • 18
  • 29

3 Answers3

3

UPDATE: Since then I found an answer that said for some reason on the reboot command the runsv process gets stuck. To solve this you have to:

  • sudo service gitlab-runsvdir start
  • sudo gitlab-ctl reconfigure
  • sudo gitlab-ctl restart

I wanted to post the same above question. This happens to me when i do the monthly upgrade/updates to our GitLab. I tried your solution before, but to me they didn't solve the problem.

I present my solution, but use these tips with caution, as I still sometimes struggle with this problem in different ways.

I do a combination of these command, the order of these steps are really important(!):

  • sudo systemctl disable gitlab-runsvdir.service
  • sudo gitlab-ctl reconfigure
  • sudo gitlab-rake db:migrate
  • sudo gitlab-ctl restart gitlab-runsvdir--has to be aborted with Ctrl-C
  • sudo /opt/gitlab/embedded/bin/runsvdir-start--has to be aborted with Ctrl-C

Reference: In this issue Stan Hu's answer helped me: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3744

Now yesterday I messed up and didn't do the first step, which could cause my problems, because somehow then for 10 hours the postgresql server was down and it couldn't make it back up again.

If i would have stick to the order, maybe it wouldn't have happened, because it didn't happen in the past while I upgraded/updated GitLab several times.

This was the connection error yesterday, same issue as questioner's

Automatic restart this morning

This morning magically the server at 7:13 am, was accepting connections again. (althought i tried to restart it yesterday, it didn't make any difference then) Still GitLab wasn't reachable online from web browser as of 7:30 am. One hour later, and a few runsvdir restarting/starting commands after (added it to the commands order), the GitLab is working. I have no idea why.

Viktória
  • 46
  • 5
1

Got the same problem, and I ended up running /opt/gitlab/embedded/bin/runsvdir-start manually. I fixed my problem, launched reconfigure, and then it worked from there.

MappaM
  • 809
  • 8
  • 15
0

I ran into a similar runsv error, but only saw it for once service, not the whole list you have. These steps are a log of my attempts to get it working - probably not a direct line, but my local Gitlab does work now:

In the CentOS vm:

  • vi /etc/gitlab/gitlab.rb

  • sudo gitlab-ctl reconfigure

    • first observed the error runsv not running

  • yum update -y

  • sudo gitlab-ctl status

  • sudo gitlab-ctl restart

  • sudo gitlab-ctl reconfigure

  • systemctl start gitlab-runsvdir.service

  • systemctl status gitlab-runsvdir.service

  • sudo gitlab-ctl reconfigure

    • still saw an error about runsv not running, several times, but it was never a blocker and the reconfigure was successful

On host

  • navigate to 192.168.1.131

  • See the prompt for root password

As for the issue with Postgres, I'm not sure

Dan Bitter
  • 229
  • 4
  • 12