3

I am randomly getting the following error when I am executing the binary (sometime it works):

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7f07a019b448]

Here is the entire stacktrace.

The OS version where I am running the binary:

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Go version:

go version
go version go1.16.9 linux/amd64

Bazel options:

go_binary(
    name = "app-operator",
    embed = [":app-operator_lib"],
    gc_linkopts = [
        "-linkmode",
        "external",
        "-extldflags",
        "-static",
    ],
    static = "on",
    visibility = ["//visibility:public"],
)

I think it's happening in this goroutine:

goroutine 408 [select]:
net/http.(*Transport).getConn(0x22f49e0, 0xc00080b440, {{}, 0x0, {0x16482cc, 0x4}, {0xc0006d9590, 0x30}, 0x0})
    GOROOT/src/net/http/transport.go:1372 +0x5d2
net/http.(*Transport).roundTrip(0x22f49e0, 0xc0005b0400)
    GOROOT/src/net/http/transport.go:581 +0x774
net/http.(*Transport).RoundTrip(0x30, 0x1817c80)
    GOROOT/src/net/http/roundtrip.go:18 +0x19
net/http.send(0xc0005b0400, {0x1817c80, 0x22f49e0}, {0x15f8740, 0x201, 0x0})
    GOROOT/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0x2382ea0, 0xc0005b0400, {0x380, 0x0, 0x0})
    GOROOT/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0x2382ea0, 0xc0005b0400)
    GOROOT/src/net/http/client.go:725 +0x908
net/http.(*Client).Do(...)
    GOROOT/src/net/http/client.go:593
somecompany.com/operators/app-operator/http/client.(*APIClient).callAPI(0xc00078e030, 0xc0005b0400)
    operators/app-operator/http/client/client.go:151 +0x12e
somecompany.com/operators/app-operator/http/client.(*BackuprestoreApiService).Poll(0xc0004de148, 0xc00016b8a8)
    operators/app-operator/http/client/api_backuprestore.go:216 +0x5bd
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Poll(0xc0004de120, {0xc00076a4b0, {0xc00021c268, 0xc00016ba38}, 0xc00076a4c0})
    operators/app-operator/controllers/backup_controller.go:165 +0x16d
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Poll(0xc0004de120, {0xc00076a4b0, {0xc00021c268, 0x4f416a}, 0xc00076a4c0})
    operators/app-operator/controllers/backup_controller.go:186 +0x396
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Poll(0xc0004de120, {0xc00076a4b0, {0xc00021c268, 0x4f4226}, 0xc00076a4c0})
    operators/app-operator/controllers/backup_controller.go:186 +0x396
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Backup(0xc0004de120, 0xc0006ab4a0)
    operators/app-operator/controllers/backup_controller.go:147 +0x396
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).createBackupHandler(0xc00016bd78, 0x183e758)
    operators/app-operator/controllers/backup_controller.go:93 +0x1d
somecompany.com/operators/app-operator/controllers.(*BackupReconciler).Reconcile(0xc0004de120, {0x183e758, 0xc000772d50}, {{{0xc0000596b8, 0x155d9a0}, {0xc0000596b0, 0xc000338100}}})
    operators/app-operator/controllers/backup_controller.go:87 +0x1c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000da320, {0x183e6b0, 0xc0000b4000}, {0x14fda20, 0xc0009a4000})
    external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:298 +0x303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000da320, {0x183e6b0, 0xc0000b4000})
    external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:253 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:214 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
    external/io_k8s_sigs_controller_runtime/pkg/internal/controller/controller.go:210 +0x354
Vishrant
  • 15,456
  • 11
  • 71
  • 120
  • 1
    What fails—bulding (with bazel) or execution of a binary built by bazel? – kostix Oct 29 '21 at 18:50
  • The execution of the binary, the build is working fine. – Vishrant Oct 29 '21 at 18:52
  • 1
    What makes you think it's that goroutine specifically? Have you tried with go1.17? Have you tried when building with the standard toolchain and without the external linker, and without `-static`? You need to narrow down the possibilities (and note that statically linking glibc isn't really supported anyway, and must be done on the same system). Have you tried without cgo at all? – JimB Oct 29 '21 at 19:43
  • Yes I tried with go version `go1.17.2 linux/amd64`. I have a POST and a GET request, every time the POST request is passing, but the GET request was failing *randomly* in this goroute. Though after removing the `gc_linkopts` it seems to be working fine. (Earlier when I was running the app locally on go1.17.2 everything seems to be working fine, the binary was getting killed when I was running it in a container). The build environment and the execution environment was same though. – Vishrant Oct 29 '21 at 20:01
  • @JimB would you like to post your comment about `-static` in the answer section as it helped in fixing the issue? – Vishrant Oct 29 '21 at 20:09

1 Answers1

3

You should be able to remove the gc_linkopts options from the build. Statically linking glibc is not really supported, and may be causing problems.

If you have other cgo libraries you need to link statically, you will need to provide the linker flags for those libraries specifically.

If the only use of cgo is for name resolution on linux, you can almost always build without cgo and rely on the native Go implementations when a static binary is desired without a portable C implementation.

JimB
  • 104,193
  • 13
  • 262
  • 255
  • could you explain why would be the reason that's it's happening randomly, I am calling the same method, and sometimes it pass and sometimes it throws `segmentation violation` – Vishrant Oct 30 '21 at 00:34
  • 1
    @Vishrant, I don’t really even have a guess, I was just eliminating confounding variables, and getting rid the known problem of statically linking glibc. There may be a bug somewhere, ld, the go compiler, glibc, but I would have to specifically debug the crash to even know where to start. – JimB Oct 30 '21 at 00:45