28

We are converting our internal codebase from the dep dependency manager to go modules (vgo or built in with go1.11.2). Imagine we have code like this:

$GOPATH/src/mycompany/myprogram/main.go:

package main

import (
        "fmt"
        lib "mycompany/mylib" )

func main() {
        fmt.Println("2+3=", lib.add(2, 3)) 
}

$GOPATH/src/mycompany/myprogram/go.mod:

module mycompany/myprogram

(it doesn't have any dependencies; our real-world code does).

$GOPATH/src/mycompany/mylib/lib.go:

package mylib

func Add(x int, y int) int {
        return x + y
}

I didn't module-ize this code; it doesn't seem to matter whether I do or don't.

These are trivial examples but our internal code follows a similar structure as this worked historically.

Since these directories are on the Gopath, export GO111MODULE=auto still builds as before and this works fine (modules not used because we are on the gopath). However, when I set export GO111MODULE=on I immediately get the error:

build mycompany/myprogram: cannot find module for path mycompany/mylib

So I did some research and I would like to validate my understanding. First let me say our old approach worked, but I am more interested in changing to use go modules as it appears to be where the go project itself is headed. So.

  1. It seems the intention of the golang authors was that "dotless" paths belong to the standard repository only; that is there should be a binding between domain name and project. We don't use go get on our internal project, unsurprisingly. Here is the source specifically:

    Dotless paths in general are reserved for the standard library; go get has (to my knowledge) never worked with them, but go get is also the main entry point for working with versioned modules.

    Can anyone with more knowledge of golang than me confirm this?

  2. My key assumption is that once go decides to use modules, all dependencies must be modules and the gopath becomes somewhat irrelevant, except as a cache (for downloaded modules). Is this correct?

  3. If this is true, we need to use a private gitlab (in our case) repository on the path. There's an open issue on handling this that I'm aware of so we can implement this if necessary. I'm more interested in the consequences, specifically for iterating in the private repositories. Previously we could develop these libraries locally before committing any changes; now it seems we have a choice:

    1. Accept this remote dependency and iterate. I was hoping to avoid needing to push and pull remotely like this. There are workarounds to needing an internet connection if strictly necessary.
    2. Merge everything into one big git repository.

If it matters, I'm using go version go1.11.2 linux/amd64 and my colleagues are using darwin/amd64. If it helps, my golang is exactly as installed by Fedora's repositories.

So, tl;dr, my question is: are go modules all-or-nothing, in that any dependency must be resolved using the module system (go get, it seems) and the gopath has become redundant? Or is there something about my setup that might trigger this to fail? Is there some way to indicate a dependency should be resolved explicitly from the gopath?

Updates since asking the question:

  1. I can move myprogram out of the gopath. The same issue occurs (mylib has been left in the gopath).
  2. I can run, or not run, go mod init mycompany/mylib in the mylib directory; it makes no difference at all.
  3. I came across Russ Cox's blog post on vgo. My concerns about offline development that I tried not to dive into too far are resolved by $GOPROXY.
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
diagprov
  • 443
  • 1
  • 4
  • 8
  • It look like you never ran `go mod init mycompany` in $GOPATH/src/mycompany, therefore there is no module yet. // The comment about dotless paths you quoted is by bcmills, one of the Go maintainers (or author, even?). You can take that as authorative. // Modules are not "all-or-nothing". You can depend on non-modules just fine. – Peter Nov 28 '18 at 10:03
  • @Peter I left that out of my question, but I did (otherwise it wouldn't build as a module, no?). I'll edit that into the question. So if this is true my question becomes: if I can depend on non-modules, why doesn't it work? :P – diagprov Nov 28 '18 at 10:23
  • Ah, I see now. You have to add a [replace statement](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive) in mycompany/myprogram to make this work. – Peter Nov 28 '18 at 12:39
  • @Peter this sounds like exactly what I need. I added `replace mycompany/mylib => ../mylib/` to my example, didn't work. So, domain name issue. So I renamed all `mycompany` references to `mycompany.com`... and now the build (of the example in my Q) hangs until the internet times out and then gives me the usual `build mycompany.com/myprogram: cannot find module for path mycompany.com/mylib` :( – diagprov Nov 28 '18 at 13:20
  • So I guess we need to combine the two approaches; build from private but remote repositories with url-like imports and then we can replace locally to speed up dev. – diagprov Nov 28 '18 at 14:25
  • @diagprov I am in the same boat and get this error. Were you able to find any solution to your problem? – Tarun Tyagi Oct 04 '19 at 21:24
  • @TarunTyagi not that was satisfying with gitlab. We could only use top level repositories (gitlab.com/org/repo) and not gitlab.com/org/subproj/repo) because of some crazy gitlab feature. We also had to configure ssh instead of https because there's no token support on gitlab. We've since migrated to github, thankfully. – diagprov Oct 05 '19 at 22:44
  • Go vanity urls is something I am also investigating right now as a way to provide the required redirection. It still requires the remote end to support the go modules protocol to discover the correct repo format, and some kind of token support for http auth, but it can at least allow you to redirect go.yourcompany.com/repo -> gitlab.com/org/subproj/repo.git (.git required because gitlab doesn't provide support in subprojects for go modules). – diagprov Oct 05 '19 at 22:45

3 Answers3

28

I use a workaround with GITHUB_TOKEN to solve this.

  1. Generate GITHUB_TOKEN here https://github.com/settings/tokens
  2. export GITHUB_TOKEN=xxx
  3. git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic@github.com/mycompany".insteadOf "https://github.com/mycompany"
030
  • 10,842
  • 12
  • 78
  • 123
Alex Pliutau
  • 21,392
  • 27
  • 113
  • 143
12

I wrote up a solution for this on Medium: Go Modules with Private Git Repositories.

The way we handle it is basically the same as the answer above from Alex Pliutau, and the blog goes into some more detail with examples for how to set up your git config with tokens from GitHub/GitLab/BitBucket.

The relevant bit for GitLab:

git config --global \
  url."https://oauth2:${personal_access_token}@privategitlab.com".insteadOf \
  "https://privategitlab.com"

#or 

git config --global \
  url."https://${user}:${personal_access_token}@privategitlab.com".insteadOf \
  "https://privategitlab.com"

I hope it's helpful.

timjonesdev
  • 179
  • 1
  • 11
  • Is there a way to use `http` instead of `https`. We have a legacy `git` server behind a firewall where everything is `http` and the change above does not work when invoking the `go` command (which is trying to fetch the repo via `https` on port 443) – TPPZ Jul 08 '19 at 15:09
  • This is a common solution, but it fails with `https://repos.example.com/my-library?go-get=1` request made by go tool before reaching cloning phase. – Paweł Szczur Aug 12 '19 at 11:42
  • 1
    You should be more specific and state that your answer, where it mentions BitBucket, is purposely for BitBucket server not BitBucket cloud. Luckily you mention the [`.netrc` solution](https://confluence.atlassian.com/bitbucketserver/permanently-authenticating-with-git-repositories-776639846.html#PermanentlyauthenticatingwithGitrepositories-Usingthe.netrcfile) in your article and this happens to work in go @1.13 onwards by also specifying the `GOPRIVATE`](https://golang.org/doc/go1.13#modules) environment variable. – n1nsa1d00 Jul 28 '20 at 13:42
12

I use ls-remote git command to help resolve private repo tags and go get after it.

$ go env GO111MODULE=on
$ go env GOPRIVATE=yourprivaterepo.com
$ git ls-remote -q https://yourprivaterepo.com/yourproject.git
$ go get
utkusonmez
  • 1,486
  • 15
  • 22
  • 3
    You can use `go env -w GOPRIVATE` for private Github repositories as well: `go env -w GOPRIVATE=github.com/username/*` – lmika Apr 18 '20 at 22:10
  • if nothing helps: https://stackoverflow.com/a/71617473/3025289 –  Mar 25 '22 at 13:11