replace retool with tools.go for multi developer and CI environments using go modules

Question

I'd like to replace retool with go modules tools.go "tools as dependencies". However I'm struggling to understand how this works when my developers and CI env all use different operating systems.

I want to make sure each environment uses the exact same version of tools.

For a concrete example, my app requires the protoc compiler to generate go code via github.com/golang/protobuf/protoc-gen-go. I have 3 OS, all needing to execute protoc with the protoc-gen-go plugin/generator:

Ryan: uses MacOS
Joe: Linux (Ubuntu)
CI: Linux (CentOS)

I currently use retool to make sure ALL environments are locked in on the same version of tools (protoc-gen-go in this ex):

retool do build/bin/protoc -Ibuild/protoc/include -I. rpc/platform/platform.proto --go_out=.

My new go modules / "tools as dependencies" setup

tools.go:

// +build tools

package tools

import (
    _ "github.com/golang/protobuf/protoc-gen-go"
)

Set the path that go install will use:

export GOBIN=$PWD/bin

Install:

go install github.com/golang/protobuf/protoc-gen-go

If Ryan runs the go install .. a bin/protoc-gen-go MacOS executable is created.

Questions:

At this point, why isprotoc-gen-go tool version (or git hash) NOT listed in go.mod?
When Joe clones the app repo, how does he get and compile the same version of protoc-gen-go that Ryan used?
How does protoc know to use the protoc-gen-go executable generator in my ./bin dir?

score 3 · Accepted Answer · answered Jun 18 '19 at 01:14

I was able to accomplish vendored tools build for protoc (and plugins like Twirp) following the Go Modules tools guidelines, plus a little Makefile-Fu for the protoc binary.

A complete working example can be found in the Aspiration Labs pyggpot repo. Following are the essential details. Worth noting: getting the import path right for some of the tools was very fiddly, but ultimately, successful.

For protoc itself, I vendor the binary release in the Makefile and set it up into a tools/bin dir:

TOOLS_DIR := ./tools
TOOLS_BIN := $(TOOLS_DIR)/bin

# protoc
PROTOC_VERSION := 3.7.1
PROTOC_PLATFORM := osx-x86_64
PROTOC_RELEASES_PATH := https://github.com/protocolbuffers/protobuf/releases/download
PROTOC_ZIP := protoc-$(PROTOC_VERSION)-$(PROTOC_PLATFORM).zip
PROTOC_DOWNLOAD := $(PROTOC_RELEASES_PATH)/v$(PROTOC_VERSION)/$(PROTOC_ZIP)
PROTOC := $(TOOLS_BIN)/protoc

# protoc
$(PROTOC): $(TOOLS_DIR)/$(PROTOC_ZIP)
    unzip -o -d "$(TOOLS_DIR)" $< && touch $@  # avoid Prerequisite is newer than target `tools/bin/protoc'.

$(TOOLS_DIR)/$(PROTOC_ZIP):
    curl --location $(PROTOC_DOWNLOAD) --output $@

The PROTOC_PLATFORM string can be automated with something like OS detecting makefile. The version of that we use is at https://github.com/aspiration-labs/pyggpot/blob/master/build/makefiles/osvars.mk.

On to building the go tools. Create a tools.go something like

// +build tools

package tools

import (
    // protocol buffer compiler plugins
    _ "github.com/golang/protobuf/protoc-gen-go"
    _ "github.com/twitchtv/twirp/protoc-gen-twirp"
    _ "github.com/twitchtv/twirp/protoc-gen-twirp_python"
    _ "github.com/thechriswalker/protoc-gen-twirp_js"
)

Note: the // +build tools tag will keep go build from over-building tools imports in your final build.

Finally, some make code to build your go tools:

# go installed tools.go
GO_TOOLS := github.com/golang/protobuf/protoc-gen-go \
            github.com/twitchtv/twirp/protoc-gen-twirp \
            github.com/twitchtv/twirp/protoc-gen-twirp_python \
            github.com/thechriswalker/protoc-gen-twirp_js \

# tools
GO_TOOLS_BIN := $(addprefix $(TOOLS_BIN), $(notdir $(GO_TOOLS)))
GO_TOOLS_VENDOR := $(addprefix vendor/, $(GO_TOOLS))

setup_tools: $(GO_TOOLS_BIN)

$(GO_TOOLS_BIN): $(GO_TOOLS_VENDOR)
    GOBIN="$(PWD)/$(TOOLS_BIN)" go install -mod=vendor $(GO_TOOLS)

And finally, a make setup target to run go mod vendor and process the targets above.

setup: setup_vendor $(TOOLS_DIR) $(PROTOC) setup_tools

# vendor
setup_vendor:
    go mod vendor

$(TOOLS_DIR):
    mkdir -v -p $@

this is EXACTLY what I was looking for, and very close to what I came up with last night. I was using `--plugin=protoc-gen-twirp=./build/gobin/protoc-gen-twirp` to point to the built tool for each plugin, like you PATH trick MUCH better. THANK YOU. — rynop, Jun 18 '19 at 15:40
Bit unrelated: I notice you `.gitignore` `vendor` - why? IMO if your vendoring you want it in source so all devs are in lock-step, & your CI don't have to hit the network. — rynop, Jun 18 '19 at 15:57
On the first point, I'm willing to trust same inputs will produce same outputs, with the versions locked down by go.mod and go.sum. On the second, yes, speeding up CI might make this worthwhile, but still don't like the idea of committing vendor to the repo. I'd opt for a separate "tools" docker image build for CI. Good point though. — rm -rf, Jun 18 '19 at 16:49
That’s a bit too complex setup for protoc… [goprotoc](https://github.com/jhump/goprotoc) works flawlessly (so far) and is compatible with C++ protoc. I also don’t like the idea of having an extra `tools.json` file since everything is already tracked by Go modules, so I’ve written a [program](https://github.com/tie/tools.go) that just runs `go install` for each underscore import in `tools.go` file. This is far from perfect since it installs binaries to `$GOBIN` (or wherever Go decides), but it can be easily fixed — I’m just hesitating to store binaries in the project folder (`retools`). — tie, Jun 25 '19 at 21:43

score 0 · Answer 2 · answered Jun 17 '19 at 21:05

Go modules work with the imports of your .go files. If they find an import, they will automatically download the latest version that satisfies your requirements. You have to read https://github.com/golang/go/wiki/Modules and understand how modules work since Go 1.11 and later.

At this point, why is protoc-gen-go tool version (or git hash) NOT listed in go.mod?

This is because protoc-gen-go is just an external tool as far Go modules are concerned. You don't import golang/protobuf/tree/master/protoc-gen-go but the code it generates.

When Joe clones the app repo, how does he get and compile the same version of protoc-gen-go that Ryan used?

Use:

GIT_TAG="v1.2.0" # change as needed
go get -d -u github.com/golang/protobuf/protoc-gen-go
git -C "$(go env GOPATH)"/src/github.com/golang/protobuf checkout $GIT_TAG
go install github.com/golang/protobuf/protoc-gen-go

to install a specific version on each machine of the users. Probably write a build script that automates the process.

How does protoc know to use the protoc-gen-go executable generator in my ./bin dir?

From the github docs: The compiler plugin, protoc-gen-go, will be installed in $GOPATH/bin unless $GOBIN is set. It must be in your $PATH for the protocol compiler, protoc, to find it.

Thanks for the quick response. Having to write a script per dependency, that manually specifies a git tag is anti-dependency management. Having 10 "tools" is not so far fetched. There has got to be a better way.... Your `git -C` command is working on the gopath, which when using go modules, is also going against the grain (should be outside of $GOPATH). Lastly, I do import - thats what the `tools.go` is for, see https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module — rynop, Jun 18 '19 at 01:05

replace retool with tools.go for multi developer and CI environments using go modules

2 Answers2