10

I have a Go project I would like to open source but there are certain elements which are not suitable for OSS, e.g. company specific logic etc.

I have conceived of the following approach:

  • interfaces are defined in the core repository.
  • Plugins can then be standalone repositories whose types implement the interfaces defined in core. This allows the plugins to be housed in completely separate modules and therefore have their own CI jobs etc.
  • Plugins are compiled into the final binary via symlinks.

This would result in a directory structure something like the following:

|- $GOPATH
  |- src
    |- github.com
      |- jabclab
        |- core-system
          |- plugins <-----|
      |- xxx               | 
        |- plugin-a ------>| ln -s
      |- yyy               |  
        |- plugin-b ------>|

With an example workflow of:

$ go get git@github.com:jabclab/core-system.git
$ go get git@github.com:xxx/plugin-a.git
$ go get git@github.com:yyy/plugin-b.git
$ cd $GOPATH/src/github.com
$ ln -s ./xxx/plugin-a/*.go ./jabclab/core-system/plugins
$ ln -s ./yyy/plugin-b/*.go ./jabclab/core-system/plugins
$ cd jabclab/core-system
$ go build

The one issue I'm not sure about is how to make the types defined in plugins available at runtime in core. I'd rather not use reflect but can't think of a better way at the moment. If I was doing the code in one repo I would use something like:

package plugins

type Plugin interface {
  Exec(chan<- string) error
}

var Registry map[string]Plugin

// plugin_a.go
func init() { Registry["plugin_a"] = PluginA{} }

// plugin_b.go
func init() { Registry["plugin_b"] = PluginB{} }

In addition to the above question would this overall approach be considered idiomatic?

jabclab
  • 14,786
  • 5
  • 54
  • 51
  • 1
    Why not do it like `database/sql`? Have the "core" (i.e. `database/sql`), have your "plugins" (e.g. `github.com/lib/pq`), have your "main" which imports the "core" and all "plugins" you want to use. No need for symlinking. – Danilo Feb 29 '16 at 20:14
  • you can have `go generate` render a main file that imports the plugins, and in each plugin's `init()` function register it in the main repository. The importing, even as `import _ "foo/bar/plugin"` will cause the init func to run on start and register the plugins. – Not_a_Golfer Feb 29 '16 at 20:45
  • 5
    See this answer http://stackoverflow.com/questions/28001872/golang-events-eventemitter-dispatcher-for-plugin-architecture/28003144#28003144 – Not_a_Golfer Feb 29 '16 at 20:46
  • @Not_a_Golfer thanks :-) would this approach allow the external plugins to be compilable standalone? Would they not fail to compile due to the registry variable not being present? If not, I guess the `//go:generate` could either be (a) in the plugin itself, or (b) in the core app which then built further on your `import _ "foo/bar/plugin"` in that it would also generate the addition of the plugin's `type` to the plugin registry. – jabclab Mar 01 '16 at 19:44
  • 1
    This thread on reddit detailed the options for a plugin architecture. Go lang *will have*, in its further releases a buildmode allowing externally loaded plugin. https://www.reddit.com/r/golang/comments/3xyaro/plugins_in_golang/ – Bactisme Sep 20 '16 at 11:27

1 Answers1

5

This is one of my favorite issues in Go. I have an open source project that has to deal with this as well (https://github.com/cpg1111/maestrod), it has pluggable DB and Runtime (Docker, k8s, Mesos, etc) clients. Prior to the plugin package that is in the master branch of Go (so it should be coming to a stable release soon) I just compiled all of the plugins into the binary and allowed configuration decide which to use.

As of the plugin package, https://tip.golang.org/pkg/plugin/, you can use dynamic linking for plugins, so similar to C's dlopen() in its loading, and the behavior of go's plugin package is pretty well outlined in the documentation.

Additionally I recommend taking a look at how Hashicorp addresses this by doing RPC over a local unix socket. https://github.com/hashicorp/go-plugin

The added benefit of running a plugin as a separate process like Hashicorp's model does, is that you get great stability in the event that the plugin fails but the main process is able to handle that failure.

I should also mention Docker does its plugins in Go similarly, except Docker uses HTTP instead of RPC. Additionally, a Docker engineer has posted about embedding a Javascript interpreter for dynamic logic in the past http://crosbymichael.com/category/go.html.

The issue I wanted to point out with the sql package's pattern that was mentioned in the comments is that that's not a plugin architecture really, you're still limited to whatever is in your imports, so you could have multiple main.go's but that's not a plugin, the point of a plugin is such that the same program can run either one piece of code or another. What you have with things like the sql package is flexibility where a separate package determines what DB driver to use. Nonetheless, you end up modifying code to change what driver you are using.

I want to add, with all of these plugin patterns, aside from the compiling into the same binary and using configuration to choose, each can have its own build, test and deployment (i.e. their own CI/CD) but not necessarily.

Christian Grabowski
  • 2,782
  • 3
  • 32
  • 57