24

As far as I'm aware (see here, and here) there is no type discovery mechanism in the reflect package, which expects that you already have an instance of the type or value you want to inspect.

Is there any other way to discover all exported types (especially the structs) in a running go package?

Here's what I wish I had (but it doesn't exist):

import "time"
import "fmt"

func main() {
    var types []reflect.Type
    types = reflect.DiscoverTypes(time)
    fmt.Println(types)
}

The end goal is to be able to discover all the structs of a package that meet certain criteria, then be able to instantiate new instances of those structs.

BTW, a registration function that identifies the types is not a valid approach for my use case.


Whether you think it's a good idea or not, here's why I want this capability (because I know you're going to ask):

I've written a code generation utility that loads go source files and builds an AST to scan for types that embed a specified type. The output of the utility is a set of go test functions based on the discovered types. I invoke this utility using go generate to create the test functions then run go test to execute the generated test functions. Every time the tests change (or a new type is added) I must re-run go generate before re-running go test. This is why a registration function is not a valid option. I'd like to avoid the go generate step but that would require my utility to become a library that is imported by the running package. The library code would need to somehow scan the running namespace during init() for types that embed the expected library type.

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Michael Whatcott
  • 5,603
  • 6
  • 36
  • 50
  • The only thing I can think about is generating code that calls a register function in `init()` for you. – THUNDERGROOVE Aug 21 '15 at 04:49
  • Yeah @THUNDERGROOVE , that's basically what I've implemented. But I don't want to generate code. I rather run `go test` and have the library discover all the types, instantiate them, and then invoke them. All in one process execution. It's a doozy of the problem. – Michael Whatcott Aug 21 '15 at 05:06

8 Answers8

32

In Go 1.5, you can use the new package types and importer to inspect binary and source packages. For example:

package main

import (
    "fmt"
    "go/importer"
)

func main() {
    pkg, err := importer.Default().Import("time")
    if err != nil {
        fmt.Printf("error: %s\n", err.Error())
        return
    }
    for _, declName := range pkg.Scope().Names() {
        fmt.Println(declName)
    }
}

You can use the package go/build to extract all the packages installed. Or you can configure the Lookup importer to inspect binaries outside the environment.

Before 1.5, the only no-hacky way is to use the package ast to compile the source code.

Alvivi
  • 3,263
  • 3
  • 26
  • 28
  • Wow, didn't know this new functionality existed. What good timing! Now for the golden question: After getting retrieving pkg.Scope().Names(), can I somehow hook into the reflect package to instantiate new instances of the Names() that represent types? – Michael Whatcott Aug 21 '15 at 14:41
  • 1
    Seems that you are looking for the plugin architecture. Right now, it is not implemented in Go 1.5 . Read https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit# for more information – Alvivi Aug 21 '15 at 14:50
  • @hlin117 Well, what I need is the ability to load all types in the executing package and instantiate new instances of many of them at runtime during test execution. Can the code in this answer be extended/modified to accomplish that? – Michael Whatcott Jan 17 '18 at 23:17
  • Could you elaborate on this answer more? I got error cannot find pkg "time" https://play.golang.org/p/yuCFQ8r0DrS – vda8888 Jan 31 '18 at 01:24
  • @vda8888 That is because the playground runs on sandboxed environment. *"The playground can use most of the standard library, with some exceptions. The only communication a playground program has to the outside world is by writing to standard output and standard error."* This example is still working on a standard environment with go 1.9. – Alvivi Jan 31 '18 at 07:48
  • @Alvivi Does this mean that this would not work if I only have access to the built binary at runtime (i.e. no access to Golang source code or my source code)? – vda8888 Feb 05 '18 at 04:40
  • @vda8888, nope. This is based on importer (https://golang.org/pkg/go/importer/) which needs access to golang environment. – Alvivi Feb 05 '18 at 09:42
  • If the package is under GOPATH, run `go install ` first to install the .a file to the GOPATH/pkg/... or else the package will not be found. This works for me with go 1.10.4 on windows 10. – minghua Sep 27 '18 at 23:34
  • 1
    Does this work for local packages? – Vivere Sep 19 '22 at 10:30
19

(see bottom for 2019 update)

Warning: untested and hacky. Can break whenever a new version of Go is released.

It is possible to get all types the runtime knows of by hacking around Go's runtime a little. Include a small assembly file in your own package, containing:

TEXT yourpackage·typelinks(SB), NOSPLIT, $0-0
    JMP reflect·typelinks(SB)

In yourpackage, declare the function prototype (without body):

func typelinks() []*typeDefDummy

Alongside a type definition:

type typeDefDummy struct {
    _      uintptr           // padding
    _      uint64            // padding
    _      [3]uintptr        // padding
    StrPtr *string           
}

Then just call typelinks, iterate over the slice and read each StrPtr for the name. Seek those starting with yourpackage. Note that if there are two packages called yourpackage in different paths, this method won't work unambiguously.

can I somehow hook into the reflect package to instantiate new instances of those names?

Yeah, assuming d is a value of type *typeDefDummy (note the asterisk, very important):

t := reflect.TypeOf(*(*interface{})(unsafe.Pointer(&d)))

Now t is a reflect.Type value which you can use to instantiate reflect.Values.


Edit: I tested and executed this code successfully and have uploaded it as a gist.

Adjust package names and include paths as necessary.

Update 2019

A lot has changed since I originally posted this answer. Here's a short description of how the same can be done with Go 1.11 in 2019.

$GOPATH/src/tl/tl.go

package tl

import (
    "unsafe"
)

func Typelinks() (sections []unsafe.Pointer, offset [][]int32) {
    return typelinks()
}

func typelinks() (sections []unsafe.Pointer, offset [][]int32)

func Add(p unsafe.Pointer, x uintptr, whySafe string) unsafe.Pointer {
    return add(p, x, whySafe)
}

func add(p unsafe.Pointer, x uintptr, whySafe string) unsafe.Pointer

$GOPATH/src/tl/tl.s

TEXT tl·typelinks(SB), $0-0
    JMP reflect·typelinks(SB)

TEXT tl·add(SB), $0-0
    JMP reflect·add(SB)

main.go

package main

import (
    "fmt"
    "reflect"
    "tl"
    "unsafe"
)

func main() {
    sections, offsets := tl.Typelinks()
    for i, base := range sections {
        for _, offset := range offsets[i] {
            typeAddr := tl.Add(base, uintptr(offset), "")
            typ := reflect.TypeOf(*(*interface{})(unsafe.Pointer(&typeAddr)))
            fmt.Println(typ)
        }
    }
}

Happy hacking!

thwd
  • 23,956
  • 8
  • 74
  • 108
  • 1
    That's a nice one ;-) – Volker Aug 21 '15 at 09:21
  • @thwd: Wow, that's crazy. Now for the golden question: After invoking typelinks() to get the []string type names, can I somehow hook into the reflect package to instantiate new instances of those names? – Michael Whatcott Aug 21 '15 at 14:45
  • Thanks for the additional info. I'm interested to see if it works. – Michael Whatcott Aug 21 '15 at 19:25
  • @mdwhatcott I've tested everything and it works. I have linked a working gist in the answer. This was on Go 1.4.2. – thwd Aug 23 '15 at 23:11
  • 1
    @thwd, this answer *could* be everything I've been looking for, for weeks! Now it's 2019, are there any other updates? Also, out of interest, if I'm building this on MacOS then zasm_linux_amd64.h won't exist - what's the alternative I'm looking for? – Jimbo Jan 14 '19 at 09:34
  • 1
    @Jimbo I amended the answer ;) – thwd Jan 14 '19 at 11:19
  • @thwd I've been playing with this for the past day, does it *definitely* pluck out all of the structs available in all packages?? I created a basic struct in the main package with some vars and obviously named methods and this doesn't seem, pick it up. – Jimbo Jan 15 '19 at 09:50
  • PS - given this struct type, I would like to instantiate it. That is my goal :-) – Jimbo Jan 15 '19 at 11:58
  • 1
    @Jimbo wups, another change I hadnt noticed is that now they only include pointer, channel, map, slice and array types in typelinks. It's documented in src/reflect/type.go :( – thwd Jan 15 '19 at 12:48
  • So effectively there’s no way for us to go from the string “MyStruct” -> an instance of MyStruct without writing the type literal in-code... this was my last hope! Thanks for the update anyway @thwd. – Jimbo Jan 15 '19 at 13:16
  • @thwd - Your 2019 update works like a charm! Thanks very much for coming back. – Michael Whatcott Feb 24 '19 at 05:03
  • I'm noticing that the typelinks (`tl`) code/package can't be contained inside my own github package. I get a compilation failure: "github.com/smartystreets/gunit/tl.Typelinks: relocation target github.com/smartystreets/gunit/tl.typelinks not defined " Is there a way around that? @thwd – Michael Whatcott Feb 27 '19 at 00:06
  • Are you guys all using old version of Go? I'd give anything to have this working on modern Go versions. – Jimbo Aug 08 '19 at 13:32
  • 1
    Excellent work, thwd :) mdwhatcott, if you want to have the tl package live inside a package you can instead define "TEXT ·typelinks(SB), $0-0", and Go will know it's in your package (https://golang.org/doc/asm#symbols, starting "Most hand-written assembly files do not include the full package path"). In Go 1.13.4 I'm getting some errors about reflect.add, but I've successfully copied its definition into tl.go from https://github.com/golang/go/blob/563287ae0636a592880dc616b50fde217fa1100f/src/reflect/type.go#L1026-L1035 and the solution here works as described. – icio Jan 14 '20 at 00:08
  • 1
    Hey! You can also use this instead of assembly: ``` //go:linkname reflect_typelinks reflect.typelinks func reflect_typelinks() ([]unsafe.Pointer, [][]int32) ``` – Sergey Kamardin May 11 '21 at 09:30
  • I adapted the answer to work with `//go:linkname` in Go 1.18 https://stackoverflow.com/a/72740994/1009836 – mbert Jun 24 '22 at 08:18
3

Update 2022 with Go 1.18

With Go 1.18 the accepted answer doesn't work anymore, but I could adapt it to use go:linkname. Using this directive and the unsafe package these internal functions can now be accessed without any extra assembly code.

package main

import (
    "fmt"
    "reflect"
    "unsafe"
)

//go:linkname typelinks reflect.typelinks
func typelinks() (sections []unsafe.Pointer, offset [][]int32)

//go:linkname add reflect.add
func add(p unsafe.Pointer, x uintptr, whySafe string) unsafe.Pointer

func main() {
    sections, offsets := typelinks()
    for i, base := range sections {
        for _, offset := range offsets[i] {
            typeAddr := add(base, uintptr(offset), "")
            typ := reflect.TypeOf(*(*interface{})(unsafe.Pointer(&typeAddr)))
            fmt.Println(typ)
        }
    }
}
mbert
  • 1,495
  • 2
  • 11
  • 17
2

Unfortunately, I don't think this is possible. Packages are not "actionable" in Go, you can't "call a function" on it. You can't call a function on a type either, but you can call reflect.TypeOf on an instance of the type and get reflect.Type which is a runtime abstraction of a type. There just isn't such mechanism for packages, there isn't a reflect.Package.

With that said, you could file an issue about the absence of (and practicality of adding) reflect.PackageOf etc.

Ainar-G
  • 34,563
  • 13
  • 93
  • 119
2

Thanks @thwd and @icio, follow your direction it still worked on 1.13.6 today.

Follow your way the tl.s will be:

TEXT ·typelinks(SB), $0-0
    JMP reflect·typelinks(SB)

yes, no package name and no "add" function in it.

then follow @icio's way change "add" function to:

func add(p unsafe.Pointer, x uintptr, whySafe string) unsafe.Pointer {
    return unsafe.Pointer(uintptr(p) + x)
}

then all worked now. :)

EricHe
  • 21
  • 4
1

Version for go 1.16(tested for go version go1.16.7 linux/amd64)

  • This can only generate code and strings. You have to paste generated code somewhere then compile it again
  • Works if only sources are available.
import (
  "fmt"
  "go/ast"
  "golang.org/x/tools/go/packages"
  "reflect"
  "time"
  "unicode"
)

func printTypes(){
  config := &packages.Config{
    Mode:  packages.NeedSyntax,
  }
  pkgs, _ := packages.Load(config, "package_name")
  pkg := pkgs[0]

  for _, s := range pkg.Syntax {
    for n, o := range s.Scope.Objects {
      if o.Kind == ast.Typ {
        // check if type is exported(only need for non-local types)
        if unicode.IsUpper([]rune(n)[0]) {
          // note that reflect.ValueOf(*new(%s)) won't work with interfaces
          fmt.Printf("ProcessType(new(package_name.%s)),\n", n)
        }
      }
    }
  }
}

full example of possible use case: https://pastebin.com/ut0zNEAc (doesn't work in online repls, but works locally)

quant2016
  • 447
  • 4
  • 13
  • Interesting approach! The original hope was to be able to get actual `reflect.Type` instances for each type in a package with a single method call (and not have to resort to any assembly hacks). I doubt such a utility will ever be provided by the `reflect` package. – Michael Whatcott Sep 02 '21 at 22:17
0
  • After go 1.11 dwarf debugging symbols added runtime type information, you can get the runtime type by using this address
  • DW_AT_go_runtime_type
  • gort You can see more content
package main

import (
    "debug/dwarf"
    "fmt"
    "log"
    "os"
    "reflect"
    "runtime"
    "unsafe"

    "github.com/go-delve/delve/pkg/dwarf/godwarf"
    "github.com/go-delve/delve/pkg/proc"
)

func main() {
    path, err := os.Executable()
    if err != nil {
        log.Fatalln(err)
    }

    bi := proc.NewBinaryInfo(runtime.GOOS, runtime.GOARCH)
    err = bi.LoadBinaryInfo(path, 0, nil)
    if err != nil {
        log.Fatalln(err)
    }
    mds, err := loadModuleData(bi, new(localMemory))
    if err != nil {
        log.Fatalln(err)
    }

    types, err := bi.Types()
    if err != nil {
        log.Fatalln(err)
    }

    for _, name := range types {
        dwarfType, err := findType(bi, name)
        if err != nil {
            continue
        }

        typeAddr, err := dwarfToRuntimeType(bi, mds, dwarfType, name)
        if err != nil {
            continue
        }

        typ := reflect.TypeOf(*(*interface{})(unsafe.Pointer(&typeAddr)))
        log.Printf("load type name:%s type:%s\n", name, typ)
    }
}

// delve counterpart to runtime.moduledata
type moduleData struct {
    text, etext   uint64
    types, etypes uint64
    typemapVar    *proc.Variable
}

//go:linkname findType github.com/go-delve/delve/pkg/proc.(*BinaryInfo).findType
func findType(bi *proc.BinaryInfo, name string) (godwarf.Type, error)

//go:linkname loadModuleData github.com/go-delve/delve/pkg/proc.loadModuleData
func loadModuleData(bi *proc.BinaryInfo, mem proc.MemoryReadWriter) ([]moduleData, error)

//go:linkname imageToModuleData github.com/go-delve/delve/pkg/proc.(*BinaryInfo).imageToModuleData
func imageToModuleData(bi *proc.BinaryInfo, image *proc.Image, mds []moduleData) *moduleData

type localMemory int

func (mem *localMemory) ReadMemory(data []byte, addr uint64) (int, error) {
    buf := *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{Data: uintptr(addr), Len: len(data), Cap: len(data)}))
    copy(data, buf)
    return len(data), nil
}

func (mem *localMemory) WriteMemory(addr uint64, data []byte) (int, error) {
    return 0, fmt.Errorf("not support")
}

func dwarfToRuntimeType(bi *proc.BinaryInfo, mds []moduleData, typ godwarf.Type, name string) (typeAddr uint64, err error) {
    if typ.Common().Index >= len(bi.Images) {
        return 0, fmt.Errorf("could not find image for type %s", name)
    }
    img := bi.Images[typ.Common().Index]
    rdr := img.DwarfReader()
    rdr.Seek(typ.Common().Offset)
    e, err := rdr.Next()
    if err != nil {
        return 0, fmt.Errorf("could not find dwarf entry for type:%s err:%s", name, err)
    }
    entryName, ok := e.Val(dwarf.AttrName).(string)
    if !ok || entryName != name {
        return 0, fmt.Errorf("could not find name for type:%s entry:%s", name, entryName)
    }
    off, ok := e.Val(godwarf.AttrGoRuntimeType).(uint64)
    if !ok || off == 0 {
        return 0, fmt.Errorf("could not find runtime type for type:%s", name)
    }

    md := imageToModuleData(bi, img, mds)
    if md == nil {
        return 0, fmt.Errorf("could not find module data for type %s", name)
    }

    typeAddr = md.types + off
    if typeAddr < md.types || typeAddr >= md.etypes {
        return off, nil
    }
    return typeAddr, nil
}
lsg2020
  • 1
  • 1
-2

No there is not.

If you want to 'know' your types you'll have to register them.

Volker
  • 40,468
  • 7
  • 81
  • 87