go.importer fails to import local (non-std) packages although regular import works

Question

I have a project that looks like this:

mymodule
- mypackage1
-- generate.go  # Generates documentation from models.go
- mypackage2
-- models.go  # Defines structs
- go.mod  # First line "module github.com/mymodule"
- go.sum

In mypackage1/generate.go, I want to import and document the structs defined in mypackage2/models.go. To do that, I tried executing this code in mypackage1/generate.go, modified from here and here:

_ = mypackage2.Mytype{}
pkg, err := importer.Default().Import("example.com/mymodule/mypackage2")
if err != nil {
    fmt.Println("error:", err)
}
for _, declName := range pkg.Scope().Names() {
    fmt.Println(declName)
}

mypackage2 is successfully imported and the first line of this block compiles without problem. But the call to importer fails and prints this error:

error: can't find import: "example.com/mymodule/mypackage2"

The import does work for fmt, but not for github.com/gorilla/mux. Does anyone have an idea what might be going wrong?

This question (How to use go importer ) is the most relevant, but I get the same error following these instructions for github.com/gorilla/mux (a dependency of my package).

This question explains to use the module name defined in go.mod, but I already do that. Neither that nor the mypackage2 standalone name works.

This question is related but uses a deprecated API and has no answer.

Answer 1

The importer package is apparently superceded by packages. I suspect that the problem is related to the GOPATH setting, which has been abandoned in favor of using modules. It appears that packages (written in 2018) knows how to parse the modules, while importer (written in 2012) does not.

The code below suggested by @mkopriva (in the comments and here) using packages instead of importer does what I was attempting to do, and parses the package contents.

To print each type/struct from the loaded package:

package main

import (
    "fmt"
    "go/ast"
    "go/token"
    "go/types"

    "golang.org/x/tools/go/packages"
)

const loadMode = packages.NeedName |
    packages.NeedFiles |
    packages.NeedCompiledGoFiles |
    packages.NeedImports |
    packages.NeedDeps |
    packages.NeedTypes |
    packages.NeedSyntax |
    packages.NeedTypesInfo

func main() {
    loadConfig := new(packages.Config)
    loadConfig.Mode = loadMode
    loadConfig.Fset = token.NewFileSet()
    pkgs, err := packages.Load(loadConfig, "github.com/me/mymodule/mypackage2")
    if err != nil {
        panic(err)
    }

    for _, pkg := range pkgs {
        for _, syn := range pkg.Syntax {
            for _, dec := range syn.Decls {
                if gen, ok := dec.(*ast.GenDecl); ok && gen.Tok == token.TYPE {
                    // print doc comment of the type
                    fmt.Println(gen.Doc.List[0])
                    for _, spec := range gen.Specs {
                        if ts, ok := spec.(*ast.TypeSpec); ok {
                            obj, ok := pkg.TypesInfo.Defs[ts.Name]
                            if !ok {
                                continue
                            }
                            typeName, ok := obj.(*types.TypeName)
                            if !ok {
                                continue
                            }
                            named, ok := typeName.Type().(*types.Named)
                            if !ok {
                                continue
                            }
                            // print the full name of the type
                            fmt.Println(named)

                            s, ok := named.Underlying().(*types.Struct)
                            if !ok {
                                continue
                            }

                            // print the struct's fields and tags
                            for i := 0; i < s.NumFields(); i++ {
                                idx := fmt.Sprint(i)
                                fmt.Println("s.Field(", idx, ").Name(): ", s.Field(i).Name())
                                fmt.Println("s.Tag(", idx, "): ", s.Tag(i))
                            }
                        }
                    }
                }
            }
        }
    }
}

To print each struct field's comments as well (using the AST):

[...]

                if gen, ok := dec.(*ast.GenDecl); ok && gen.Tok == token.TYPE {
                    fmt.Println("type comment:", gen.Doc.List[0].Text)
                    for _, spec := range gen.Specs {
                        if ts, ok := spec.(*ast.TypeSpec); ok {
                            fmt.Println("type name:", ts.Name)
                            st, ok := ts.Type.(*ast.StructType)
                            if !ok {
                                continue
                            }
                            
                            for i := 0; i < len(st.Fields.List); i++ {
                                f := st.Fields.List[i]
                                if f.Doc != nil {
                                    for j := 0; j < len(f.Doc.List); j++ {
                                        fmt.Println("field comment:", f.Doc.List[j].Text)
                                    } 
                                }
                                fmt.Println("field name:", f.Names[0])
                                fmt.Println("field tag:", f.Tag)
                            }
                        }
                    }
                }
            }
        }
    }
}