How can I open files relative to my GOPATH?

Question

I'm using io/ioutil to read a small text file:

fileBytes, err := ioutil.ReadFile("/absolute/path/to/file.txt")

And that works fine, but this isn't exactly portable. In my case, the files I want to open are in my GOPATH, for example:

/Users/matt/Dev/go/src/github.com/mholt/mypackage/data/file.txt

Since the data folder rides right alongside the source code, I'd love to just specify the relative path:

data/file.txt

But then I get this error:

panic: open data/file.txt: no such file or directory

How can I open files using their relative path, especially if they live alongside my Go code?

(Note that my question is specifically about opening files relative to the GOPATH. Opening files using any relative path in Go is as easy as giving the relative path instead of an absolute path; files are opened relative to the compiled binary's working directory. In my case, I want to open files relative to where the binary was compiled. In hindsight, this is a bad design decision.)

The GOPATH hasn't a big meaning once your program is compiled, and even less when you distribute it. — Denys Séguret, Jun 12 '13 at 17:09
What you seem to want looks more like some embedding of the files in your compiled program. — Denys Séguret, Jun 12 '13 at 17:10
Kind of... except I want the data files separate from the source. The data files are vital to the program's functionality. So when somebody pulls down my source code (with the data files along side it) and compiles and runs it, the data files are loaded using a relative path because they exist near the source code, or near where the program is executing. — Matt, Jun 12 '13 at 17:11
The the compiled binary should have no dependence on the location of the source files, but it would be nice if there was a way to create an executable bundle that contained a copy of external resources upon which packages may depend. — , Jun 12 '13 at 17:24
@CrazyTrain Sounds good to me... I'm somewhat new to Go, and my typical approach to this (like Python or PHP) is to bundle the data files with the source code and reference them relatively. So for Go, my real question might be: how do I access these external resources? (If they are missing, of course, I'd handle accordingly.) — Matt, Jun 12 '13 at 17:32
@CrazyTrain is that what fileembed-go does? (https://bitbucket.org/rj/fileembed-go/) — Rick-777, Jun 12 '13 at 17:51
Here's a related question about bundling resources which may be sufficient, though this isn't my preferred method in my case: http://stackoverflow.com/questions/13904441/whats-the-best-way-to-bundle-static-resources-in-a-go-program -- or this one: http://stackoverflow.com/q/9443418/1048862 — Matt, Jun 12 '13 at 17:58
Check using pwd , _ := os.Getwd() your working dir then traverse to the path like opening a file "content, err := os.OpenFile(("..\\..\\pkg\\service\\profile\\TTL.json"), os.O_RDONLY, 0755)" — infiniteLearner, May 06 '21 at 09:57

score 104 · Accepted Answer · answered Jun 12 '13 at 17:54

104

Hmm... the path/filepath package has Abs() which does what I need (so far) though it's a bit inconvenient:

absPath, _ := filepath.Abs("../mypackage/data/file.txt")

Then I use absPath to load the file and it works fine.

Note that, in my case, the data files are in a package separate from the main package from which I'm running the program. If it was all in the same package, I'd remove the leading ../mypackage/. Since this path is obviously relative, different programs will have different structures and need this adjusted accordingly.

If there's a better way to use external resources with a Go program and keep it portable, feel free to contribute another answer.

answered Jun 12 '13 at 17:54

Matt

22,721
17
71
112

Ha, nearly a year later into Go, and I have never had a problem opening files by relative path since, nor have I had to use this little trick. I guess that means I've learned something. – Matt Mar 24 '14 at 04:26
12

What have you learned? :) Should we not share files like this across packages? – srt32 Sep 19 '14 at 20:47
8

@Matt, share the love. Don't want to use this "little trick" if you've got something better for us. You're the top result on google. – OGHaza Nov 17 '16 at 23:44
4

This question was from my first weeks programming in Go. The better question would have been how to bundle required data with a program. My answer here is for my very specific situation, I hope people aren't going off and using this haphazardly. – Matt Nov 19 '16 at 21:52
@Matt I am facing this issue because depending on how I start the program/tests in GoLand the working directory changes and hence the path to my file. If that is consistent then there probably isn't a reason to find the absolute path on every execution – wyatt Dec 07 '18 at 14:43
1

I read this and thought I had to expand relative paths for `ioutil.ReadFile` to work correctly, but it is completely unecessary. This works fine: `ioutil.ReadFile("../mypackage/data/file.txt")`. There's probably many other cases where you want to construct an absolute path, but it appears use with `io.ReadFile` is not one of them. – Davos May 02 '19 at 02:03
1

@Davos I think relative paths are resolved relative to the cwd. What will happen if you run the program from a different directory? Like 'go run ./some/directory/main.go' instead of 'go run main.go' – Aruna Herath Mar 20 '21 at 02:11
@ArunaHerath you can verify that by trying it out – Davos Mar 23 '21 at 05:13
@Davos Verified. If you run your program from a different directory it wont work. Because files are resolved relative to the current working directory. – Aruna Herath Mar 24 '21 at 00:45
Thanks @Matt. I am hoping this was from the early days of Caddy. Let us know if this is still the solution. May I also add that in Matt's case `"../mypackage"` is where the `*.go` is opening the file. It could also be a folder inside a package. Then the folder name becomes the new `mypackage`. – Edwinner Jan 03 '23 at 01:22
1

@Edwinner This whole question and answer should not be considered as relevant anymore. The GOPATH has long-been deprecated, and resources that ship with the binary should be embedded into it with go:embed. (It wasn't for Caddy though, which I didn't start developing until 4 months later.) – Matt Jan 04 '23 at 00:01

score 61 · Answer 2 · answered Jul 16 '15 at 21:14

61

this seems to work pretty well:

import "os"
import "io/ioutil"

pwd, _ := os.Getwd()
txt, _ := ioutil.ReadFile(pwd+"/path/to/file.txt")

answered Jul 16 '15 at 21:14

spencercooly

6,548
2
23
15

22

This is bad form because / is os specific, better to use filepath.join() when concatenating paths because it will use os.PathSeperator – Ace.C May 11 '18 at 23:45
11

Let me save the time for those willing to follow the (good) advice from the previous comment: use `import "path/filepath"` and call `filepath.Join(...)` (with capital `J`). :-) – Bojan Komazec Dec 27 '18 at 23:44
2

`os.Open(pwd + "../mocks/product/default.json")` how do you handle this case? it is taking this as literal. – kamal Feb 19 '19 at 09:28
The os.Open is fine for me: csvfile, err := os.Open("/mnt/c/Users/dmotta/Downloads/archivos_csv/tb_desp_new_201904081633.csv") – dmotta Apr 20 '19 at 05:07

score 13 · Answer 3 · answered Jun 14 '13 at 02:29

I wrote gobundle to solve exactly this problem. It generates Go source code from data files, which you then compile into your binary. You can then access the file data through a VFS-like layer. It's completely portable, supports adding entire file trees, compression, etc.

The downside is that you need an intermediate step to build the Go files from the source data. I usually use make for this.

Here's how you'd iterate over all files in a bundle, reading the bytes:

for _, name := range bundle.Files() {
    r, _ := bundle.Open(name)
    b, _ := ioutil.ReadAll(r)
    fmt.Printf("file %s has length %d\n", name, len(b))
}

You can see a real example of its use in my GeoIP package. The Makefile generates the code, and geoip.go uses the VFS.

@BurntSushi5 has a good point; my data files are hundreds of MB in size. This is cool, but I don't think that compiling the plain text files into the binary is a feasible option. — Matt, Jun 14 '13 at 17:02

score 6 · Answer 4 · answered Mar 01 '22 at 18:09

Starting from Go 1.16, you can use the embed package. This allows you to embed the files in the running go program.

Given the file structure:

-- main.go
-- data
  \- file.txt

You can reference the file using a go directive

package main

import (
  "embed"
  "fmt"
)

//go:embed data/file.txt
var content embed.FS

func main() {
  text, _ := content.ReadFile("data/file.txt")
  fmt.Println(string(text))
}

This program will run successfully regardless of where the program is executed. This is useful in case the file could be called from multiple different locations, for instance, from a test directory.

IMHO, this answer should be nominated as correct answer. Works perfect for my use case here: https://github.com/VOU-folks/HoGo/blob/main/apps/hogo-manager/main.go#L74 — num8er, Jul 15 '23 at 11:17

score 4 · Answer 5 · answered Jun 14 '13 at 16:54

I think Alec Thomas has provided The Answer, but in my experience it isn't foolproof. One problem I had with compiling resources into the binary is that compiling may require a lot of memory depending on the size of your assets. If they're small, then it's probably nothing to worry about. In my particular scenario, a 1MB font file was causing compilation to require somewhere around 1GB of memory to compile. It was a problem because I wanted it to be go gettable on a Raspberry Pi. This was with Go 1.0; things may have improved in Go 1.1.

So in that particular case, I opt to just use the go/build package to find the source directory of the program based on the import path. Of course, this requires that your targets have a GOPATH set up and that the source is available. So it isn't an ideal solution in all cases.

How can I open files relative to my GOPATH?

5 Answers5

Linked