How to bundle an SQLite database in a Go binary?

Question

I am try to use go-bindata and packr, but those packages do not show how to pack an SQLite database file in to a binary file.

I don't need to update the database in any way, I just want to read the data from it on startup.

How can I embed an SQLite database file in a Go binary file?

Answer 1

The SQLite driver can't read a database file from memory (e.g. from a byte slice). But you can write the data to a temporary file, and open that:

//go:generate go run gen.go

package main

import (
    "database/sql"
    "fmt"
    "io/ioutil"
    "log"
    "os"

    _ "github.com/mattn/go-sqlite3"
)

func main() {
    // Create temporary file for database.
    tmpDB, err := ioutil.TempFile("", "db*.sqlite3")
    if err != nil {
        log.Fatal(err)
    }
    // Remove this file after on exit.
    defer func() {
        err := os.Remove(tmpDB.Name())
        if err != nil {
            log.Print(err)
        }
    }()

    // Write database to file.
    _, err = tmpDB.Write(sqlDB)
    if err != nil {
        log.Print(err)
    }
    err = tmpDB.Close()
    if err != nil {
        log.Print(err)
    }

    // Open DB.
    db, err := sql.Open("sqlite3", tmpDB.Name()+"?mode=ro")
    if err != nil {
        log.Fatal(err)
    }

    // Make sure it's loaded correct.
    rows, err := db.Query("select * from test")
    if err != nil {
        log.Fatal(err)
    }

    for rows.Next() {
        var c string
        err := rows.Scan(&c)
        if err != nil {
            log.Fatal(err)
        }
        fmt.Println(c)
    }
}

And you can write the database to db.go with something like:

// +build generate

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "os"
    "strings"
)

func main() {
    // Read source database file.
    d, err := ioutil.ReadFile("source.sqlite3")
    if err != nil {
        log.Fatal(err)
    }

    fp, err := os.Create("db.go")
    if err != nil {
        log.Fatal(err)
    }

    _, err = fmt.Fprintf(fp, "// Code generated by gen.go; DO NOT EDIT.\n\n"+
        "package main\n\n"+
        "var sqlDB = %s\n", asbyte(d))
    if err != nil {
        log.Fatal(err)
    }
}

// Write any data as byte array.
func asbyte(s []byte) string {
    var b strings.Builder
    for i, c := range s {
        if i%19 == 0 {
            b.WriteString("\n\t\t")
        }
        b.WriteString(fmt.Sprintf("%#x, ", c))
    }
    return "[]byte{" + b.String() + "}"
}

You can also use go-bindata or packr for that if you prefer, but I don't really see an advantage.

---

An alternative way is to use a memory database, which may be faster depending on what you want to do.

1. Embed the SQL schema and rows you want in your Go binary as strings.
2. Open a new memory database when your program starts (sql.Open("sqlite3",:memory:`) and create the schema and insert the rows.

There is no disk access with this method, so querying it will probably be a bit faster at the expensive of slower startup times (benchmark to be sure!)