Golang Serialize go-radix Tree to file?

Question

I've been working on serializing a radix tree (used for indexing) to a file in golang. The radix tree nodes are storing 6-bit roaring bitmaps (see https://github.com/RoaringBitmap/roaring). The following code is what I am using, and the output I am getting when trying to load it back into memory:

serializedTree := i.index.ToMap()
    encodeFile, err := os.Create(fmt.Sprintf("./serialized/%s/%s", appindex.name, i.field))
    if err != nil {
        panic(err)
    }
    e := gob.NewEncoder(encodeFile)
    err = e.Encode(serializedTree)
    encodeFile.Close()
    // Turn it back for testing
    decodeFile, err := os.Open(fmt.Sprintf("./serialized/%s/%s", appindex.name, i.field))
    defer decodeFile.Close()
    d := gob.NewDecoder(decodeFile)
    decoded := make(map[string]interface{})
    err = d.Decode(&decoded)
    fmt.Println("before decode", serializedTree)
    fmt.Println("after decode", decoded)
    if err != nil {
        fmt.Println("!!! Error serializing", err)
        panic(err)
    }

Output:

before decode map[dan:{1822509180252590512} dan1:{6238704462486574203} goodman:{1822509180252590512,6238704462486574203}]
after decode map[]
!!! Error serializing EOF
panic: EOF

goroutine 1 [running]:
main.(*appIndexes).SerializeIndex(0xc000098240)

(I understand the decode is empty because the gob package doesn't modify on EOF error)

I've noticed that when trying with bytes directly, only 15 bytes are being stored on disk (which is way too few). Trying with the encoding/json package with json.Marshall() and json.Unmarshall() and I see 33 bytes stored, but they are loading in empty (the roaring bitmaps are gone):

post encode map[dan:map[] dan1:map[] goodman:map[]]

I feel like this has something to do with the fact that I am trying to serialize a map[string]interface{} rather than something like a map[string]int, but I am still fairly green with golang.

See https://repl.it/@danthegoodman/SelfishMoralCharactermapping#main.go for an example and my testing.

score 0 · Answer 1 · answered Oct 25 '20 at 23:47

I believe I fixed it by converting the map[string]interface{} into a map[string]*roaring64.Bitmap before writing to disk, then decoding it back into a map[string]*roaring64.Bitmap then converting it back to a map[string]interface{}

m2 := make(map[string]*roaring64.Bitmap)

// Convert m1 to m2

for key, value := range m1 {
  m2[key] = value.(*roaring64.Bitmap)
}

fmt.Println("m1", m1)
fmt.Println("m2", m2)

encodeFile, err := os.Create("./test")
    if err != nil {
        panic(err)
    }
    e := gob.NewEncoder(encodeFile)
    err = e.Encode(m2)
    encodeFile.Close()
    // Turn it back for testing
    decodeFile, err := os.Open("./test")
    defer decodeFile.Close()
    d := gob.NewDecoder(decodeFile)
    decoded := make(map[string]*roaring64.Bitmap)
    err = d.Decode(&decoded)
    fmt.Println("before decode", m2)
    fmt.Println("after decode", decoded)
    if err != nil {
        fmt.Println("!!! Error serializing", err)
        panic(err)
    }

m3 := make(map[string]interface{})

// Convert m2 to m3

for key, value := range m2 {
  m3[key] = value
}

afterDecTree := radix.NewFromMap(m3)

See https://repl.it/@danthegoodman/VictoriousUtterMention#main.go for a working example

Golang Serialize go-radix Tree to file?

1 Answers1