How do I dump the struct into the byte array without reflection?

Question

I already found encoding/binary package to deal with it, but it depended on reflect package so it didn't work with uncapitalized(that is, unexported) struct fields. However I spent a week to find that problem out, I still have a question: if struct fields should not be exported, how do I dump them easily into binary data?

EDIT: Here's the example. If you capitalize the name of fields of Data struct, that works properly. But Data struct was intended to be an abstract type, so I don't want to export these fields.

package main
import (
    "fmt"
    "encoding/binary"
    "bytes"
)

type Data struct {
    id int32
    name [16]byte
}


func main() {
    d := Data{Id: 1}
    copy(d.Name[:], []byte("tree"))
    buffer := new(bytes.Buffer)
    binary.Write(buffer, binary.LittleEndian, d)
    // d was written properly
    fmt.Println(buffer.Bytes())
    // try to read...
    buffer = bytes.NewBuffer(buffer.Bytes())
    var e = new(Data)
    err := binary.Read(buffer, binary.LittleEndian, e)
    fmt.Println(e, err)
}

Just to be sure to understand : you want to serialize the structs in order to be able to deserialize them after ? Will your deserializer know into what struct deserialize ? — Denys Séguret, Oct 12 '12 at 07:23
Hi! Do you have any example code? And an example of the output you expect? — minikomi, Oct 12 '12 at 07:24
If you are just trying to a raw binary dump, the `unsafe` is capable of doing what you want. It depends on needs. — ANisus, Oct 12 '12 at 07:24
@dystroy First time, I tried to deserialize the struct like this: `type Data struct { id int32, name [16]byte }` — kroisse, Oct 12 '12 at 07:43
It seems to `name` field was the cause for me, so I started to find how to read the array field in the struct. That's a reason why I spent lots of time. — kroisse, Oct 12 '12 at 07:46
@ANisus I need the platform-independent conversion, like a network transfer. so `unsafe` isn't suitable for me. — kroisse, Oct 12 '12 at 07:50
@kroisse: As said, it depends on the needs :) . And yes, with platform independance in mind, skip `unsafe` — ANisus, Oct 12 '12 at 08:00
The traditional way to do this (mostly language agnostic), is to create a separate struct used to put stuff on the network wire from the struct you use for logic in your program. i.e. you create a separate WireData struct, and you convert from your struct Data to WireData before serializing WireData, and the oposite when you receive the data. WireData keeps only the information from Data that needs to be serialized. — nos, Oct 12 '12 at 08:23

Denys Séguret · Accepted Answer · 2012-10-12T08:29:05.963

35

Your best option would probably be to use the gob package and let your struct implement the GobDecoder and GobEncoder interfaces in order to serialize and deserialize private fields.

This would be safe, platform independent, and efficient. And you have to add those GobEncode and GobDecode functions only on structs with unexported fields, which means you don't clutter the rest of your code.

func (d *Data) GobEncode() ([]byte, error) {
    w := new(bytes.Buffer)
    encoder := gob.NewEncoder(w)
    err := encoder.Encode(d.id)
    if err!=nil {
        return nil, err
    }
    err = encoder.Encode(d.name)
    if err!=nil {
        return nil, err
    }
    return w.Bytes(), nil
}

func (d *Data) GobDecode(buf []byte) error {
    r := bytes.NewBuffer(buf)
    decoder := gob.NewDecoder(r)
    err := decoder.Decode(&d.id)
    if err!=nil {
        return err
    }
    return decoder.Decode(&d.name)
}

func main() {
    d := Data{id: 7}
    copy(d.name[:], []byte("tree"))
    buffer := new(bytes.Buffer)
    // writing
    enc := gob.NewEncoder(buffer)
    err := enc.Encode(d)
    if err != nil {
        log.Fatal("encode error:", err)
    }
    // reading
    buffer = bytes.NewBuffer(buffer.Bytes())
    e := new(Data)
    dec := gob.NewDecoder(buffer)
    err = dec.Decode(e)
    fmt.Println(e, err)
}

edited Oct 12 '12 at 08:29

answered Oct 12 '12 at 07:59

Denys Séguret

372,613
87
782
758

Great solution! I tried like this way to use binary.Read/Write manually for each fields, however this code is closer to standard. But... I should match the layout and size of serialized struct. Gob stream is attached some metadata to self-describe the data itself, so the layout was broken. – kroisse Oct 12 '12 at 08:53
@dystroy, how can we approach your solution if the object contains another object and so on? – Minty Aug 07 '13 at 01:38
1

@Minty it's automatically recursive, so there's no problem. Just add the two functions when you have unexported fields you want to serialize or when you want to customize the serialization. – Denys Séguret Aug 07 '13 at 06:10
Thanks, I need to give it a try. – Minty Aug 07 '13 at 15:46
@dystroy I'm trying write custom GobEncode/decode for struct type which has **Channels and function types** .Your solution helped me to move but stuck on above type.Could you please extend your advice on that.Thanks!! – faisal_kk Dec 02 '14 at 12:54
I think this should be corrected: in `main()`, `enc.Encode(d)` -> `enc.Encode(&d)`. – starriet Apr 20 '22 at 09:12

How do I dump the struct into the byte array without reflection?

1 Answers1

Linked