empty slice when parsing msgpack encoded slice of msgpack encoded individual slice

Question

I am trying to decode a gzip response from server which an msgpack array or msgpack array which is finally gzipped.

so to illustrate this my response looks something like:

gzip(msgpack([msgpack([]), msgpack([]), msgpack([]) ....]))

this is what I have done so far, but I am getting msgpack error

msgpack: invalid code=3f decoding array length and hence all I get is empty slices.

this getBytes function I have taken from this SO answer

func getBytes(key interface{}) ([]byte, error) {
    var buf bytes.Buffer
    enc := gob.NewEncoder(&buf)
    err := enc.Encode(key)
    if err != nil {
        return nil, err
    }
    return buf.Bytes(), nil
}

var unZipped io.Reader
unZipped, err = gzip.NewReader(resp.Body)
var dataBytes bytes.Buffer
_, err = dataBytes.ReadFrom(unZipped)
if err != nil {
    logger.Println("error un gzipping", err)
}
var packedSlices []interface{}
bytesSlice := dataBytes.Bytes()
err = msgpack.Unmarshal(bytesSlice, &packedSlices)
for _, c := range packedSlices {
    var packedSlice []interface{}
    byts, _ := getBytes(c)
    err = msgpack.Unmarshal(byts, &packedSlice)
    logger.Println(err, packedSlice)
}

When I do the same thing in python 3.7 it is working fine:

resp = requests.post(url, headers=headers, json=payload, stream=True)
datas = msgpack.loads(gzip.decompress(resp.raw.data))
datas = [msgpack.loads(k) for k in datas]

Is there something I am missing here ??

Vadim Ashikhman · Accepted Answer · 2019-03-29T17:47:24.587

1

Instead of using []interface{} type as unpacked type use [][]byte. Unmarshal root msg to [][]byte then iterate over slice of slices of bytes and unmarshal every child byte slice. You do not need getBytes func.

Here some example:

var root [][]byte
if err :=  msgpack.Unmarshal(bytesSlice, &root); err != nil {
    panic(err)
}

for _, childBytes := range root {
    var child []interface{}
    if err := msgpack.Unmarshal(childBytes, &child); err != nil {
        panic(err)
    }

    _ = child
}

edited Mar 29 '19 at 17:47

answered Mar 29 '19 at 17:35

Vadim Ashikhman

9,851
1
35
39

thanks a lot this worked like a charm, just one more question is there any specific reason my method was not working and what if just for information I wanted to start unmarshalling first to []interface{} and then convert it to bytes ? – warl0ck Mar 29 '19 at 18:06
1

Actually you can use `[]interface{}` but then the convertion to bytes should be `byts := c.([]bytes)`. I think `getBytes` func didn't work because it encodes type metadata along with the bytes data, you can read more about `gob` package used by `getBytes` func here https://golang.org/pkg/encoding/gob/ – Vadim Ashikhman Mar 29 '19 at 18:18
that makes sense and yes conversion like you said also worked. – warl0ck Mar 30 '19 at 04:07

empty slice when parsing msgpack encoded slice of msgpack encoded individual slice

1 Answers1