Golang unexpected EOF

Question

Here's my code, I'm new to Go. I tried googling the issue, but I can't quite put my finger on it. I think it has something to do with the Read() method.

package main

import (
    ...
)

type compressor struct {
    content []byte
}

func (r *compressor) compress() []byte {
    ...
}

func (r *compressor) decompress() []byte {
    var buffer bytes.Buffer
    dc := flate.NewReader(&buffer)
    _, err := dc.Read(r.content)
    if err != nil {
        if err != io.EOF {
            log.Fatal(err)
        }
    }

    return buffer.Bytes()
}

func main() {
    fileName := os.Args[1]
    fmt.Println(os.Args)
    contents, err := ioutil.ReadFile(fileName)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Print("Uncompressed data: ")
    fmt.Println(len(contents))

    comp := compressor{contents}
    buffer := comp.decompress()
    fmt.Print("Uncompressed data: ")
    fmt.Println(len(comp.decompress()))

    err = ioutil.WriteFile(fileName+".decjc", buffer, 0644)
    if err != nil {
        log.Fatal(err)
    }
}

Here's the output

dylan@skynet:~/Documents/EXP/jc$ ./jc data.txt.jc 
[./jc data.txt.jc]
Uncompressed data: 2364480
2018/06/29 21:41:35 unexpected EOF

You're reading from an empty buffer, so there's nothing to decompress. — JimB, Jun 29 '18 at 16:20
Perhaps your first line in `decompress()` you meant `bytes.NewReader(r.content)` — k1m190r, Jun 29 '18 at 21:11
@Dylan: no it does not. You're reading from `buffer` which is a `bytes.Buffer` which you never put any data into. — JimB, Jun 29 '18 at 22:28
Have another look at [io.Reader.Read](https://golang.org/pkg/io/#Reader). It reads *into* its argument, not *from* it. Where it's reading from depends entirely on the implementation. flate's reader reads from the thing you pass to NewReader. — Peter, Jun 30 '18 at 06:48
Please take a look at this Github issue comment: https://github.com/revel/revel/issues/566#issuecomment-42019967 — andy, Mar 19 '19 at 06:19
@andy: Thanks! I ran into the same issue and the comment you linked is the true solution. — Khoi, Jan 24 '21 at 03:14

score 11 · Accepted Answer · answered Aug 27 '19 at 04:28

After doing a trace on the particular code in question I have come to the following answer.

/src/bytes/reader.go 70

func (r *Reader) ReadByte() (byte, error) {
    ...

    if r.i >= int64(len(r.s)) {
        return 0, io.EOF
    }

    ....
}

There are four functions in bytes/reader that can return io.EOF, and zero functions that can return io.ErrUnexpectedEOF. The four functions that can return io.EOF are:

Read(b []byte)
ReadAt(b []byte, off int64)
ReadByte()
ReadRune()

/src/compress/flate/inflate.go 698

func (f *decompressor) moreBits() error {
    c, err := f.r.ReadByte()
    if err != nil {
        return noEOF(err)
    }

    ...
}

Of the four functions that can return io.EOF, only one function in flate/inflate.go calls any of them: moreBits() calls ReadByte()

/src/compress/flate/inflate.go 690

func noEOF(e error) error {
    if e == io.EOF {
        return io.ErrUnexpectedEOF
    }

    ...
}

When moreBits() receives an error it calls noEOF(), which checks if it had received an io.EOF. If this was the case then io.ErrUnexpectedEOF is returned backed. Everything seems to be working as intended, and it appears that it is the user's responsibility to be on the look out for this particular case. A suggested edit to the code above to handle what appears to be defined behavior is:

func (r *compressor) decompress() []byte {
    dc := flate.NewReader(bytes.NewReader(r.content))
    defer dc.Close()
    rb, err := ioutil.ReadAll(dc)
    if err != nil {
        if err != io.EOF && err != io.ErrUnexpectedEOF {
            log.Fatalf("Err %v\n read %v", err, rb)
        }
    }
    return rb
}

This was checked under go1.12.9

I ran in this problem in a test and your solution worked there. I don't think it's right, though, but for a test, I can have a few hacks like this. — Alexis Wilke, Oct 02 '19 at 02:29

score 4 · Answer 2 · edited Oct 01 '19 at 23:59

4

You got the in and outputs mixed up.

flate.NewReader takes the compressed input as an io.Reader and it returns a io.ReadCloser that can be used to get the uncompressed output:

func (r *compressor) decompress() []byte {
    dc := flate.NewReader(bytes.NewReader(r.content))
    defer dc.Close()
    rb, err := ioutil.ReadAll(dc)
    if err != nil {
        if err != io.EOF {
            log.Fatalf("Err %v\n read %v", err, rb)
        }
    }
    return rb
}

edited Oct 01 '19 at 23:59

Alexis Wilke

19,179
10
84
156

answered Jun 29 '18 at 21:10

RickyA

15,465
5
71
95

1

Reading into an empty byte slice is futile. You probably meant ioutil.ReadAll or so. – Peter Jun 30 '18 at 06:44
another thing, i found out, is that something goes wrong in the encoding while compressing. So while decompressing one of the bytes get decoded to an EOF. – Dylan Jun 30 '18 at 08:33
1

note that `compress/flate` does not uncompress gzip files. there is `compress/gzip` for that. – RickyA Jun 30 '18 at 09:27

Golang unexpected EOF

2 Answers2