3

I'm using golang net/http package to retrieve the uploaded zip file via postman. The attachment file link. It is not dangerous file. Feel free to check out.

Development env

  • local machine m1 macbook pro golang 1.17.2 - no issue
  • server docker image golang:1.17.5-stretch - got issue.

Code to capture the post form transSourceFile file.

func HandleFileReqTest(w http.ResponseWriter, req *http.Request, params map[string]string) err {

    if err := req.ParseMultipartForm(32 << 20); err != nil {
       return err
    }

    file, header, err := req.FormFile("transSourceFile")
    if err != nil {
       return err
    }
    defer file.Close()
    fmt.Println("header.Size:", header.Size)
    return nil
}

I tried below code also no use

func HandleFileReqTest(w http.ResponseWriter, req *http.Request, params map[string]string) err {
    if err := req.ParseForm(); err != nil {
        return err
    }
    req.ParseMultipartForm(32 << 20)
    file, header, err := req.FormFile("transSourceFile")
    if err != nil {
        return err
    }
    defer file.Close()
    fmt.Println("header.Size:", header.Size)
    return nil
}

Result: Local machine got the same file size as the origin file. Server with golang:1.17.5-stretch got the different file size compare to origin file.

As the result on this, i'm unable to unzip the file in the server. Anyone can help?

Chrest Koo
  • 31
  • 1
  • 5
  • Nothing in the question is specific to docker or that it is a ZIP file - it is only about uploading a file and not receiving what was uploaded at the server. I've removed these misleading tags and also changed the title. – Steffen Ullrich Jan 13 '22 at 06:22
  • *"Server with golang:1.17.5-stretch got the different file size compare to origin file."* - what exactly is different size: more or less than the original? If less: do the transferred bytes match, i.e. it is only missing the end? If more: do the original bytes match, i.e. is there only junk at the end? Or what exactly is different here? Also maybe the upload itself is broken and the brokeness is only differently interpreted in different golang versions? Please provide exactly how the upload looks like (like with a packet capture). – Steffen Ullrich Jan 13 '22 at 06:24
  • @SteffenUllrich Thank you removing the misleading tags if i did. it is added because it might be development issue. – Chrest Koo Jan 13 '22 at 07:35
  • @SteffenUllrich the file size in server is more than origin file size. btw. so far the main problem is not on unable to unzip the file. The main issue is on the zip file header.Size value in from req.FormFile("transSourceFile") shows more bytes compare to origin. Origin have 171 bytes, header.Size value is 205. Can you suggest me on how to compare this 2 files? – Chrest Koo Jan 13 '22 at 07:43
  • @SteffenUllrich I had undo back the title with zip. So far it is only happen in server handle zip file. i had tried server handle non-zip file. non-zip file is working. i face no issue on local zip file. – Chrest Koo Jan 13 '22 at 08:09
  • *"Can you suggest me on how to compare this 2 files?"* - in UNIX there is a `cmp` command for this which provides the information on the first difference in binary files. Also, does the headerSize actually reflect the final file size? And again, please provide some packet capture of the transmission (like with tcpdump) so that one can see what gets actually transmitted. It is hard to reproduce your problem without having a way to recreate exactly what you transmit. – Steffen Ullrich Jan 13 '22 at 08:11
  • Ok! This issue seems to be present on postman community itself here : https://community.postman.com/t/get-response-with-application-zip-as-content-type-creation/8005 . Try with something else instead of postman like `curl` – Ashutosh Singh Jan 13 '22 at 08:47
  • @SteffenUllrich Thanks for the cmp suggestion. “ Also, does the headerSize actually reflect the final file size? ” i have no idea on this. I mention on this is becasue they are having the same size after i saved it using code below: f, err := os.Create("some.zip") defer f.Close() n, err := io.Copy(f, file) comparison result link https://1drv.ms/u/s!ApOAtDwy99qQ9mMnjoPa-iKdXEIg?e=DLK3Ug how can i provide this "provide some packet capture of the transmission (like with tcpdump)" ? – Chrest Koo Jan 13 '22 at 08:56
  • @ChrestKoo: The comparison shows lots of `EF BF BD` which suggests that something is trying to treat the binary zip file as UTF-8 and failing - see https://stackoverflow.com/questions/6543548/. As for doing the packet capture: use tcpdump as desribed [here](https://byteplumbing.net/2018/01/inspecting-docker-container-network-traffic/) to record packets into a file and then provide the file. – Steffen Ullrich Jan 13 '22 at 09:27

2 Answers2

2

You need to copy form file to the actual file:

f, err := os.Create("some.zip")
defer f.Close()
n, err := io.Copy(f, file)
serge-v
  • 750
  • 2
  • 8
  • 2
    it depends on uploaded size. file can be in memory: https://cs.opensource.google/go/go/+/refs/tags/go1.17.6:src/mime/multipart/formdata.go;l=107;drc=refs%2Ftags%2Fgo1.17.6 – serge-v Jan 13 '22 at 04:44
  • Thanks for the idea. i did copied form file to actual file but this is process after i get the file value. So far what I'm facing is the header.Size value from req.FormFile("transSourceFile") are different. Current situation is In my local machine, the header.Size value i got is 171 which is same as origin file. while in server, the header.Size value i got is 205 which is different with origin file. This is happened in zip file for so far as i know. Others file type no idea yet. – Chrest Koo Jan 13 '22 at 05:33
0

Data isn't being flushed to the file completely. You should close the file first to ensure that the data is fully flushed.

  // create a local file filename
    dst, err := os.Create("filename.zip")

      // save it
     fl, err = io.Copy(dst, src)

     // Close the file
     dst.Close()

     stat, _ := dst.Stat()

 //Now check the size stat.Size() or header.Size after flushing the file.
Ashutosh Singh
  • 721
  • 3
  • 6
  • It is not related on creating the new file yet. The problem is i got the wrong data info from req.FormFile("transSourceFile"). If got the wrong data, it will save with wrong data also. This issue only happenned on server zip file. i face no issue on local zip file – Chrest Koo Jan 13 '22 at 08:06