-2

ioutil.ReadAll() can cause a memory spike if the file is too large .

Aswin A.B.
  • 27
  • 4
  • Well i guess your question is duplicate of [this](https://stackoverflow.com/questions/52539695/alternative-to-ioutil-readall-in-go/52540512) check the second comment – Arman Aug 20 '20 at 04:22
  • Are you asking how to limit the amount of memory used when reading the response body? – Charlie Tumahai Aug 20 '20 at 04:22
  • The memory spike is because you read all of it. If you don't want to read all of it, then read it in chunks. If this is a JSON, use json.Decoder. – Burak Serdar Aug 20 '20 at 04:23
  • 1
    If you want to read the response body fully to an in-memory buffer, then you have to pay the memory price. It does not matter if you use `ioutil.ReadAll` or some other method. If you do not want to read the full file to an in-memory buffer, you should consider using something like multipart encoding https://stackoverflow.com/questions/16958448/what-is-http-multipart-request – Sankar Aug 20 '20 at 04:35
  • 1
    If you want to read it full you have to read it full and if ioutil.ReadAll uses too much memory in this process your own "better" solution will use also too much memory. Redesign. – Volker Aug 20 '20 at 05:56

1 Answers1

2

You can use io.Copy which takes io.Writer and io.Reader.

io.Copy uses 32KB buffer to copy from reader to writer until it reaches EOF. Even the source is much larger it uses 32KB buffer.

io.Copy(writer, reader)
rezam
  • 627
  • 1
  • 12
  • 17