Golang JSON Marshaling

Question

I'm trying to marshal an array into a string, separating all elements with newlines. I'm running out of memory and think about a more efficient way to do this.

buffer := ""
for _, record := range all_data {

    body, _ := json.Marshal(record)
    buffer += string(body) + "\n" // i run out of memory here

Question:

Is there a way to append a newline character to a byte array? Right now I'm casting via string(body), but I think that this operation allocates a lot of memory (but maybe I'm wrong).

You can change buffer to a slice of byte and use the builtin func append(...), but if you're trying to append GB of data you can get out of memory anyway. — Massimo Zerbini, Aug 03 '15 at 14:12
Consider to read http://stackoverflow.com/questions/24863164/how-to-analyse-golang-memory — Alex Netkachov, Aug 03 '15 at 14:24
`string` is constant in Golang. It's better to use `bytes.Buffer` instead and also other language which `string` is constant. — Bryce, Aug 04 '16 at 10:10

score 4 · Accepted Answer · answered Aug 03 '15 at 16:04

4

Assuming your data isn't inherently too big for the computer it's running on, the problem is likely the inefficient building of that string. Instead you should be using a bytes.buffer and then callings it's String() method. Here's an example;

var buffer bytes.Buffer

for _, record := range all_data {
    body, _ := json.Marshal(record)
    buffer.Write(body)
    buffer.WriteString("\n")
}

fmt.Println(buffer.String())

answered Aug 03 '15 at 16:04

evanmcdonnal

46,131
16
104
115

Heh, I was just about to post this as I just changed my code to use bytes.Buffer. Thanks! – K2xL Aug 03 '15 at 16:28

score 0 · Answer 2 · answered Nov 01 '18 at 22:22

0

To add to evanmcdonnal's answer: you don't even need an intermediate buffer created by json.Marshal:

var buf bytes.Buffer
enc := json.NewEncoder(&buf)
for _, record := range allData {
  if err := enc.Encode(record); enc != nil {
    // handle error
  }
  buf.WriteString("\n") // optional
}
fmt.Println(buf.String())

https://play.golang.org/p/5K9Oj0Xbjaa

answered Nov 01 '18 at 22:22

abbot

27,408
6
54
57

Inside the for loop, the check should be `err != nil`. – Gautam Aug 13 '21 at 12:12

Golang JSON Marshaling

2 Answers2