Slow json marshaling and unmarshaling

Question

My project has a requirement API to return an array(about 500 element struct)

I have tried to use lib like fasthttp, easyjson, rapidjson(call with cgo),but result is not good enough.

Do you guys have any better recommendation?

Here is my code:

type Line struct {
    Time   string  `json:"time" bson:"time"`
    Open   float64 `json:"open" bson:"open"`
    Close  float64 `json:"close" bson:"close"`
    High   float64 `json:"high" bson:"high"`
    Low    float64 `json:"low" bson:"low"`
    Volume float64 `json:"volume" bson:"volume"`
    Amount float64 `json:"amount" bson:"amount"`
}

type MultiLines struct {
    AllLines []Line `json:"lines"`
}

Test Code:

func BenchmarkJson500(b *testing.B) {
    for i := 0; i < b.N; i++ {
        _, err := json.Marshal(&sliceData)
        if err != nil {
            panic(err)
        }
    }
}

func BenchmarkUnmarshalJson500(b *testing.B) {
    lines := make([]Line, 500)
    for i := 0; i < b.N; i++ {
        err := json.Unmarshal(sliceJson, &MultiLines{lines})
        if err != nil {
            panic(err)
        }
    }
}

func BenchmarkEasyJson500(b *testing.B) {
    for i := 0; i < b.N; i++ {
        _, err := sliceData.MarshalJSON()
        if err != nil {
            panic(err)
        }
    }
}

func BenchmarkEasyUnmarshalJson500(b *testing.B) {
    for i := 0; i < b.N; i++ {
        slice := MultiLines{}
        err := slice.UnmarshalJSON(sliceJson)
        if err != nil {
            panic(err)
        }
    }
}

And benchmark tests result:

BenchmarkUnmarshalJson500-4          500           2821450 ns/op
BenchmarkJson500-4                   500           2151984 ns/op

Because EasyJson Rewrite the UnmarshalJSON/MarshalJSON,so i test with the generated code at different time.

BenchmarkEasyJson500-4              1000           1434724 ns/op
BenchmarkEasyUnmarshalJson500-4     1000           1276298 ns/op

Anyway,ffjson is very similar with easyjson.

By the way,rapidjson is slow because cgo call cost lots of time — Keto, Jul 21 '16 at 08:48
Without seeing any code you'd like to improve, we can just point you to similar questions: [Go JSON decoding is very slow. What would be a better way to do it?](http://stackoverflow.com/questions/29282231/go-json-decoding-is-very-slow-what-would-be-a-better-way-to-do-it); and [Speeding up JSON parsing in Go](http://stackoverflow.com/questions/33222075/speeding-up-json-parsing-in-go) — icza, Jul 21 '16 at 08:54
I have appended my code to the question.may i ask for you help again. — Keto, Jul 21 '16 at 10:11

score 1 · Answer 1 · answered May 31 '18 at 10:57

Try fastjson. It should parse the input string much faster than encoding/json and faster than easyjson. Additionally, it doesn't require pre-defining Go structs for the parsed JSON and doesn't need code generation. See the example below:

var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
    log.Fatal(err)
}

for _, vv := range v.GetArray() {
    fmt.Printf("time=%s\n", vv.GetStringBytes("time"))
    fmt.Printf("open=%f\n", vv.GetFloat64("open"))
    // etc...
    // There is no need in extracting all the fields -
    // extract only required fields in order to get better performance.
}

Thank you for this library. Really fantastic. Makes life much easier — Jiulin Teng, Sep 15 '21 at 22:18

score 0 · Answer 2 · answered Jul 21 '16 at 08:58

0

What do you mean by "the results are not good enough"?

I'd use the the json package of the standard library:

https://golang.org/pkg/encoding/json/

As far as I can tell, it's used everywhere and pretty fast and resource-friendly. If that does not cut it either, you might want to consider ffjson:

https://github.com/pquerna/ffjson

Let me know if this was helpfull!

answered Jul 21 '16 at 08:58

enzian

733
5
13

thx for answer me, appreciate a lot for that, ffjson is not satisfied me neither. – Keto Jul 21 '16 at 09:41

score 0 · Answer 3 · answered Jul 21 '16 at 14:28

0

While using easyjson you should use easyjson.Unmarshal() function instead of json.Unmarshal(). Generated custom MarshalJSON()/UnmarshalJSON() are only a small part of performance optimization (they don't use reflect package). The main profit from easyjson you get from optimized easyjson.Unmarshal() and easyjson.Marshal(). First one doesn't make JSON pre-validation which is very slow. The second one uses concurrency to boost encoding.

From my experience easyjson boosts unmarshaling up to 2 times for small JSON or JSON with simple structure (e.g. array of simple objects), and up to 4 times for big or/and deep-nested JSON structures. Unfortunately, I don't have any statistics for marshaling.

answered Jul 21 '16 at 14:28

Vadim Petrov

350
1
4

when compile with generate code of easyjson,easyjson.Marshal() and easyjson.Unmarshal() are just a wrapper of MarshalEasyJSON()/UnmarshalEasyJSON() ,i have try it according to your recommendation,but it did't make any difference. – Keto Jul 22 '16 at 01:57
My test has reach the performance of about 2 times faster than go-json.But when marshal/unmarshal big array,it become wrose. – Keto Jul 22 '16 at 02:02
Yes, you're right, sorry. Your tests are correct. Anyway, x2 performance boost is not so bad. All other alternative libs are not better, but easyjson is easiest to use. – Vadim Petrov Jul 22 '16 at 10:51

Slow json marshaling and unmarshaling

3 Answers3