Correctly log protobuf messages as unescaped JSON with zap logger

Question

I have a Go project where I'm using Zap structured logging to log the contents of structs. That's how I initialise the logger:

zapLog, err := zap.NewProductionConfig().Build()
if err != nil {
    panic(err)
}

Initially I started with my own structs with json tags and it all worked perfectly:

zapLog.Info("Event persisted", zap.Any("event", &event))

Result:

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id", 
 "type":"updated", "value":"{...}", "context":{"foo":"bar"}}}

I now switched to protobuf and I'm struggling to achieve the same result. Initially I just got the "reflected map" version, when using zap.Any():

zapLog.Info("Event persisted", zap.Any("event", &event))

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":"sourceType:TYPE_X sourceId:\"some-source-id\", 
 type:\"updated\" value:{...}, context:<key: foo, value:bar>}

I tried marshalling the object with the jsonpb marshaller, which generated the correct output on itself, however, when I use it in zap.String(), the string is escaped, so I get an extra set of '\' in front of each quotation mark. Since there's processing of the logs at a later point, this causes problems there and hence I want to avoid it:

m := jsonpb.Marshaler{}
var buf bytes.Buffer
if err := m.Marshal(&buf, msg); err != nil {
    // handle error
}
zapLog.Info("Event persisted", zap.ByteString("event", buf.Bytes()))

Result:

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":"{\"sourceType\":\"TYPE_X\", \"sourceId\":\"some-source-id\", 
 \"type\":\"updated\", \"value\":\"{...}\", \"context\":{\"foo\":"bar\"}}"}

I then tried using zap.Reflect() instead of zap.Any() which was the closest thing I could get to what I need, except that enums are rendered as their numerical values (the initial solution did not have enums, so that didn't work in the pre-protobuf solution either):

zapLog.Info("Event persisted", zap.Reflect("event", &event))

Result:

{"level":"info","ts":1626448680.69099,"caller":"persisters/log.go:56",
 "msg":"Event persisted","event":{"sourceType":4, "sourceId":"some-source-id", 
 "type":"updated", "value":"{...}", "context":{"foo":"bar"}}}

The only option I see so far is to write my own MarshalLogObject() function:

type ZapEvent struct {
    event *Event
}

func (z *ZapEvent) MarshalLogObject(encoder zapcore.ObjectEncoder) error {

  encoder.AddString("sourceType", z.event.SourceType.String()
  // implement encoder for each attribute

}

func processEvent(e Event) {
   ...
   zapLog.Info("Event persisted", zap.Object("event", &ZapEvent{event: &e}))
}

But since it's a complex struct, I would rather use a less error prone and maintenance heavy solution. Ideally, I would tell zap to use the jsonpb marshaller somehow, but I don't know if that's possible.

blackgreen · Accepted Answer · 2021-07-20T16:03:50.317

Use zap.Any with a json.RawMessage. You can convert directly the byte output of jsonpb.Marshaler:

    foo := &pb.FooMsg{
        Foo: "blah", 
        Bar:  1,
    }

    m := jsonpb.Marshaler{}
    var buf bytes.Buffer
    if err := m.Marshal(&buf, foo); err != nil {
        // handle error
    }

    logger, _ := zap.NewDevelopment()
    logger.Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes())))

The bytes will be printed as:

Event persisted {"event": {"foo":"blah","bar":"1"}}`

I believe that's the easiest way, however I'm also aware of a package kazegusuri/go-proto-zap-marshaler (I'm not affiliated to it) that generates MarshalLogObject() implementations as a protoc plugin. You may want to take a look at that too.

Is it possible to make it work with Desugar ? logger.Desugar().Info("Event persisted", zap.Any("event", json.RawMessage(buf.Bytes()))) — serkan, Mar 10 '23 at 21:22

complynx · Answer 2 · 2022-04-21T08:08:29.613

I used another way to jsonify protos.

Since protos can be naturally marshaled, I just wrapped them in the strict-to-json marshaler.

And you can modify the internals to use protojson (newer jsonpb).

Unlike the marshaler in the previous solution, this one doesn't require ahead-of-logging processing.


type jsonObjectMarshaler struct {
    obj  any
}

func (j *jsonObjectMarshaler) MarshalJSON() ([]byte, error) {
    bytes, err := json.Marshal(j.obj)
    // bytes, err := protojson.Marshal(j.obj)
    if err != nil {
        return nil, fmt.Errorf("json marshaling failed: %w", err)
    }
    return bytes, nil
}

func ZapJsonable(key string, obj any) zap.Field {
    return zap.Reflect(key, &jsonObjectMarshaler{obj: obj})
}

Then to use it, just

logger, _ := zap.NewDevelopment()
logger.Info("Event persisted", ZapJsonable("event", buf))

Correctly log protobuf messages as unescaped JSON with zap logger

2 Answers2