How to pass remote parent span properly using NATS?

Question

I have a dummy example on this repo

I tried to pass current span context to a remote one so it would show the trace properly, what I've done:

go func() {
    _, span := otel.Tracer("natsC").Start(context.Background(), "publish")
    defer span.End()

    // send current span context as header
    spanCtx := span.SpanContext()
    spanJson, _ := spanCtx.MarshalJSON()
    log.Println(string(spanJson))
    msg, err := nc.RequestMsg(&nats.Msg{
        Subject: topic1, Data: []byte("whatever"), Header: nats.Header{
            "otelTrace": []string{string(spanJson)},
        },
    }, 2*time.Second)
    if L.IsError(err, `nc.Publish`) {
        return
    }
    log.Println(`reply:`, msg)
}()

On the receiver server:

_, err = nc.QueueSubscribe(topic1, "my-queue", func(msg *nats.Msg) {
    // take header and deserialize back to spanContext
    rsc := msg.Header.Get(`otelTrace`)
    parentSpanCtx := trace.SpanContext{}
    err := json.Unmarshal([]byte(rsc), &parentSpanCtx)
    L.IsError(err, `json.Unmarshal`)

    // use remote context as parent context
    _, span := otel.Tracer(`natsC`).Start(trace.ContextWithRemoteSpanContext(context.Background(), parentSpanCtx), topic1)
    defer span.End()

    data := string(msg.Data)
    fmt.Println(data)
    err = msg.Respond(msg.Data)
    L.IsError(err, `msg.Respond`) // ignore error
})

then I run it using this command go run main.go natsC.

both span shown on Jeager (localhost:16686) as separate spans, not correlated like in http/grpc example, what should I modify so it would considered as child span of the parent?

equivalent http/grpc example:

score 5 · Accepted Answer · answered Nov 17 '22 at 14:21

go.opentelemetry.io/otel/trace@v1.11.1/trace.go has the definitions for the context right, you can call MarshalJSON and have it spit something that looks useful but here's the thing. There is no equivalent unmarshalling function and the output is a string while the internal format is a fixed length byte array...

So to get it to work just dump the trace and span IDs into whatever format you like:

// Attach telemetry headers
headers := nats.Header{}
headers.Set(otelTraceID, span.SpanContext().TraceID().String())
headers.Set(otelSpanID, span.SpanContext().SpanID().String())

Then on the receive side you have to rebuild it manually into a SpanContext:

func getParentContext(msg *nats.Msg) (spanContext trace.SpanContext, err error) {
    var traceID trace.TraceID
    traceID, err = trace.TraceIDFromHex(msg.Header.Get(otelTraceID))
    if err != nil {
        return spanContext, err
    }
    var spanID trace.SpanID
    spanID, err = trace.SpanIDFromHex(msg.Header.Get(otelSpanID))
    if err != nil {
        return spanContext, err
    }
    var spanContextConfig trace.SpanContextConfig
    spanContextConfig.TraceID = traceID
    spanContextConfig.SpanID = spanID
    spanContextConfig.TraceFlags = 01
    spanContextConfig.Remote = true
    spanContext = trace.NewSpanContext(spanContextConfig)
    return spanContext, nil
}

Then actually use it:

remoteCtx, err := getParentContext(msg)
if err != nil {
    logrus.Fatal(err)
}

_, span := otel.Tracer(fqpn).Start(trace.ContextWithRemoteSpanContext(context.Background(), remoteCtx), msg.Subject)
defer span.End()

How to pass remote parent span properly using NATS?

1 Answers1