3

I am trying to export metrics and traces from my Akka app written in Scala using OpenTelemetry agent with the purpose of consuming the data in OpenSearch.

Technology stack for my application:

  • Akka - 2.6.*
  • RabbitMQ (amqp client 5.12.*)
  • PostgreSQL (jdbc 42.2.*)

I've added OpenTelemetry instrumentation runtime dependency to build.sbt:

  val runtimeDependencies: Seq[ModuleID] = Seq(
    "io.opentelemetry.instrumentation" % "opentelemetry-instrumentation-api" % otelInstrumentationVersion % "runtime"
  )
...
  libraryDependencies ++= compileDependencies ++ testDependencies ++ runtimeDependencies,

I am passing OpenTelemetry configurations in a properties file:

export JAVA_OPTS="... \
-javaagent:lib/opentelemetry/opentelemetry-javaagent-all-v1.6.0.jar \
-Dotel.javaagent.configuration-file=lib/opentelemetry/otel.properties"

The only other related piece in my code is the properties file:

otel.service.name=my-app
otel.traces.exporter=jaeger
otel.propagators=jaeger

I do receive some traces in OpenSearch, but they are disparate and unrelated whereas I would expect them to be linked. For example a message is received on RabbitMQ topic, it makes it's way into an actor, the latter eventually issues a SQL query. As a result I could see for each execution how much time did each step take.

This is an approximate view that I get in OpenSearch: enter image description here

I would love to be able to follow documentation, but I find that OpenTelemetry's configuration guide is scarce at this point.

Update:

Not sure whether this is relevant, but I get a warning on datapreper:

2021-09-29T16:50:50,861 [raw-pipeline-prepper-worker-5-thread-1] WARN  com.amazon.dataprepper.plugins.prepper.oteltrace.OTelTraceRawPrepper - Missing trace group for SpanId: 922097e31cf96c72
vasigorc
  • 882
  • 11
  • 22

1 Answers1

2

Ok so I got around by running across this issue and then reading about how to surpress specific instrumentations.

So to reduce clutter in tracing dashboard, one would add something as following to the properties file (or equivalent via environment variables):

otel.instrumentation.rabbitmq.enabled=false
otel.instrumentation.grpc.enabled=false

Note that I removed the two cluttering instrumentation libraries peculiar for my use case. For another application one wold choose other libraries from link # 2 above. In this way the spans that you as application developer declare will become roots.

vasigorc
  • 882
  • 11
  • 22