OpenTelemetry Export to Elastic Search without Elastic APM

Question

I'd like to export trace/metrics data to Elastic Search using OpenTelemetry, but I'd prefer to avoid Elastic APM. Is it possible? The opentelemetry contrib repo apparently suggests it is possible, however, I did not find anything on elastic.co documentation. By the way, openapm.io implies see here, OpenTelemetry can export to elastic beats (which is extremely desirable), but again, I did not find anything in Elastic.co docs.

Answer 1

When this question was asked OpenTelemetry standard and implementation was still quite young. Even as I write this answer some things are not quite fully implemented or settled yet but the TLDR; Yes, its possible, it works and its epic!

(original class was refactored - just browse the repo for example - its also a really good good package)

A good reference I go back to quite often can be found here

You need to understand there are three key things in Observability paradigm. Trace, Logs and Metrics. All these are separate, work and implemented separately.

The key findings that I have learnt so far to configure this in .NET Framework and AspNetCore are the same apart from how you configure it.

Trace and Metrics work with SDK Builders and Logging uses the ILoggerProvider pattern

These examples use the non DI methods for manual configuration. You can use Contrib.Hosting and read the guides there for DI approach.

Trace

System.Diagnostics.Activity.DefaultIdFormat = System.Diagnostics.ActivityIdFormat.W3C;

That is required in NetFramework apps in your global in NetCore that is default

var traceProviderBuilder = Sdk.CreateTracerProviderBuilder();

Gets your builder setup

traceProviderBuilder
    .AddSource("*")
    .SetResourceBuilder(
        ResourceBuilder.CreateDefault()
            .AddService(ResourceNameHelper.ServiceName)
            .AddAttributes(otelAttributes)
            .AddTelemetrySdk());

There's a few things happening here

• AddSource("*") subsribes the SDK to ALL Events (System.Diagnostics) - I use this for ease of use
• SetResourceBuilder is used to set the name of your service so that Elastics can group these
• OtelAttributes is a KeyValue collection that has new KeyValuePair<string, object>("deployment.environment", environmentName) in it so you can get the Environment drop down working in Elastics

That is the core configuration required so you can subscribe to your own Traces but you want to add Instrumentation afterwards like

• AddAspNetInstrumentation(options => { options.RecordException = true; })
• .AddHttpClientInstrumentation((httpConfig) => { httpConfig.RecordException = true; })
• AddSqlClientInstrumentation((sqlConfig) => { sqlConfig.EnableConnectionLevelAttributes = true; sqlConfig.SetDbStatement = true; })
• .AddAWSInstrumentation()

Now comes the import part, the OtlpExporter

traceProviderBuilder.AddOtlpExporter(options)

passing in your options

otlpExporter.Protocol = OtlpExportProtocol.Grpc;
otlpExporter.Endpoint = new Uri(otlpEndpoint);
otlpExporter.Headers = otlpHeaders;

The header is bearer token that you need from Kibana.

And finally you build it

.Build()

When manually configuring like this you need to manage the disposal of the SDK. That is why using DI is recommended.

Phew.. That is quite some configuration going on there. But once you get that running in the most basic form you will start to see Traces appear in the Observability part of Elastics. The first time you do this you need to give Elastics some time to sort out its indexing and trace streams (in the cloud version all that is done automagically given you have the NEW version of APM enabled in Fleet - Theres no mention of OpenTelemtry configuration on that page. just copy and paste the URLs and keys it gives you)

Metrics

The same applies for Metrics basically.. you have to do it all again but using

_meterProvider = Sdk.CreateMeterProviderBuilder().AddRuntimeMetrics()

And then all the instrumentation and OtlpExporter options.

In Framework the Metrics are quite limited abotu runtime metrics but in AspNetCore they are decent.

Logging

This is quite easy to implement following the Dotnet OpenTelemetry documentation but as of today is a bit buggy (Hopefully it will be stable in version 1.3)

---

I am sorry about long post but I hope this helps people a bit since I my self struggled with little or no documentation for this on Elastics, OpenTelemetry or the Dotnet Github pages.

Answer 2

After trying this for some time, I've finally made it to work with the latest version of opentelemetry-collector-contrib (0.60.0 at the time of this writting).

First I've created a multi-node cluster with Docker Compose using the steps here https://www.elastic.co/guide/en/elasticsearch/reference/8.4/docker.html#docker-compose-file. I also had to follow the instructions to set the vm.max_map_count to at least 262144.

Then, using the same net, I started an opentelemetry-collector-contrib docker image using a command like:

docker run --net certs_default --name opentelemetry-collector -v C:\docker\elastic\otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml -p 4317:4317 -p 4318:4318 otel/opentelemetry-collector-contrib:0.60.0

The configuration file is very simple:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug

  elasticsearch/trace:
    endpoints: [https://es01:9200]
    user: elastic
    password: 
    api_key: 
    tls:
      insecure_skip_verify: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [elasticsearch/trace]

Finally, I've just configured my application to use the OtlpExporter:

.AddOtlpExporter(configure =>
{
    configure.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
})

Doing these steps, I can see my traces in Kibana. Now it would be nice to have some template to display a single trace with all the spans.