1

Here is python script that sends test trace directly to tempo instance :

import base64
from typing import Tuple

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import \
    OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Tracer, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def create_span_exporter_for_tempo() -> Tuple[Tracer, BatchSpanProcessor]:

    resource = Resource(attributes={"service.name": constants.APP_NAME})

    headers = {
        "Authorization": f"Basic {base64.b64encode(f'{constants.GRAFANA_USERNAME}:{constants.GRAFANA_PASSWORD}'.encode('ascii')).decode('ascii')}"
    }
    tempo_exporter = OTLPSpanExporter(endpoint=constants.TEMPO_ENDPOINT, headers=headers)

    trace.set_tracer_provider(TracerProvider(resource=resource))
    tracer = trace.get_tracer(__name__)
    span_processor = BatchSpanProcessor(tempo_exporter)

    trace.get_tracer_provider().add_span_processor(span_processor)
    return tracer, span_processor

tracer, span_processor = create_span_exporter_for_tempo()
with tracer.start_as_current_span("test") as span:
        print("test!")
        span.set_attribute("check", "works")

My Grafana Cloud setup looks like this : enter image description here

Here are some of variations of tempo endpoint tried :

1. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo
2. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/traces
3. https://tempo-prod-04-prod-us-east-0.grafana.net/v1/traces
4. https://tempo-prod-04-prod-us-east-0.grafana.net:443/tempo/v1/traces
5. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/trace

All of these endpoint return the same output :

Failed to export batch code: 404, reason: 404 page not found

enter image description here

Please note that, I have tested out this tempo instance by setting up grafana agent & I was able to send traces via grafana agent.

My requirement is to send traces without deploying grafana agent as I am sending traces from lambda & do not want to run agent somewhere else 24X7.

Here is python library being used

Utsav Chokshi
  • 1,357
  • 1
  • 13
  • 35

1 Answers1

2

You have to use OTLP Gateway. Make note of the following values that you’ll need in order to write OLTP:

  • OTLP Endpoint URL: https://otlp-gateway-<zone>.grafana.net/otlp (ex: https://otlp-gateway-prod-us-central-0.grafana.net/otlp).
  • Basic Auth Username: <instanceID from Grafana details page>
  • Basic Auth Password: <Grafana API Key>

Doc: https://grafana.com/docs/grafana-cloud/data-configuration/otlp/send-data-otlp/

Jan Garaj
  • 25,598
  • 3
  • 38
  • 59
  • For anyone stumbling across this: If you want to send traces directly from the code as in the example above, append `/v1/traces` to the endpoint mentioned above, so `https://otlp-gateway-.grafana.net/otlp/v1/traces` (see https://community.grafana.com/t/opentelemetry-endpoint-of-grafana-cloud/85359/5) – moertel Aug 20 '23 at 12:38
  • @jan-garaj, @moertel : Now this is not working. I have been trying to send traces to instance setup using new grafana cloud account and it keeps giving me `authentication error: invalid authentication credentials`. URL I am using is : `https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces` I tried cloud policy token as well as api key both. Any idea if something has been changed ? – Utsav Chokshi Aug 21 '23 at 20:34
  • @UtsavChokshi `authentication error: invalid authentication credentials` is not a problem with URL, but with used credentials. You have wrong authentication, e.g. wrong `Basic Auth Username` and/or `Basic Auth Password`. Only you have access to your credentials/org, so only you can find what is wrong. – Jan Garaj Aug 21 '23 at 21:31
  • @JanGaraj That's the problem. The code I have put it in question work for one grafana instance and does not work with another one. I have checked my credentials so many times. It is correct. I thought maybe the way auth works at grafana has changed with new instances. – Utsav Chokshi Aug 21 '23 at 21:59
  • @UtsavChokshi that OTLP gateway doesn't have SLA. But you can still complain to your Grafana cloud support. – Jan Garaj Aug 21 '23 at 22:11
  • @UtsavChokshi it's hard to tell what might be giving you the credentials error. I can at least confirm that the process as such is working for me. Things to check are whether your "prod-us-east-0" is indeed the right location and whether you accidentally included a newline `\n` in your base64 encoded credentials. For future reference, if anyone wants working Python code: https://gist.github.com/moertel/32ed043d77268d17c28c4c285136bff9 – moertel Aug 23 '23 at 16:41