I'm setting up a proof of concept featuring two ASP.NET Core applications that are both instrumented with Jaeger to demonstrate how it can propagate a trace between services over the wire. Both applications are being deployed to Azure App Services.
I'm using the OpenTracing Contrib package to automatically inject the Jaeger trace context into my inter-service traffic in the form of HTTP Headers (the package is hardcoded to use that form of transmission). But it appears that those headers are going missing along the way, as the receiving application is unable to resume the tracing context.
Before deploying to Azure, I'm testing the applications locally with Docker Compose, and with that setup the context propagation works fine. It's only once the apps are in Azure that things break.
The applications communicate over HTTPS and I've disabled HSTS and HTTPS redirection in case that might be causing Azure to drop the headers, based on the answer in this previous thread.
I've also tried running both applications in Azure Container Instances, and that seems to be a non-starter - it doesn't fix the context propagation and seems to introduce more bugs around span relationships.
The two applications are nearly identical in their setup, and differ only in the API endpoints they serve.
My CreateWebHostBuild from program.cs:
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.ConfigureServices(services =>
{
// Registers and starts Jaeger (see Shared.JaegerServiceCollectionExtensions)
services.AddJaeger(CheckoutConfiguration.JaegerSettings.Host);
// Enables OpenTracing instrumentation for ASP.NET Core, CoreFx, EF Core
services.AddOpenTracing();
});
The contents of the AddJaeger extension method which is largely borrowed from the Contrib sample:
public static IServiceCollection AddJaeger(this IServiceCollection services, string jaegerHost = "localhost")
{
if (services == null)
throw new ArgumentNullException(nameof(services));
services.AddSingleton<ITracer>(serviceProvider =>
{
string serviceName = Assembly.GetEntryAssembly().GetName().Name;
ILoggerFactory loggerFactory = serviceProvider.GetRequiredService<ILoggerFactory>();
ISampler sampler = new ConstSampler(sample: true);
var reporter = new RemoteReporter.Builder()
.WithSender(new UdpSender(jaegerHost, 6831, 0))
.Build();
ITracer tracer = new Tracer.Builder(serviceName)
.WithLoggerFactory(loggerFactory)
.WithReporter(reporter)
.WithSampler(sampler)
.Build();
GlobalTracer.Register(tracer);
return tracer;
});
var jaegerUri = new Uri($"http://{jaegerHost}:14268/api/traces");
// Prevent endless loops when OpenTracing is tracking HTTP requests to Jaeger.
services.Configure<HttpHandlerDiagnosticOptions>(options =>
{
options.IgnorePatterns.Add(request => jaegerUri.IsBaseOf(request.RequestUri));
// We don't need to track Prometheus scraping requests
});
services.Configure<AspNetCoreDiagnosticOptions>(options => {
// We don't need to trace Prometheus scraping requests
options.Hosting.IgnorePatterns.Add(context => context.Request.Path.Equals("/metrics", StringComparison.OrdinalIgnoreCase));
});
return services;
}
My startup.cs configure method to show I'm not doing anything weird with the headers (the metrics extensions are for prometheus-net)
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.UseHttpMetrics();
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
// Do release exception handling
}
app.UseMetricServer();
app.UseMvc();
}
I expect any calls from one application to the other to propagate the active Jaeger trace context. Instead, the two applications log their traces separately and no link can be discerned between them in the Jaeger UI.
Here's a screenshot of a trace that should have spanned both services, but instead only shows spans from the first service:
