In https://charity.wtf/tag/observability/ in the section HOW TO GATHER AND STRUCTURE YOUR DATA Charity Majors presents the following scheme:
- Early in the pipeline initialize a "blob" accessible everywhere in code with every request having its own copy (in effect tied to the
HttpContext) - Any code that wishes to record some information adds key-value pairs to this "blob"
- When the request exits the "blob" is delivered to the observability server as a structured event.
This scheme makes a lot of sense and is quite elegant. It avoids verbal diarrhea of the regular logs. However, it is also oversimplified, because it ignores the batch processing requests, where a request has to process N records which is likely to produce N sets of similar structured events nested within the parent structured event corresponding to the request itself. In this case we have a choice of either sending the batch record event as soon as a record is processed or accumulate the nested events within the "blob" to be sent when the request is over. In any case this complication is not discussed in the post.
Before potentially inventing the wheel I am trying to figure out whether we have something ready to use this observability scheme. Here is what I have so far:
- We are using Azure and its Application Insights TrackEvent API seems to be perfect as the low level API. It supports events of the form
<Event Name, string-to-string map, string-to-float map>which seems to be rich enough. The collected events can be then queried across all the dimensions anyway we like it. So, the observability server is not a problem for me. - The "blob" implementation is a problem. I was looking for Honeycomb implementation for .Net Core and came across this page - https://docs.honeycomb.io/getting-data-in/dotnet/. However the 2 options presented there do not seem mature enough:
- HoneyComb .NET leaves it up to us to figure out how to deal with nested events.
- OpenTelemetry .NET does not seem to be about observability at all, but about the "3 pillars of observability" - logging, metrics and tracing. Which do not give observability according to https://www.infoq.com/news/2019/02/rethinking-observability/
- The Microsoft.Extensions.Logging.ILogger interface seems to present an adequate interface. It looks like the BeginScope method exists explicitly to delimit the different batches. But it is just an interface.
So, my question is there an implementation for Asp.Net Core for observability as structured events that also takes into account the nested events in one way or another and that can be customized to deliver to the Application Insights bucket?
I could elaborate on the challenges of the batch processing, but this question is already too long, so I stop here.
