We are having an issue with identifying a problem with the serilog elastic search sink library. We end up with a stack trace and error message that provides nothing meaningful. I was able to test with different credentials and everything worked. However the main issue is not having proper error messages to make troubleshooting/bugfixing more efficient.
Stack trace one, uploading to elastic search in bulk.
2023-08-18T13:49:23.1060981Z Caught exception while preforming bulk operation to Elasticsearch: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute
at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 196
at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 179
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 61
at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\ElasticLowLevelClient.cs:line 58
at Elasticsearch.Net.ElasticLowLevelClient.Bulk[TResponse](PostData body, BulkRequestParameters requestParameters) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\ElasticLowLevelClient.Generated.cs:line 26
at Serilog.Sinks.Elasticsearch.ElasticsearchSink.EmitBatchChecked[T](IEnumerable`1 events)
at Serilog.Sinks.Elasticsearch.ElasticsearchSink.EmitBatch(IEnumerable`1 events)
Stack trace two, template setup
2023-08-18T13:49:20.9371951Z Failed to create the template. Elasticsearch.Net.ElasticsearchClientException: Request failed to execute
at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 196
at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 179
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 61
at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\ElasticLowLevelClient.cs:line 58
at Elasticsearch.Net.ElasticLowLevelClient.IndicesExistsTemplateForAll[TResponse](String name, IndexTemplateExistsRequestParameters requestParameters) in C:\Users\russc\source\git\elasticsearch-net-master\src\Elasticsearch.Net\ElasticLowLevelClient.Generated.cs:line 1202
at Serilog.Sinks.Elasticsearch.ElasticsearchSinkState.RegisterTemplateIfNeeded()
Code to capture/output errors by the sink explicitly.
Serilog.Debugging.SelfLog.Enable(Console.Error);
ElasticsearchSinkOptions elasticSearchSinkOptions = new(new Uri(serviceSettings.Aws.ElasticSearch.Url))
{
IndexFormat = serviceSettings.Aws.ElasticSearch.Index,
FailureCallback = e =>
{
if (e.Exception != null)
{
Console.Error.WriteLine($"{e.RenderMessage()}");
Console.Error.WriteLine(e.Exception);
}
else
{
Console.WriteLine(e.RenderMessage());
}
},
EmitEventFailure = EmitEventFailureHandling.RaiseCallback | EmitEventFailureHandling.WriteToSelfLog,
TypeName = "_doc",
AutoRegisterTemplate = true,
InlineFields = false,
MinimumLogEventLevel = defaultLogEventLevel
};