I've recently inherited ownership of some search code used by our application, and am working on cleaning up some changes by the last developer. We were hitting issues trying to query for sorted lists of documents when I started digging through the index structure using PostMan. I happened upon something that changed from the last version of our index structure to the new on after changes. The tree structure for holding our dynamically generated values changed and now contains the same two headers twice. Here's an example of what I mean:
The original structure:
"_doc": {
"properties": {
*some other properties*
"values": {
"properties": {
"field_10": {
"type": "text"
},
"field_11": {
"type": "text"
},
*more dynamically generated fields*
}
}
}
}
And the new structure, the only difference being the repeated values:
"_doc": {
"properties": {
*some other properties*
"values": {
"properties": {
"values": {
"properties": {
"f_10": {
"type": "text"
},
"f_11": {
"type": "text"
},
*more dynamically generated fields*
}
}
}
}
}
}
I believe these headers are what's causing the issue when we try to get a sorted list. The error handed back from NEST is the following when I send a query to sort on field f_5.
Invalid NEST response built from a unsuccessful low level call on POST: /td_3dfb600bfd4140d4bd229a356496aca4_documents/_search?typed_keys=true
# Audit trail of this API call:
- [1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.0150146
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (400) Bad Request.. Call: Status code 400 from: POST /td_3dfb600bfd4140d4bd229a356496aca4_documents/_search?typed_keys=true. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
--- End of inner exception stack trace ---
# Request:
{"from":0,"query":{"bool":{"must":[{"term":{"formTypeProductId":{"value":1}}},{"bool":{"minimum_should_match":1,"should":[{"term":{"activeStep":{"value":5}}}]}},{"bool":{"minimum_should_match":1,"should":[{"term":{"documentStatus":{"value":1}}},{"term":{"documentStatus":{"value":0}}},{"term":{"documentStatus":{"value":2}}}]}}]}},"size":100,"sort":[{"values.f_5.keyword":{"missing":"_last","order":"asc"}},{"documentId":{"order":"asc"}}]}
# Response:
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"No mapping found for [values.f_5.keyword] in order to sort on","index_uuid":"Fz6gWcm4TdGmh1bDAspjBg","index":"td_3dfb600bfd4140d4bd229a356496aca4_documents"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"td_3dfb600bfd4140d4bd229a356496aca4_documents","node":"uM9FlJg1SuCN4-gZdeDHig","reason":{"type":"query_shard_exception","reason":"No mapping found for [values.f_5.keyword] in order to sort on","index_uuid":"Fz6gWcm4TdGmh1bDAspjBg","index":"td_3dfb600bfd4140d4bd229a356496aca4_documents"}}]},"status":400}
The code we use to query hasn't changed, and worked perfectly previously. I believe the added Index tree objects are what's causing the client to be unable to find a valid mapping.
So far, I've tried modifying our call to NEST's .CreateIndex() method, in which we use .AutoMap to pull the valid document information from a class, SearchDocument.cs. I've also tried modifying the SearchDocument class itself, as this is where the recent changes occurred, but none of that has made a difference.
The calls within CreateIndex are done using the following:
var createResponse = client.CreateIndex(
index,
c => c
.Settings(
s => s
.NumberOfShards(1)
.NumberOfReplicas(0))
.Mappings(
ms => ms
.Map<SearchDocument>(
m => m
.DateDetection(false)
.NumericDetection(false)
.AutoMap(new ValuesVisitor())
.Properties(
properties => properties
*Some other mappings*
.Object<Dictionary<string, string>>(obj => obj
.Name(name => name.Values)
)
Meanwhile, the SearchDocument.cs class uses the following method to populate the members used by AutoMap to generate the values structures. We have 2: Values, which holds strings, and ValuesAsNumber, which holds decimals. The changes recently were to add this ValuesAsNumber property and add it to the index.
private void BuildValuesFields(IEnumerable<MetaDataValue> metaDataValues, IEnumerable<TableValue> tableValues)
{
Values = new Dictionary<string, string>();
ValuesAsNumber = new Dictionary<string, decimal?>();
if (metaDataValues != null)
{
foreach (var value in metaDataValues)
{
var field = _fields.GetOrDefault(value.FieldId, null);
var processedValue = PrepareValue(value.TextValue);
Values[SearchDocumentFields.GetValueFieldName(value.FieldId)] = processedValue;
// if field is Number or Date, we store a numerical representation for sorting
if (field != null && (field.DataType == FieldType.Number || field.DataType == FieldType.Date))
{
ValuesAsNumber[SearchDocumentFields.GetValueFieldAsNumberName(value.FieldId)] =
MetaDataValue.ConvertToDecimal(processedValue);
}
}
}
if (tableValues is null)
{
return;
}
foreach (var value in tableValues)
{
var fieldName = SearchDocumentFields.GetValueFieldName(value.FieldId);
if (!Values.ContainsKey(fieldName))
{
Values[fieldName] = string.Empty;
}
Values[fieldName] += $"{PrepareValue(value.TextValue)} ";
}
}
This is called in the SearchDocument class contructor to populate the properties as we instantiate the class.
Does anyone have any advice on places to look for where these extra Index objects are getting created? I've been unable to figure it out.
