Azure Search SDK null field not set on Merge action

Question

I'm using Microsoft.Azure.Search version 3.0.1,

I'm trying the following:

// subset of my index's fields
private class SyncFields
{   
    public string Id { get; set; }
    public DateTimeOffset? ApprovedOn { get; set; }
    public DateTimeOffset? IgnoredOn { get; set; }
}

public void Sync()
{
    var sync = new SyncFields
    {
        Id = "94303",
        ApprovedOn = null,
        IgnoredOn = DateTime.UtcNow
    };

    var searchClient = new SearchServiceClient("xxxx",
        new SearchCredentials("xxxx"));
    searchClient.SerializationSettings.NullValueHandling = NullValueHandling.Include;

    using (var client = searchClient.Indexes.GetClient("xxxx"))
    {
        client.SerializationSettings.NullValueHandling = NullValueHandling.Include;
        var batch = IndexBatch.Merge<SyncFields>(new[] { sync });
        client.Documents.Index<SyncFields>(batch);
    }
}

This isn't settings ApprovedOn to null. It ignores it. If I set a non-null value, it does set it.

According to the documentation here the merge operation updates the field to be null. And in fact, if I make this Http post request manually with JSON, this is true. But the SDK isn't updating the field(s) to null. What am I missing?

score 4 · Accepted Answer · answered Dec 13 '16 at 19:56

4

This is a known limitation of the typed overloads of the Index family of methods. The issue is described in detail here: https://github.com/Azure/azure-sdk-for-net/issues/1804

Some workarounds:

Use the untyped version of Index instead for merge scenarios.
Use Upload instead of Merge.
Put [JsonProperty(NullValueHandling = NullValueHandling.Include)] on the properties of your model class that you need to explicitly set to null in a merge operation (not recommended if you have many fields in your index).
Implement a custom converter.

answered Dec 13 '16 at 19:56

Bruce Johnston

8,344
3
32
42

Thank you Bruce! I was unaware of the untyped version (using Document). I will use that instead of custom converters. I had also forgotten about the JsonPropertyAttribute. Can you say why it's not recommended to have the JsonProperty on every field? – Aaron B Dec 13 '16 at 23:28
Performance mainly. If you have a very wide schema (think 100s of fields), sending 100s of nulls for every document in an Index batch request is very wasteful of network bandwidth. – Bruce Johnston Dec 14 '16 at 03:11

Aaron B · Answer 2 · 2016-12-13T23:38:04.307

I found the culprit in the Azure Search SDK source.

Line 51, settings.NullValueHandling = NullValueHandling.Ignore; is overriding the setting I tried to set. I will probably be making an issue about this in Github.

For now, I am using a custom converter as a workaround.

public class DefaultDateTimeOffsetIsNullConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return (objectType == typeof(DateTimeOffset?));
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var date = (DateTimeOffset?)value;
        if (date == default(DateTimeOffset))
        {
            writer.WriteNull();
        }
        else
        {
            writer.WriteValue(date);
        }
    }

    public override bool CanRead => false;

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

As in

var sync = new SyncFields
{
    Id = "94303",
    ApprovedOn = default(DateTimeOffset), // set to null
    IgnoredOn = DateTime.UtcNow
};

// ...

client.SerializationSettings.Converters.Add(new DefaultDateTimeOffsetIsNullConverter());

// ...

Edit:

Two other superior options listed by Bruce: using Document which is untyped, and using the JsonPropertyAttribute on the field to get the correct serialization. Using Document is ideal for my use case, no serialization problem or custom converters:

var sync = new Document
{
    ["Id"] = "94303",
    ["ApprovedOn"] = null,
    ["IgnoredOn"] = null
};

// ... the same as before:
var batch = IndexBatch.Merge(new[] { sync });
await client.Documents.IndexAsync(batch);

Azure Search SDK null field not set on Merge action

2 Answers2