Count of documents 0 after inserting data with Nest

Question

I am using Nest with the following connection settings:

    var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
    var settings = new ConnectionSettings(connectionPool, new InMemoryConnection());
    settings.DisableDirectStreaming(true); // needed to see good looking debug log on insert
    settings.DefaultIndex(Index);
    Client = new ElasticClient(settings);

With new InMemoryConnection() I hope to query with Nest - changing data inside an Azure Cloud function.

Strangely the debug logs look promising Indexing:

    /*
    var res = await Client.IndexManyAsync(response.Elements, Index); //
    Console.WriteLine(res.DebugInformation);
    */
    /*
    var res = await Client.IndexAsync(response, i => i.Index(Index)); // Index = "data"
    Console.WriteLine(res.DebugInformation); // <--
    */

And logging directly after the insertions the count is 0:

// var anyDocs = await Client.CountAsync<OverpassElement>(c => c.Index(Index));
   var anyDocs = await Client.CountAsync<OverpassElement>(c => c);
   Console.WriteLine("count: " + anyDocs.Count);

..but the entire json data being logged with the insertion.

How come i can't count it (so that I can search in a next step), after insertion?

Actually I get:

Invalid NEST response built from a successful (200) low level call on POST: /data/_doc

And there is 0 Items in the on the IndexResponse inserting.

The data is of Element looking like the following part of an array containing 4221 such items:

{
    "type": "relation",
    "id": 8353694,
    "timestamp": "2018-06-04T22:54:27Z",
    "version": 1,
    "changeset": 59551528,
    "user": "asdf2",
    "uid": 1416503,
    "members": [
      {
        "type": "way",
        "ref": 89956942,
        "role": "from"
      },
      {
        "type": "node",
        "ref": 1042756547,
        "role": "via"
      },
      {
        "type": "way",
        "ref": 89956938,
        "role": "to"
      }
    ],
    "tags": {
      "restriction": "no_left_turn",
      "type": "restriction"
    }
  },

score 1 · Answer 1 · answered Mar 15 '21 at 14:53

ElasticSearch has many similarities to a NoSql data store. In this case, "read after write" is not guaranteed by default. When the index API call returns success, it doesn't mean "this document is now available for searching"; it means "ElasticSearch has accepted your document and it will be available for searching shortly". ElasticSearch uses eventual consistency by default.

However, this can be annoying during testing. So ElasticSearch has a Refresh API that essentially just blocks until all documents already indexed are available for searching. I strongly recommend that you do not call this in production; only in test code.

I am essentially looking for insert, make search, clear data - flow — lolelo, Mar 15 '21 at 16:30

score 0 · Answer 2 · answered Jan 19 '22 at 08:07

As the risk of reviving an old question, this answer from Russ Cam explains that InMemoryConnection does not actually run the operation against Elasticsearch.

InMemoryConnection doesn't actually send any requests or receive any responses from Elasticsearch; used in conjunction with .SetConnectionStatusHandler() on Connection settings (or .OnRequestCompleted() in NEST 2.x+), it's a convenient way to see the serialized form of requests.

So you can inspect the query that NEST generates from your code but you won't be able to observe the results.

score -1 · Answer 3 · answered Mar 14 '21 at 17:48

-1

I don`t know what Nest is, but I'd bet 100$ that if it use Transactional concepts, maybe you should commit it in order to see count correctly ?

answered Mar 14 '21 at 17:48

Laurent Morissette

46
1
6

Count of documents 0 after inserting data with Nest

3 Answers3