Vespa: Can we aggregate on nested fields?

Question

In search definition the fields inside struct can not have "attribute" indexing.

http://docs.vespa.ai/documentation/reference/search-definitions-reference.html#field_types

Also, struct and maps are not attribute by default. Resulting search definition would look something like this:

struct nlp {
  field token type string {
      match: text //can't add indexing here
  }
}

field n type nlp {
      indexing: summary //can't add attribute here
}

How to add search definition such that we can group by "n.token"? Is it possible to add attribute or indexing for struct fields? Or group by on fields which is not attribute?

score 5 · Accepted Answer · answered Oct 24 '17 at 19:08

Struct field type cannot have attribute which is necessary pre-requisite if you want to run grouping with indexed search, see http://docs.vespa.ai/documentation/reference/search-definitions-reference.html#struct

The only thing you can really do with struct fields with mode=index is to have them as part of the summary (response). You could add a custom searcher which does the aggregation over the struct field analyzing the top K retrieved hits. See http://docs.vespa.ai/documentation/searcher-development.html

With mode=streaming you can run grouping over struct fields, more about streaming here http://docs.vespa.ai/documentation/streaming-search.html

score 1 · Answer 2 · answered Oct 25 '17 at 07:01

1

can't comment yet hence posting as answer.

@jkb does this amount to us running a search on some field not in a struct, returning a lot of documents and then synthesising them in a searcher (chaining a searcher would essentially do similar stuff)?

is it also correct to say that nested fields in a document cannot be indexed and in-turn effectively searched (i don't want to use streaming search), and hence the structure must always be flat for index search to work? i can achieve a flat structure in most cases but what about fields having array of objects where i'd want to search on some attribute of such an object?

"x" : [
    object: { attributes},
    object: {attributes}
 ]

answered Oct 25 '17 at 07:01

shwetank

115
3
7

Your understanding is correct. Fields of type struct does not support indexing nor search in Vespa. – Jo Kristian Bergum Oct 25 '17 at 07:31
It does support search and grouping with mode=streaming but that is a mode which is for very specific use cases, e.g search in personal data where the qps per user is limited. – Jo Kristian Bergum Oct 25 '17 at 07:56
1

@jkb in order to flatten the nested document what is the best strategy we can use? the field name does not allow special characters. so, "a.b.c" or "a-b-c" or "a_b_c" is not a valid field name. any easy way to flat the nested objects? – enator Oct 25 '17 at 09:24
1

sorry, found that, underscore is allowed. – enator Oct 25 '17 at 09:59

Vespa: Can we aggregate on nested fields?

2 Answers2