What data is stored inside of a Couchbase Index?

Question

Situation

I'm working w/ a Couchbase database that keeps running into OOM issues, a lot of the time it's with the Index Service running out of memory. When I fire up the Couchbase dashboard to get more info I can see that index foobar has X amount of items that take up Y amount of memory.

Question

Is there a way to view the data stored in the index service? Is it a clone of the document in a new list? Is a pointer filter + a list of references of foo documents? What is an "item"?

I've combed through the docs and the closest I've come is here where they talk about the "items" in the dashboard table, but they don't actually define what the items actually are.

Using Node.js Package couchbase: "2.6.12"

Answer 1

Is there a way to view the data stored in the index service?

Yes you can use one of the dump utilities in the Couchbase bin directory.. For example the plasma_dump tool is used to extract data from a index on Couchbase Server Enterprise Edition 7.x.

Here's an example extracting the index data from an index created when the built-in travel sample database is loaded into a Linux cluster.

/opt/couchbase/bin/plasma_dump dump /opt/couchbase/var/lib/couchbase/data/\@2i/travel-sample_def_airportname_4970026472206047478_0.index/mainIndex/


{"k":Raleigh Durham Intlairport_3626
                                    ","v":""},
{"k":Ralph Wien Memairport_3693
                               ","v":""},
{"k":Ramona Airportairport_8608
                               ","v":""},
{"k":Rampart Airportairport_7112
                                ","v":""},
{"k":Rancho Murietaairport_3643
                               ","v":""},
{"k":Rancho San Simeon Airportairport_9104
                                          ","v":""},
{"k":Randall Airportairport_8531
                                ","v":""},
{"k":Randolph Afbairport_3757
                             ","v":""},
{"k":Rapid City Regional Airportairport_4087
                                            ","v":""},
{"k":Rawlins Municipal Airport-Harvey Fieldairport_7986
                                                       ","v":""},
{"k":Reading Regional Carl A Spaatz Fieldairport_5764
                                                     ","v":""},
{"k":Red Bluff Municipal Airportairport_8137.....

Is it a clone of the document in a new list? Is a pointer filter + a list of references of foo documents? What is an "item"?

As you can see, each index item is a key and a value of the actual fields your index definition has defined. It is not a pointer.

This is actually how indexes are used in databases, in order to speed up the queries. You typically will want to include the document fields in the index to satisfy your queries but exclude the fields which aren't required to reduce the size of the index.

In database terminology, a covered index is an index that includes all the fields that are needed to satisfy a query, so that the database engine does not need to look up the actual data fields from the data service. This can improve the performance of the database, as looking up data in an index is generally faster than gathering all the raw data from the data service.

In order to be a covered index, an index must include all the fields that are needed to satisfy the query. If the query requires fields that are not included in the index, the database engine will have to look up the actual data in the bucket to retrieve the necessary data. This can reduce the performance benefits of using an index.

Thanks, Ian McCloy (Couchbase Product Manager)