Elasticsearch indices eating too much space

Question

I'm using Elasticsearch 7.5.2 on Ubuntu. Recently, I began using Elasticsearch to display relevant search results on every page load. This shot up the volume, but I also found out that it has created large index files. Note that I'm using 'app-search' to power my queries.

Here's the sample index files that are occupying too much space:

.app-search-analytics-logs-loco_togo_production-7.1.0-2020.01.26 => 52 GB
.app-search-analytics-logs-loco_togo_production-7.1.0-2020.01.27 => 53 GB

I tried deleting these using CURL, but they reappear and show lesser space (~5 GB each).

I want to know if there is a way to control these indexes. I'm not sure what purpose do these indices solve and if there is a way to prevent them?

Answer 1

I tried deleting these using CURL, but they reappear and show lesser space (~5 GB each).

Obviously your delete-action was executed. It seems like that the indices still get written to. If documents still get into elasticsearch, the index gets re-created.

So for example:

The index from 2020.01.27 has 53 GB before the deletion. After you delete it, the data is gone and the index itself too. But as soon as new documents of the very same day (2020.01.27) get indexed, the index gets re-created containing the documents after the deletion which is probably the 5GB.

If this is not what you want, you need to check if there are some sources still sending data.

Hope this helps.

EDIT:

Q: However, is there a way to manage these indices? I don't want them to eat up too much space.

Yes! Index Lifecycle Management (ILM) is what you are looking for. It aims to automate the maintenance/management of indices. So for example you could define a rollover every 30GB to a new index in order to keep them small. Another example is to delete the index after X days. Take a look at all the phases and actions.