1

I have an elasticsearch node with following specs:

Disk indices and disk used in elasticsearch

I have deleted several documents from the index but my disk space doesn't seem to free up. The screenshot above shows that the actual documents in Elasticsearch only occupies 1.5GB as shown by disk.indices whereas the disk.used is 73.6GB. I have found that just deleting the document doesn't remove the document from elasticsearch, but only marks it as deleted. I have tried freeing disk space by using forcemerge as mentioned in this answer, but still my disk space shows the same.

How do I permanently delete documents from elasticsearch and free up my disk space?

Kshitij Bajracharya
  • 811
  • 2
  • 14
  • 37
  • What do you get when running `du -h` from the root folder of your host? Can you see where the disk space is being taken? – Val Aug 12 '20 at 07:54
  • I can see that `.` folder has occupied 74GB, everything else is in MB or KB. – Kshitij Bajracharya Aug 12 '20 at 09:22
  • Can you run this to only return folders which contains GB of data `du -h | grep -e '^\d*,\d*G'` ? – Val Aug 12 '20 at 09:27
  • please check if forcemerge is actually happend. _GET '/_cat/segments/{index_name}'. there should be 1 segment per shard if you run forcemerge correctly. – hamid bayat Aug 12 '20 at 09:42
  • and check _cat/indices deleted doc column. – hamid bayat Aug 12 '20 at 10:02
  • @Val I coundn't run the command. Got something like `du: cannot access './proc/6716/task/6716/fd/4': No such file or directory` in response – Kshitij Bajracharya Aug 12 '20 at 10:16
  • @hamidbayat `_cat/indices` gave me the same 1.5GB. I couldn't understand all of the response from `_GET '/_cat/segments/{index_name}'` but the sum of the space occupied here is roughly around 1.5GB – Kshitij Bajracharya Aug 12 '20 at 10:20
  • One thing I forgot to mention is that when I run **forcemerge**, I get `{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}` in response. Then I had to run `curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'` first and then run **forcemerge** again to get rid of the error. Does this have anything to do? – Kshitij Bajracharya Aug 12 '20 at 10:23
  • https://stackoverflow.com/questions/55546176/clustor-block-expection/55547430#55547430 – hamid bayat Aug 12 '20 at 10:24
  • then do forcemerge again. and check in doc_deleted column _cat/indices you have zero doc. – hamid bayat Aug 12 '20 at 10:25
  • @hamidbayat this is what I got in response after following your steps: `yellow open {index_name} {uuid} 5 1 58279 1119 1.5gb 1.5gb` – Kshitij Bajracharya Aug 12 '20 at 10:35
  • 2
    You cannot forcemerge because you don't have enough space on your disk. The problem is not in ES but elsewhere on your hard disk space is taken by something else than ES data. Once you find out, you can delete it and ES will work properly again – Val Aug 12 '20 at 10:45
  • Check in `/var/log/elasticsearch` if you don't have too many log files that have been stacking up... – Val Aug 12 '20 at 10:47

2 Answers2

1

first of all you should free some space. check in '.' folder and see what is in it?

try du -sh *

and then:

as _cat/indices response you have 1119 deleted document. and according to error you should first reverse read_only_indices:

Clustor block expection

then run this command:

curl  -H'Content-Type:application/json' -XPOST localhost:9200/{AN-INDEX-NAME}/_forcemerge?max_num_segments=1
hamid bayat
  • 2,029
  • 11
  • 20
  • 1
    forcemerge won't work because there's not enough disk space. The solution is outside ES reach. The OP has something taking a lot of space on the disk and needs to be removed – Val Aug 12 '20 at 10:46
  • 1
    @val yes. you are right . I think he should first look in "." folder – hamid bayat Aug 12 '20 at 10:47
  • He needs to find the right folder as I think `.` was the root folder – Val Aug 12 '20 at 10:48
  • `.` simply means the current folder, wherever the OP is located in the filesystem tree. – Val Aug 12 '20 at 10:50
  • thank you so much @Val and hamidbayat for your help. I found out that logs in my tomcat was occupying around 50GB of space. – Kshitij Bajracharya Aug 12 '20 at 11:17
1

Something on that host is taking up space, but the problem is not related to ES data because it's almost impossible to have such a small ratio between disk.indices and disk.used, as the merging process would take care of freeing up space as frequently as possible.

It turns out that it could be log files located somewhere on your disk that have not been properly rotated and have been accumulating for months.

Val
  • 207,596
  • 13
  • 358
  • 360