Open Search Server is crashing while crawling files. OSS is running as a daemon on an Ubuntu box. This is a production server with 64gb ram and 12 cores, crawling files on an extremely fast nas that it mounts, about 20 gb of files. 2gb memory allotted for OSS. The largest file that should get crawled is about 1.3gb. There are 5 mp4 files that are all over 1gb.
Usually at some point during the crawl process, OSS will become completely unresponsive. Restarting OSS fixes the problem. Today I monitored a crawl, which usually uses one or two cores at a time. When it crashed it was maxing out all 12 cores. Total memory usage on the server was fine, but I'm not sure how much OSS was using.
We've looked at the oss log files and there's not a single error that happens before each crash, but there are two errors that are pretty common in the logs:
WARN: org.apache.cxf.jaxrs.utils.JAXRSUtils - Both com.jaeksoft.searchlib.webservice.crawler.database.DatabaseImpl#run and com.jaeksoft.searchlib.webservice.crawler.database.DatabaseImpl#run are equal candidates for handling the current request which can lead to unpredictable results
WARN: root - Low memory free conditions: flushing crawl buffer
We have one index that handles all files. It is based on the file crawler template—the only changes are:
- An extra analyzer that uses 4 regex replaces.
- An extra field that copies the url field and uses the analyzer from
- We added one disk location, which has all the files.
- We join another index in our query.
When we are able to crawl, querying the index works fine afterwards. I think maybe the crashes only happen if there's a search query on the index during the crawl, but haven't been able to confirm that yet.
