Lucene.Net Parallel Search

Question

I am creating a Lucene.Net based search engine application in C# with a single index. As a requirement, I need to optimize runtime for a test run with multiple (5) queries. Therefore, I wanted to use a separate thread for each search, returning results similar to this post. My code looks like this:

// load information needs
List<InformationNeed> informationNeeds = FileReader.readInformationNeeds(collectionPath);             

// each search has its own thread referenced in this list
List<Thread> threadList = new List<Thread>();

// each search has its own result referenced in this list
List<SearchResult> searchResults = new List<SearchResult>();


foreach (InformationNeed informationNeed in informationNeeds){

    // create searchOptions
    SearchOptions searchOptions = new SearchOptions(DEBUG_pre_processQuery, informationNeed.getInput());

    // run search
    SearchResult result = null; // Used to store the return value
    var thread = new Thread(
       () =>
       {
       result = startSearch(searchOptions);
       Console.WriteLine("end search for IN nr " + informationNeed.getID() + "results = " + result);

       //add results to list
       searchResults.Add(result);

       });
    thread.Start();
    threadList.Add(thread);
}

// block main thread until all threads finished
foreach (Thread t in threadList){
    t.Join();
}

return searchResults;

However, I am getting a Lucene.Net.QueryParser.ParseException see screenshot that I do NOT get when running the searches sequentially.

Please comment if I let something unclear. I would appreciate any help on this issue.

You're not synchronizing access to `searchResults`. That's going to cause issues. — Millie Smith, Oct 21 '17 at 03:38

score 2 · Answer 1 · answered Oct 21 '17 at 03:43

You need to synchronize access to searchResults or multiple threads will modify it at the same time. Or you can use the async pattern and return a Task<SearchResult> from your async method and not use the same List for every thread.

Also, declaring SearchResult result outside of the thread is just asking for trouble in the same way that searchResults is causing trouble. Declare it inside the thread.

score 1 · Answer 2 · answered Oct 21 '17 at 11:23

1

Solved it. In the startSearch method I have a call to QueryParser which is not threadsafe as described here. Resolved this by adding a lock to the QueryParser instance.

answered Oct 21 '17 at 11:23

pad

33
6

Yes I also put a `lock` there. Thank you! – pad Oct 23 '17 at 13:54

Lucene.Net Parallel Search

2 Answers2