File search optimisation in C# using Parallel

Question

I have a folder with many CSV files in it, which are around 3MB each in size.

example of content of one CSV:

afkla890sdfa9f8sadfkljsdfjas98sdf098,-1dskjdl4kjff;
afkla890sdfa9f8sadfkljsdfjas98sdf099,-1kskjd11kjsj;
afkla890sdfa9f8sadfkljsdfjas98sdf100,-1asfjdl1kjgf;
etc...

Now I have a Console app written in C#, that searches each CSV file for a certain string. And those strings to search for are in a txt file.

example of search txt file:

-1gnmjdl5dghs
-17kn3mskjfj4
-1plo3nds3ddd

then I call the method to search each search string in all files in given folder:

private static object _lockObject = new object();
public static IEnumerable<string> SearchContentListInFiles(string searchFolder, List<string> searchList)
{
    var result = new List<string>();

    var files = Directory.EnumerateFiles(searchFolder);
    Parallel.ForEach(files, (file) =>
    {
        var fileContent = File.ReadLines(file);

        if (fileContent.Any(x => searchList.Any(y => x.ToLower().Contains(y))))
        {
            lock (_lockObject)
            {
                foreach (string searchFound in fileContent.Where(x => searchList.Any(y => x.ToLower().Contains(y))))
                {
                    result.Add(searchFound);
                }
            }
        }
    });

    return result;
}

Question now is, can I anyhow improve performance of this operation? I have around 100GB of files to search trough. It takes aproximatly 1 hour to search all ~30.000 files with around 25 search strings, on a SSD disk and a good i7 CPU.

Would it make a difference to have larger CSV files or smaller CSV? I just want this search to be as fast as possible.

UPDATE

I have tried every suggestion that you wrote, and this is now what best performed for me (Removing ToLower from the LINQ yielded best performance boost. Search time from 1hour is now 16minutes!):

public static IEnumerable<string> SearchContentListInFiles(string searchFolder, HashSet<string> searchList)
{
    var result = new BlockingCollection<string>();

    var files = Directory.EnumerateFiles(searchFolder);
    Parallel.ForEach(files, (file) =>
    {
        var fileContent = File.ReadLines(file); //.Select(x => x.ToLower());

        if (fileContent.Any(x => searchList.Any(y => x.Contains(y))))
        {
            foreach (string searchFound in fileContent.Where(x => searchList.Any(y => x.Contains(y))))
            {
                result.Add(searchFound);
            }
        }
    });

    return result;
}

Answer 1

Probably something like Lucene could be a performance boost: why don't you index your data so you can search it easily?

Take a look at Lucene .NET

You'll avoid searching data sequentially. In addition, you can model many indexes based on the same data to be able to get to certain results at the light speed.

Answer 2

Try to:

1. Do .ToLower one time for a line instead of do .ToLower for each element in searchList.

2. Do one scan of file instead of two pass any and where. Get the list and then add with lock if any found. In your sample you waste time for two pass and block all threads when search and add.

3. If you know position where to look for (in your sample you know) you can scan from position, not in all string

4. Use producer consumer pattern for example use: BlockingCollection<T>, so no need to use lock

5. If you need to strictly search in field, build HashSet of searchList and do searchHash.Contains(fieldValue) this will increase process dramatically

So here a sample (not tested):

using(var searcher = new FilesSearcher(
    searchFolder: "path", 
    searchList: toLookFor))
{
    searcher.SearchContentListInFiles();
}

here is the searcher:

public class FilesSearcher : IDisposable
{
    private readonly BlockingCollection<string[]> filesInMemory;
    private readonly string searchFolder;
    private readonly string[] searchList;

    public FilesSearcher(string searchFolder, string[] searchList)
    {
        // reader thread stores lines here
        this.filesInMemory = new BlockingCollection<string[]>(
            // limit count of files stored in memory, so if processing threads not so fast, reader will take a break and wait
            boundedCapacity: 100);

        this.searchFolder = searchFolder;
        this.searchList = searchList;
    }

    public IEnumerable<string> SearchContentListInFiles()
    {

        // start read,
        // we not need many threads here, probably 1 thread by 1 storage device is the optimum
        var filesReaderTask = Task.Factory.StartNew(ReadFiles, TaskCreationOptions.LongRunning);

        // at least one proccessing thread, because reader thread is IO bound
        var taskCount = Math.Max(1, Environment.ProcessorCount - 1);

        // start search threads
        var tasks = Enumerable
            .Range(0, taskCount)
            .Select(x => Task<string[]>.Factory.StartNew(Search, TaskCreationOptions.LongRunning))
            .ToArray();

        // await for results
        Task.WaitAll(tasks);

        // combine results
        return tasks
            .SelectMany(t => t.Result)
            .ToArray();
    }

    private string[] Search()
    {
        // if you always get unique results use list
        var results = new List<string>();
        //var results = new HashSet<string>();

        foreach (var content in this.filesInMemory.GetConsumingEnumerable())
        {
            // one pass by a file
            var currentFileMatches = content
                .Where(sourceLine =>
                {
                    // to lower one time for a line, and we don't need to make lowerd copy of file
                    var lower = sourceLine.ToLower();

                    return this.searchList.Any(sourceLine.Contains);
                });

            // store current file matches
            foreach (var currentMatch in currentFileMatches)
            {
                results.Add(currentMatch);
            }                
        }

        return results.ToArray();
    }

    private void ReadFiles()
    {
        var files = Directory.EnumerateFiles(this.searchFolder);

        try
        {
            foreach (var file in files)
            {
                var fileContent = File.ReadLines(file);

                // add file, or wait if filesInMemory are full
                this.filesInMemory.Add(fileContent.ToArray());
            }
        }
        finally
        {
            this.filesInMemory.CompleteAdding();
        }
    }

    public void Dispose()
    {
        if (filesInMemory != null)
            filesInMemory.Dispose();
    }
}

Answer 3

This operation is first and foremost disk bound. Disk bound operations do not benefit from Multithreading. Indeed all you will do is swamp the Disk controler with a ton of conflictign requests at the same time, that a feature like NCQ has to striahgten out again.

If you had loaded all the files into memory first, your operation would be Memory Bound. And memory bound operations do not benefit from Multithreading either (usually; it goes into details of CPU and memory architecture here).

While a certain amount of Multitasking is mandatory in Programming, true Multithreading only helps with CPU bound operations. Nothing in there looks remotely CPU bound. So multithreading taht search (one thread per file) will not make it faster. And indeed likely make it slower due to all the Thread switching and synchronization overhead.