Opening multiple files on multiple thread

Question

I have to process, roughly 170.000 files, and would like to use multiple threads. The name of the files is sequential following the Year-Number format and are sorted by year in folders. (but they can be all in the same folder). Different years have different file counts. Files are small size, only a few (10<size<20) KB each.

The order in witch they are processed is indifferent, since the output of the processing task is going to stored in a SQL Database. What would be the best way to this? Without opening the same file twice?

Answer 1

One of the possible solution would be to use producer / consumer design patter.

Your producer would get a list of files and feed some ProducerConsumer queue. Your consumer would process a file (or the file path) taken from the queue and process it (insert into your database). With that approach every file would be processed only once.

Problem of ProducerConsumer queue is described in C# producer/consumer SO question.

EDIT

However, the task might get complicated e.g.

• What happen if one of the existing files will change. Do you have to update the database with the new file content? If so, you would have to have a mechanism of "markers", saying that file has changed (file last update date could work in some cases)
• What happen if during the process new files are added. etc.

Answer 2

I'd say 1 thread per year. Each 'Year Thread' reads the files that start with that year number, and reads them sequentally. As for going to the database i'd suggest you either

• If all goes to a single table, remove indexes so no index locking occurs, and later create the indes
• If you can't remove the indexes, at least use row locking, and a wait period for transanctions before timing out (two or more threads may be inserting at the same time)

Another solution, would be for the threads to generate the insert statement to a file and then execute that file to do the inserts or you could use a bulk insert tool. But this depends on the table structure and your DBMS

Answer 3

Here is a little example:

public static class FilesProcessor
{
    private static List<FileProcessor> m_FileProcessors;

    public static void Start()
    {
        m_FileProcessors = new List<FileProcessor>();

        for (Int32 year = 2005; year < DateTime.Now.Year; ++year)
            InstanciateFileProcessor(year);

        while (!FinishedLoading())
            Application.DoEvents();
    }

    public static void Stop()
    {
        foreach (FileProcessor processor in m_FileProcessors)
            processor.Stop()

        m_FileProcessors.Clear();
        m_FileProcessors = null;
    }

    private static Boolean FinishedLoading()
    {
        foreach (FileProcessor processor in m_FileProcessors)
        {
            if (processor.IsAlive() && !processor.FinishedLoading())
                return false;
        }

        return true;
    }

    private static void InstanciateFileProcessor(Int32 year)
    {
        FileProcessor processor = new FileProcessor(year);
        processor.Start();

        m_FileProcessors.Add(processor);
    }
}

Then the FileProcessor class:

public sealed class FileProcessor
{
    private Int32 m_Year;

    public Boolean IsAlive()
    {
        return ((m_Thread != null) && m_Thread.IsAlive);
    }

    public Boolean FinishedLoading()
    {
        return ((m_Thread == null) || m_Thread.Join(10));
    }

    public FileProcessor(Int32 year)
    {
        m_Year = year;

        m_Thread = new Thread(Load);
        m_Thread.Name = "Background File Processor";
    }

    public void Start()
    {
        if (m_Thread != null)
            m_Thread.Start();
    }

    public void Stop()
    {
        if ((m_Thread != null) && m_Thread.IsAlive)
            m_Thread.Abort();
    }

    private void Load()
    {
        // Browse the Year folder...
        // Get and read all fines one by one...
    }
}

Answer 4

I can see two possible approaches here.

First, split your problem into two. 1 - work out what to process, 2 - do the processing. Part 1 probably has to run on its own so you end up with a 100% accurate list of what needs processing. Then you can implement fancy (or not very fancy) logic as regards splitting the list and introducing multiple threads.

Second, do something similar to what @CarlosGrappa suggests. So essentially you create each thread with its own "pre-programmed" filter. It could be the year, as Carlos suggests. Or, you could create 24 threads, one for each hour of the file timestamp. Or 60 threads, each looking at a particular minute past the hour. It can basically be anything which gives you a definite criterion for (a) splitting the load as evenly as possible, and (b) for guaranteeing that a data file is processed once-and-only-once.

Clearly the second of these approaches would run more quickly, but you'd have to put some extra thought into how you split the files up. With the first method, once you've got the full list, you could basically chuck 100, or 1000, or 10000 etc. files at a time at your processors without being overly smart about how you do it.

Answer 5

What's wrong with using .Net's parallel class?

just pass a collection to the parallel foreach loop. .Net does all the assigning for you. You can also pass in a custom partitioner so you can use chunk partitioning. Chunk partitioning causes the threads to keep asking for more tasks. If you don't use chunk partitioning all the work will be pre-allocated leading to some performance hits when some tasks take longer than others (which might lead to some threads being idle while one thread still has work to do).

http://msdn.microsoft.com/en-us/library/dd460720.aspx