Multithreading a File Map into an Array of Buffers

Question

I'm trying to work with nasty large xml and text documents: ~40GBs. I'm using Visual Studio 2012 on Windows 7.

I'm going to use 'Xerces' to snag the header/'footer tag' from the xmls.

I want to map an area of the file, say.. 60-120MBs.

Split the Map into (3 * processors/cores) equal parts. Setting each part as a buffer and loading the buffers into an array.

Then using (#processors/cores) while statments in new threads, I will synchronously count characters/lines/xml cycles while chewing through the the buffer array. When one buffer is completed the the process will jump to the next 'available' buffer and the completed buffer will be dropped out of memory. At the end I will add the total results into a project log.

Afterwards, I will reference the log, Split the files by character count/size(Or other option) to the nearest line or cycle and drop in the header and 'footer tag' to all the splits.

I'm doing this so I can import massive data to a MySQL server over a network with multiple computers.

My Question is, how do I create the buffer array and the file map with new threads?

Can I use :

win CreateFile

win CreateFileMapping

win MapViewOfFile

with standard ifstream operations and char buffers or should I opt something else?

Futher clarification: My thinking is that if I can have the hard drive streaming the file into memory from one place and in one direction that I can use the full processing power of the machine to chew through seperate but equal buffers.

~Flavor: It's kind of like being a Shepard trying to scoop food out from one huge bin with 3-6 Large buckets with only two arms for X sheep that need to stay inside the fenced area. But they all move at the speed of light.

A few ideas or pointers might help me along here. Any thoughts are Most Welcome. Thanks.

while(getline(my_file, myStr))
{
   characterCount += myStr.length();

   lineCount++;


   if(my_file.eof()){

      break;

   }
}

This was the only code at run time for the test. 2hours, 30+min. 45-50% total processor for the program running it on a dual core 1.6Mhz laptop with 2GB RAM. Most of the RAM loaded right now is 600+MB from ~50 tabs open in firefox, Visual Studio at 60MB, then etcs.

IMPORTANT: During the test, the program running the code, which is only a window, and a dialog box, seemed to dump it's own working and private set of ram, down to like 300K ish, and didn't respond for the length of the test. I need to make another thread for the while statement I'm sure. But this means that NONE of the file was read into a buffer. The CPU was struggling for the entire run to keep up with the tinyest effort from the hard drive.

P.S. Further proof of CPU bottlenecking. It might take me 20min to transfer than entire file to another computer over my wireless network. Which includes the read process and a socket catch to write process on the other computer.

UPDATE

I used this adorable little thing to go from the previous test time to about 15-20min which is in line with what Mats Petersson was saying.

while (my_file.read( &bufferOne[0], bufferOne.size() ))

{

int cc = my_file.gcount();

for (int i = 0; i < cc; i++)
{

    if (bufferOne[i] == '\n')
        lineCount++;

    characterCount++;

}

currentPercent = characterCount/onePercent;

SendMessage(GetDlgItem(hDlg, IDC_GENPROGRESS), PBM_SETPOS, currentPercent, 0);

}

Granted this is a single loop and it actually behaved much more appropriately than the previous test. This test was ~800% faster than the tight loop shown above this one with Getline. I set the buffer for this loop at 20MB. I jacked this code from: SOF - Fastest Example

BUT...

I would like to point out that while polling the process in resource mon and task manager, it clearly showed the first core at 75-90% usage, the second fluxuately 25-50% (Pretty standard for some minor background stuff that I have open), and the hard drive at.. wait for it... 50%. Some 100% disk time spikes but also some lows at 25%. All of which basically means that Splitting the buffer processing between two different threads could very well be a benefit. It will use all the system resources but.. that's what I want. I'll update later today when I have the working prototype.

MAJOR UPDATE: Finally finished my project after a bunch of learning. No File Map needed. Only a bunch of vector char's. I have successfully built a dynamically executing file stream line and character counter. The good news, went from the previous 10-15min marker to ~3-4min on a 5.8GB file, BOOYA!~

Answer 1

Very short answer: Yes, you can use those functions.

For reading data, it's likely the most efficient method to map the file content into memory, since it saves having to copy the memory into a buffer in the application, just read it straight into the place it's supposed to go. So, no problem as long as you have enough address space available - 64-bit machines should certainly have plenty, in a 32-bit system it may be more of a scarce resource - but for sections of a few hundred MB, it shouldn't be a huge issue.

However, using multiple threads, I'm not at all convinced. I have a fair idea that reading more than one part of a very large file will be counter productive. This will increase the amount of head movement on the disk, which is a large portion of transfer rate. You can count on some 50-100MB/s transfer rates for "ordinary" systems. If the system has some sort of raid controller or some such, maybe around double that - very exotic raid controllers may achieve three times.

So reading 40GB will take somewhere in the order of 3-15 minutes.

The CPU is probably not going to be very busy, and running multiple threads is quite likely to worsen the overall performance of the system.

You may want to keep a thread for reading and one for writing, and only actually write out the data once you have a sufficient amount of it, again, to avoid unnecessary moves of the read/write head on the disk(s).