MultiThreading with Directories

Question

I made a program using multithreading to list files in directory/drive.When executing it results in less no of files than the actual no of files inside the directory/drive.i dont know where the mistake lies.Please help me and guide me properly through the logic.

Directory-(global deque) consists of list of folders in directory/Drive

DWORD WINAPI List(LPVOID)
    {
        deque<string>subdir;
        while(true)
        {
            EnterCriticalSection(&QueueLock);
            if(directories.empty())
            {
                LeaveCriticalSection(&QueueLock);
                //Sleep(500);
                break;
            }
            else
            {
                string path = directories.front();
                //cout << path << endl;
                //spec = path + "\\" + "*";
                directories.pop_front();
                subdir.push_front(path);
                LeaveCriticalSection(&QueueLock);

            }

            //Listcontents(path,spec,subdir);
            while(!subdir.empty())
            {
                EnterCriticalSection(&Fillock);
                string subpath = subdir.front();
                string spec = subpath + "\\" + "*";
                subdir.pop_front();
                LeaveCriticalSection(&Fillock);
                HANDLE hf = FindFirstFileA(spec.c_str(),&ffd);
                if(hf == INVALID_HANDLE_VALUE)
                    continue;

            do
            {
                if(strcmp(ffd.cFileName,".") && strcmp(ffd.cFileName,".."))
                {
                    if(ffd.dwFileAttributes &FILE_ATTRIBUTE_DIRECTORY)
                    {
                        EnterCriticalSection(&Sublock);
                        cout<< subpath <<endl;
                        subdir.push_front(subpath + "\\" + ffd.cFileName);
                        LeaveCriticalSection(&Sublock);
                    }
                    else
                    {
                        EnterCriticalSection(&Veclock);
                        files.push_back(subpath + "\\" + ffd.cFileName);
                        Files++;
                        LeaveCriticalSection(&Veclock);
                    }
                }

            }while(FindNextFileA(hf,&ffd));
            FindClose(hf);
            hf = INVALID_HANDLE_VALUE;
        }
        }
        return 0;
    }

Using more than one thread is certainly the worst mistake you made. Threads buy you more cpu cycles, they don't buy you more disk drives. They are actually extremely detrimental to perf, sending the disk drive head zipping back-and-forth. By far the most expensive thing you can do with a disk drive. Just use **one** thread, that of course solves your threading bug as well. — Hans Passant, Jan 25 '14 at 14:17
@HansPassant But i need to list the files in a more faster manner how that can be acheived ...I think only multithreading is the solution ..is There are any other methods ?? — Baalki, Jan 25 '14 at 14:25
Scanning the entire hard drive is slow no matter how you slice it. You need to think outside the box, like scanning only one level at a time or using somebody else's database like the indexer. — Raymond Chen, Jan 25 '14 at 16:40
You should use your program with a small and simple directory structure and use it's output to see where it begins to misbehave. — manuell, Jan 25 '14 at 17:11
It's not clear where `ffd` is declared. Each thread should probably have its own. If you have multiple threads sharing `ffd`, that could explain the problem as there's no synchronization for that structure. — Adrian McCarthy, Jan 25 '14 at 17:54
Use one thread, and instead use `FindFirstFileEx` with the `FIND_FIRST_EX_LARGE_FETCH` flag, which will let you take advantage of the OS' optimizations instead of trying to do it yourself. — josh poley, Jan 25 '14 at 18:12
@AdrianMcCarthy..i declared it globally....Wait a Second..Oh! Wow..the problem alleviated..Thanks a lot..I just now realized the mistake i made . I have to declare it locally so that each entering having its own. Thanks again :) — Baalki, Jan 26 '14 at 11:12
@Baalki: Glad you fixed your bug, but you should really heed the advice here to not use multi-threading in this way. The operation is almost certainly limited by I/O, not by processor. Throwing multiple threads at this problem just adds complexity without speeding anything up. — Adrian McCarthy, Jan 28 '14 at 21:02
@AdrianMcCarthy.. But what else is the way to make execution time faster... am now have to improve this code to list files in multiple drives...can u please suggest any idea ??? — Baalki, Jan 29 '14 at 05:45

score 0 · Answer 1 · answered Jan 28 '14 at 15:19

@Adrian i should thank u a lot.This code i implemented runs perfectly.i corrected my mistake and now its perfect.

DWORD WINAPI List(LPVOID)
    {
        int localFileCount = 0;
        deque<string>subdir;
        WIN32_FIND_DATAA ffdT;
        string path,spec;
        while(true)
        {
            EnterCriticalSection(&QueueLock);
            if(directories.empty())
            {
                LeaveCriticalSection(&QueueLock);
                break;
            }
            else
            {
                path = directories.front();
                directories.pop_front();
                LeaveCriticalSection(&QueueLock);


                //cout << path << endl;
                //spec = path + "\\" + "*";
            }





                string subpath = path;
                spec = subpath + "\\" + "*";


                HANDLE hf = FindFirstFileA(spec.c_str(),&ffdT);
                if(hf == INVALID_HANDLE_VALUE)
                    continue;

            do
            {

                if(strcmp(ffdT.cFileName,".") && strcmp(ffdT.cFileName,".."))
                {

                    if(ffdT.dwFileAttributes &FILE_ATTRIBUTE_DIRECTORY)
                    {
                        EnterCriticalSection(&QueueLock);
                        directories.push_front(subpath + "\\" + ffdT.cFileName);
                        LeaveCriticalSection(&QueueLock);
                    }
                    else
                    {
                        EnterCriticalSection(&FileEntryLock);
                        files.push_back(subpath + "\\" + ffdT.cFileName);
                        Files++;
                        localFileCount++;
                        LeaveCriticalSection(&FileEntryLock);
                    }   
                }
            }while(FindNextFileA(hf,&ffdT)!=0);
            FindClose(hf);
            hf = INVALID_HANDLE_VALUE;


            //cout<< " No of File : "<<localFileCount<<endl;
            localFileCount = 0;
        }

        return 0;
    }

MultiThreading with Directories

1 Answers1