If you want a bulletproof protection of your files, then just protecting access to the folder is not enough, you'd have to encrypt them and there're secure containers and file system encyptions on the market.
If it does not need to be high security, you can hook into Windows I guess. You'll especially need to hook into the directory listing functions, like FindFirstFile, FindNextFile and OpenFile also probably (and into their derivates like FindFirstFileW) and maybe some others.
You do that by redirecting calls to kernel32.dll to your custom functions, see a little code example below which I found on the internet:
unsigned char Store[10];
//redirect FindNextFileW to your custom function
void HookAPI()
{
DWORD OldProtect, NewProtect = PAGE_EXECUTE_READWRITE;
HMODULE hmod = GetModuleHandle("Kernel32.dll");
long pa = (long)GetProcAddress(hmod,"FindNextFileW");
long pa2 = (long)MyFindNextFile;
long dAddr = pa2 - pa - 5;
unsigned char *p = (unsigned char *)pa;
unsigned char *p2 = (unsigned char *)(&dAddr);
VirtualProtect((void *)pa,5,NewProtect,&OldProtect);
for (int i=0;i<5;i++)
Store[i] = p[i];
p[0] = (unsigned char)0xE9;
for (int i=0;i<4;i++)
p[i + 1] = p2[i];
VirtualProtect((void *)pa,5,OldProtect,&NewProtect);
}
void UnHookAPI()
{
DWORD OldProtect, NewProtect = PAGE_EXECUTE_READWRITE;
HMODULE hmod = GetModuleHandle("Kernel32.dll");
long pa = (long)GetProcAddress(hmod,"FindNextFileW");
unsigned char *p = (unsigned char *)pa;
VirtualProtect((void *)pa,5,NewProtect,&OldProtect);
for (int i=0;i<5;i++)
p[i] = Store[i];
VirtualProtect((void *)pa,5,OldProtect,&NewProtect);
}
BOOL WINAPI MyFindNextFile(HANDLE hFindFile,LPWIN32_FIND_DATAW lpFindFileData)
{
UnHookAPI();
BOOL ans = FindNextFileW(hFindFile, lpFindFileData);
//your logic here, display password prompt to user e.g.
HookAPI();
return ans;
}
What you want to do can also be done with Java (JNI) or C# (pinvoke), but it'd be a real detour. I'd use something which can be compiled to native code.
Edit:
Aoi Karasu provided a link to a post which suggests to use a FileSystemFilterDriver, which is probably the best concept to realize the application in question.