Monitor deletions and changes on NTFS volume with MFT_ENUM_DATA

Question

I'm using this code to fill a database of all the files on disk:

TCHAR szVolumePath[_MAX_PATH] = L"\\\\.\\d:";

HANDLE hDrive = CreateFile(szVolumePath, GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, NULL, NULL);

MFT_ENUM_DATA_V0 med = { 0 };
med.StartFileReferenceNumber = 0;
med.LowUsn = 0;
med.HighUsn = MAXLONGLONG;

DWORD cb;
PUSN_RECORD pRecord;

unsigned char pData[sizeof(DWORDLONG) + 0x10000] = { 0 }; 

while (DeviceIoControl(hDrive, FSCTL_ENUM_USN_DATA, &med, sizeof(med), pData, sizeof(pData), &cb, NULL) != FALSE)
{
    pRecord = (PUSN_RECORD)&pData[sizeof(USN)];
    while ((PBYTE)pRecord < (pData + cb))
    {
        wstring sz((LPCWSTR)(PBYTE)pRecord + pRecord->FileNameOffset, pRecord->FileNameLength / sizeof(WCHAR));

        // file the database

        pRecord = (PUSN_RECORD)((PBYTE)pRecord + pRecord->RecordLength);
    }
    med.StartFileReferenceNumber = *(DWORDLONG *)pData;
}

Once the loop is finished, the database is successfully filled.

But how to continue (as a background task) to monitor in realtime the file changes / deletions? (example: display a MessageBox() "File readme.txt has been renamed.")

Should I re-launch such a loop every 1-second, with med.StartFileReferenceNumber = the highest FileReferenceNumber seen before?

Note: I'm a bit reluctant to launch this code every 1-second (99% of the time, for nothing). Doing it every 10-second instead would avoid using so much resource, but there would be a delay before detecting changes, whereas I do know some indexing software that don't have such delay.

Note2: I read How can I detect only deleted, changed, and created files on a volume? but the purpose of the main answer is to run it once, and not constantly in background.

Note3: I have looked at this useful code example from Microsoft Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained .

Note4: Should I keep FSCTL_ENUM_USN_DATA for initial database loading, and then use FSCTL_READ_USN_JOURNAL instead?

Note5: ReadDirectoryChangesW or FindNextChangeNotification (the former gives full path of changes in the notifications, the latter doesn't) cannot really be used because it won't give the FileReferenceNumber of a deleted file (one would have to open the file and use NtQueryInformationFile to get it; but this is impossible for deleted files); and FileReferenceNumber is necessary to update the file database (the file database uses a map / dictionary with FileReferenceNumbers as keys).

Maybe using/reading [fswatch](https://github.com/emcrisostomo/fswatch/blob/master/README.md) can help — Ben, Jul 30 '17 at 21:40
if you want monitor file deletions and changes in real time - you need exactly `ReadDirectoryChangesW` for volume — RbMm, Jul 30 '17 at 22:40
@RbMm I would prefer to use USN journal: FSCTL_ENUM_USN_DATA or FSCTL_READ_USN_JOURNAL (see my edited question). The problem is the "waiting" part : isn't there a sync / asynchronous version that would avoid a `Sleep(1000)`? — Basj, Jul 30 '17 at 22:44
I not understand what you mean "waiting" part and for what `Sleep` at all here. `ReadDirectoryChangesW` the best way and give you exactly what you need — RbMm, Jul 30 '17 at 22:47
why you want use `FSCTL_ENUM_USN_DATA` or `FSCTL_READ_USN_JOURNAL` instead of `ReadDirectoryChangesW` ? based on what reason ? — RbMm, Jul 30 '17 at 22:49
@RbMm [Here](https://msdn.microsoft.com/en-us/library/windows/desktop/aa363798(v=vs.85).aspx) it says that change journals (USN journal) is to be preferred for performance reasons. — Basj, Jul 30 '17 at 22:52
@RbMm moreover [`ReadDirectoryChangesW`](https://msdn.microsoft.com/en-us/library/windows/desktop/aa365465(v=vs.85).aspx) / [`FindFirstChangeNotification`](https://msdn.microsoft.com/en-us/library/windows/desktop/aa364417(v=vs.85).aspx) wouldn't give access to NTFS FileReferenceNumber, which is helpful for me to update the database. — Basj, Jul 30 '17 at 23:00
@Basj - you bad read. for performance reasons better use `ReadDirectoryChangesW` - *This is more efficient than the first method, however, it requires that an application be running at all times.* - `FileReferenceNumber` you can get by query `FileInternalInformation` with `NtQueryInformationFile`. here will be 48bit `FileReferenceNumber` + high 16 bit — RbMm, Jul 30 '17 at 23:08
@RbMm Another reason: when doing `FindFirstChangeNotification(lpDir, TRUE, FILE_NOTIFY_CHANGE_FILE_NAME);` (i.e. monitor changes for subdirs too!) with `lpDir = "c:\"`, then when a change notification message happens, the only info I get is "C:\ modified" and I don't know where it happened... So this means I should rescan the whole "C:\"... — Basj, Jul 30 '17 at 23:09
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/150550/discussion-between-basj-and-rbmm). — Basj, Jul 30 '17 at 23:11
As discussed in chat, @RbMm, you're right: ReadDirectoryChangesW gives better notification information (full path of change) than FindNextChangeNotification (not full path). — Basj, Jul 30 '17 at 23:24
@RbMm I added Note5 to the question that states why it is finally important to use USN_JOURNAL. — Basj, Jul 30 '17 at 23:32
You should be able to use [FSCTL_READ_USN_JOURNAL](https://msdn.microsoft.com/en-us/library/windows/desktop/aa364586(v=vs.85).aspx), it has `Timeout` and `BytesToWaitFor` options. I find the documentation on those options confusing, but I'm sure you can figure it out with a bit of experimentation. (Edit: the description in your Note3 link is much clearer.) — Harry Johnston, Jul 31 '17 at 00:54

Monitor deletions and changes on NTFS volume with MFT_ENUM_DATA

0 Answers0