First things first: Visual Studio Setup Projects are quite limited, you might be better off looking for a more capable deployment tool / framework: Application setup.
NTFS Only: That sounds pretty exotic. It isn't even possible to install Windows on anything but NTFS nowadays? From superuser.com: "Windows Vista and higher will not install on a FAT32 partition and can only be installed on an NTFS partition. This is probably due to the use of symlinks (which are not supported in FAT32)." I don't have an MSDN link for you to verify, but I am pretty sure the above is correct.
Built-In MSI Feature: It is possible that there is a built in property in MSI that will be set to the system drive's file system. I am not familiar with any such properties.
Custom Action: If you are on Windows XP, I suppose you need a custom action and then a call to either WMI or directly via the Win32 API to determine what the file system really is.
Win32: An answer with some sample mock-up code for a Win32 check of file system:
Determine what filesystem a partition is of. An inline version tested with VS2017, C++ console project:
#include "stdafx.h"
#include "Windows.h"
#define WIN32_LEAN_AND_MEAN // Exclude rarely used stuff from Windows.h (Crypto stuff?)
int main()
{
WCHAR FSType[512];
if (GetVolumeInformationW(L"C:\\", NULL, 0, NULL, NULL, NULL, FSType, ARRAYSIZE(FSType))) {
MessageBox(NULL, FSType, L"File System:", MB_OK);
// TEXT("File System:") instead of L"File System:" if need be?
}
}
WMI: You should be able to get the information from Win32_LogicalDisk
:
SELECT FileSystem FROM Win32_LogicalDisk WHERE DeviceID='C:'
FileSystemObject: I forgot that the COM scripting runtime can be used:
Set fso = CreateObject("Scripting.FileSystemObject")
Set drive = fso.GetDrive(fso.GetDriveName("C:"))
MsgBox drive.FileSystem