Deleting a file with I/O in progress: Is it a filesystem and/or an OS feature?

Question

I'm writing a shell script that will run on Linux but can operate on files sitting on a mounted partition,

1. that may or may not have an ext* file-system. It could eg be NTFS, FAT32, or any inode- or non-inode-based system;

2. that could get further re-mounted on other boxes running non-Linux OSes like Windows or Mac.

Further, my script needs to be able to delete a file on this shared, arbitrarily formatted partition (even while the file's being read from or written to) by a remote process running on Linux, Windows, or Mac boxes

Questions:

1. Is the ability to be able to delete a file in use a feature of,

   a. only the file-system?

   b. or, only the OS?

   c. or, a combo of both?

2. (Extension of Q1) Does it matter if the processe doing I/O on the file and the one deleting the file are both local or remote?

Answer 1

Does it matter if the processes doing I/O on the file and the one deleting the file are both local or remote?

interesting - how remote system can access file (open, read-write data, delete) on windows direct ? really this impossible. we need some agent (LANMan server), running in the local system, which will be by remote commands (send say by Network Redirector) do local operations on file. so from file system view - all operations always local.

Is the ability to be able to delete a file in use a feature of

this is implemented of course by file system driver, but this driver written for concrete OS and based on it rules. while file system data on disk is have common format (as result drive formatted(and written files) in one OS, can be read from another OS) - how file system driver process requests, open, read, write, delete files - this is OS specific. different for different OS. based on it rules. so data format on disk is common and only depend from file system. but how this data read/write/delete - already os specific.

in windows we have next rules for delete files:

Normally a file marked for deletion is not actually deleted until all open handles for the file have been closed and the link count for the file is zero. When marking a file for deletion using FILE_DISPOSITION_POSIX_SEMANTICS, the link gets removed from the visible namespace as soon as the POSIX delete handle has been closed, but the file’s data streams remain accessible by other existing handles until the last handle has been closed.

so in general file will be not deleted, until the last handle to it will be closed. file became not accessible after we try delete it - can not more open it (we get error A non close operation has been requested of a file object with a delete pending. if try do this, after file marked to delete). but if file was already opened - we can still work with it by this handle. also file can not be deleted if section exist on file - will be error An attempt has been made to remove a file or directory that cannot be deleted.

begin from win10 redstone1 build exist FILE_DISPOSITION_POSIX_SEMANTICS flag which let removed file name from the visible namespace when delete handle has been closed, but the file’s data streams remain accessible by other existing handles until the last handle has been closed

windows code test demo: (FILE_DISPOSITION_POSIX_SEMANTICS supported by ntfs begin only from _WIN32_WINNT_WIN10_RS1. FileDispositionInfoEx information class also supported begin from _WIN32_WINNT_WIN10_RS1 only. in previous build we simply got not implemented error)

void print_error(PCSTR name)
{
    PWSTR sz;
    NTSTATUS status = RtlGetLastNtStatus();
    if (FormatMessageW(FORMAT_MESSAGE_ALLOCATE_BUFFER|FORMAT_MESSAGE_FROM_HMODULE, 
        GetModuleHandle(L"ntdll"), status, 0, (PWSTR)&sz, 0, 0))
    {
        DbgPrint("%s=%x\n%S\n", name, status, sz);
        LocalFree(sz);
    }
}

HANDLE OpenFile(PCWSTR lpFileName, DWORD dwDesiredAccess)
{
    HANDLE hFile = CreateFileW(lpFileName, dwDesiredAccess, FILE_SHARE_VALID_FLAGS, 0, OPEN_EXISTING, 0, 0);

    if (hFile == INVALID_HANDLE_VALUE)
    {
        print_error("OpenFile");
        return 0;
    }

    return hFile;
}

void ReadTest(HANDLE hFile)
{
    if (hFile)
    {
        ULONG dwBytes;
        if (ReadFile(hFile, &dwBytes, sizeof(dwBytes), &dwBytes, 0))
        {
            DbgPrint("ReadFile=OK\n");
        }
        else
        {
            print_error("ReadFile");
        }
    }
}

void DeleteTest(PCWSTR lpFileName)
{
    HANDLE hFile1, hFile2, hFile3;

    if (hFile1 = OpenFile(lpFileName, DELETE))
    {
        hFile2 = OpenFile(lpFileName, FILE_GENERIC_READ);

        FILE_DISPOSITION_INFO_EX fdi = { FILE_DISPOSITION_DELETE | FILE_DISPOSITION_POSIX_SEMANTICS };
        if (!SetFileInformationByHandle(hFile1, FileDispositionInfoEx, &fdi, sizeof(fdi)))
        {
            print_error("SetFileInformationByHandle");
        }

        // file already not accessible here (open must fail) but visible
        if (hFile3 = OpenFile(lpFileName, FILE_GENERIC_READ))
        {
            CloseHandle(hFile3);
        }

        ReadTest(hFile2);

        // win10 rs1: file removed from the visible namespace here
        CloseHandle(hFile1);

        // are file still visible ?
        if (hFile3 = OpenFile(lpFileName, FILE_GENERIC_READ))
        {
            CloseHandle(hFile3);
        }

        // are possible create new file with this name &
        hFile3 = CreateFileW(lpFileName, DELETE, 
            FILE_SHARE_VALID_FLAGS, 0, CREATE_NEW, FILE_FLAG_DELETE_ON_CLOSE, 0);
        if (hFile3 == INVALID_HANDLE_VALUE)
        {
            print_error("CreateFile");
        }
        else
        {
            CloseHandle(hFile3);
            DbgPrint("CreateFile OK\n");
        }

        ReadTest(hFile2);

        if (hFile2)
        {
            CloseHandle(hFile2);
        }
    }
}

and output

OpenFile=c0000056
A non close operation has been requested of a file object with a delete pending.

ReadFile=OK
OpenFile=c0000034
Object Name not found.

CreateFile OK
ReadFile=OK

Answer 2

it depends how do you define filesystem and OS. Generally, I understand under filesystem the way how the data storage is organized on a device. OS is then responsible for the I/O of data and files. Particularly, if your script wants to delete a file it calls some utility like rm and provides name of the file. This utility is a program which makes an appropriate system call. This system call is a part of operating system which is executed in the privileged mode. It realizes what and how to do (e.g. which drivers should be used to mark HDD blocks as free on a particular drive or maybe some remote procedure should be called or samba server involved etc.) So, to give an answer to your question 1, I would tend to the answer b.