How to juggle FILE_SKIP_COMPLETION_PORT_ON_SUCCESS, IOCP, and cleanup

Question

If FILE_SKIP_COMPLETION_PORT_ON_SUCCESS is set on a file handle that is bound to an I/O completion port, then an OVERLAPPED structure needs to be deallocated when its I/O completes synchronously. Otherwise, it needs to stay alive until a worker processes the notification from an I/O completion port.

This all sounds good until you realize that this only works if you manage the file handle yourself.
But if someone else gives you the file handle, how are you supposed to know when you should free the OVERLAPPED structure? Is there any way to discover this after the fact?
Otherwise, does this basically imply you cannot correctly perform overlapped I/O on any file handle that you cannot guarantee the completion notification state of...?

Answer 1

I'm not sure that your scenario makes sense.

Your clarified scenario - successfully performing I/O on an arbitrary file handle, without even knowing whether it is asynchronous or not - is challenging, I think very unusual, and almost certainly not how the API was designed to be used, but perhaps (as you suggest) not entirely implausible.

(Although I don't think you can avoid requiring some cooperation between the caller and your code, because in the IOCP case, the caller has to be able to tell whose I/O a dequeued packet belongs to. You could do this by having the caller allocate the OVERLAPPED structures, as RbMm suggests, but it might be simpler to ask them for a completion key to use.)

I'm not certain offhand how Windows behaves if you provide a redundant event handle, e.g., when the I/O is actually synchronous or using IOCP. But I would guess that it isn't going to be a problem in practice, so provided you're not too worried about future-proofing, you're probably OK.

---

At any rate, it isn't all that difficult to deal with the particular issue your question asks about. Basically, you just need to prevent the structure from being released twice.

• Before making each call, assign a unique completion key and add it to a linked list or other suitable global structure. (The structure must be capable of an atomic find-and-remove operation, or protected by a critical section or similar.)

• If the call succeeds immediately, i.e., does not report that the I/O is pending, treat it exactly as if a queued packet were received from the IOCP queue. Typically, you would either use a common function that is called by both your IOCP thread and your I/O thread, or a call to PostQueuedCompletionStatus to manually insert a packet to the IOCP queue.

• When a packet is received (or when the call succeeds immediately) first perform a find-and-remove for the completion key against the global structure. If the find fails, you know that you have already been notified of the success of the I/O, and don't need to do anything.

• If the find-and-remove succeeds, process the I/O as appropriate and release the OVERLAPPED structure.

There are undoubtedly ways to optimize the same basic approach.

Addendum: if the caller is processing the IOCP packets, and providing you with a completion key to use, you won't be able to use a unique completion key on each request. In this scenario, you can use the pointer to the OVERLAPPED structure instead.

The reason (in the general case) for not using the pointer is that you might receive a packet containing a completion key from one I/O request along with an OVERLAPPED structure from a different one, because the OVERLAPPED structure might be both released and reassigned before a duplicate notification is processed. That doesn't matter in this case, because all of your requests will use the same completion key anyway.

Addendum^2: if you don't know anything about the handle, you'll also need to provide an event object for each OVERLAPPED structure, and wait on them in case notification of the I/O completion arrives that way. It's getting too late in the day for me to try to figure out the exact consequences of that, but it may mean that under some circumstances you get three notifications for the same I/O operation. You might be able to avoid that, but if not, this approach will still work.

Answer 2

Is there any way to discover this after the fact?

yes, exist - need use ZwQueryInformationFile with FileIoCompletionNotificationInformation FILE_IO_COMPLETION_NOTIFICATION_INFORMATION is defined in wdm.h

so code which we need for query:

FILE_IO_COMPLETION_NOTIFICATION_INFORMATION ficni;
ZwQueryInformationFile(hFile, &iosb, &ficni, sizeof(ficni), FileIoCompletionNotificationInformation);

demo code for set and query

HANDLE hFile;
IO_STATUS_BLOCK iosb;
STATIC_OBJECT_ATTRIBUTES(oa, "\\systemroot\\notepad.exe");
if (0 <= ZwOpenFile(&hFile, FILE_GENERIC_READ, &oa, &iosb, FILE_SHARE_VALID_FLAGS, 0))
{
    FILE_IO_COMPLETION_NOTIFICATION_INFORMATION ficni = { FILE_SKIP_COMPLETION_PORT_ON_SUCCESS };
    if (0 <= ZwSetInformationFile(hFile, &iosb, &ficni, sizeof(ficni), FileIoCompletionNotificationInformation))
    {
        ficni.Flags = 0x12345678;
        if (
            0 > ZwQueryInformationFile(hFile, &iosb, &ficni, sizeof(ficni), FileIoCompletionNotificationInformation)
            ||
            !(ficni.Flags & FILE_SKIP_COMPLETION_PORT_ON_SUCCESS)
            )
        {
            __debugbreak();
        }
    }

    ZwClose(hFile);
}

also let copy paste from wdm.h (for not say that this is "undocumented" )

//
// Don't queue an entry to an associated completion port if returning success
// synchronously.
//
#define FILE_SKIP_COMPLETION_PORT_ON_SUCCESS    0x1

//
// Don't set the file handle event on IO completion.
//
#define FILE_SKIP_SET_EVENT_ON_HANDLE           0x2

//
// Don't set user supplied event on successful fast-path IO completion.
//
#define FILE_SKIP_SET_USER_EVENT_ON_FAST_IO     0x4

typedef  struct _FILE_IO_COMPLETION_NOTIFICATION_INFORMATION {
    ULONG Flags;
} FILE_IO_COMPLETION_NOTIFICATION_INFORMATION, *PFILE_IO_COMPLETION_NOTIFICATION_INFORMATION;

I have question - for what reason this is declared in wdm.h ?