Winsock Kernel's "WskSendTo" function causes "DRIVER_IRQL_NOT_LESS_OR_EQUAL" BSOD on Win7 SP1

Question

I'm developing a Windows packet capture software called Npcap. And it needs to send loopback raw IP sockets based on Windows Kernel. But the WskSocket->Dispatch->WskSendTo always causes DRIVER_IRQL_NOT_LESS_OR_EQUAL BSOD on Win7 SP1. The strange thing is that my code doesn't trigger this BSoD on other systems like Win8, Win10. It only happens on Win7. So I even doubt that is this a bug of Windows itself or only my bug? Thanks!

The reproduce steps are:

Install Npcap 0.07 r17 with default options
Install Nmap 7.20 Beta 5 (don't install the shipped Npcap)
In CMD, run nmap -v -O -6 localhost to perform a localhost scan (this functionality is provided by Npcap), you will encounter the BSoD in a couple of seconds.
If you want the faulty driver's debug symbols, it can be downloaded here. Refer to \npcap-DebugSymbols\win7\x64\npcap.pdb for x64 system and \npcap-DebugSymbols\win7\x86\npcap.pdb for x86 system.

The BSOD analysis from WinDbg (I have the full dump, tell me if needed):

************* Symbol Path validation summary **************
Response                         Time (ms)     Location
OK                                             J:\npcap\packetWin7\npf\x64\Win7 Release(WinPcap Mode)
Deferred                                       SRV*J:\Symbols*http://msdl.microsoft.com/download/symbols

Microsoft (R) Windows Debugger Version 10.0.10586.567 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\Administrator\Desktop\New folder (2)\MEMORY.DMP]
Kernel Complete Dump File: Full address space is available


************* Symbol Path validation summary **************
Response                         Time (ms)     Location
OK                                             J:\npcap\packetWin7\npf\x64\Win7 Release(WinPcap Mode)
Deferred                                       SRV*J:\Symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: J:\npcap\packetWin7\npf\x64\Win7 Release(WinPcap Mode);SRV*J:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7601 (Service Pack 1) MP (2 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.18798.amd64fre.win7sp1_gdr.150316-1654
Machine Name:
Kernel base = 0xfffff800`02a0a000 PsLoadedModuleList = 0xfffff800`02c4f890
Debug session time: Thu Jun 23 13:50:07.660 2016 (UTC + 8:00)
System Uptime: 0 days 0:31:55.712
Loading Kernel Symbols
...............................................................
................................................................
..............................
Loading User Symbols
.....
Loading unloaded module list
..................Unable to enumerate user-mode unloaded modules, NTSTATUS 0xC0000147
Loading Wow64 Symbols
............................................
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck D1, {0, 2, 8, 0}


"kernel32.dll" was not found in the image list.
Debugger will attempt to load "kernel32.dll" at given base 00000000`00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to add module at 00000000`00000000
Probably caused by : npcap.sys ( npcap!WSKSendTo_NBL+d4 )

Followup:     MachineOwner
---------


************* Symbol Path validation summary **************
Response                         Time (ms)     Location
OK                                             J:\npcap\packetWin7\npf\x64\Win7 Release
Deferred                                       SRV*J:\Symbols*http://msdl.microsoft.com/download/symbols
0: kd> .reload
Loading Kernel Symbols
...............................................................
................................................................
..............................
Loading User Symbols
.....
Loading unloaded module list
..................Unable to enumerate user-mode unloaded modules, NTSTATUS 0xC0000147
Loading Wow64 Symbols
............................................
0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000000, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000008, value 0 = read operation, 1 = write operation
Arg4: 0000000000000000, address which referenced memory

Debugging Details:
------------------


"kernel32.dll" was not found in the image list.
Debugger will attempt to load "kernel32.dll" at given base 00000000`00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to add module at 00000000`00000000

DUMP_CLASS: 1

DUMP_QUALIFIER: 402

BUILD_VERSION_STRING:  7601.18798.amd64fre.win7sp1_gdr.150316-1654

SYSTEM_MANUFACTURER:  VMware, Inc.

VIRTUAL_MACHINE:  VMware

SYSTEM_PRODUCT_NAME:  VMware Virtual Platform

SYSTEM_VERSION:  None

BIOS_VENDOR:  Phoenix Technologies LTD

BIOS_VERSION:  6.00

BIOS_DATE:  07/02/2015

BASEBOARD_MANUFACTURER:  Intel Corporation

BASEBOARD_PRODUCT:  440BX Desktop Reference Platform

BASEBOARD_VERSION:  None

DUMP_TYPE:  0

BUGCHECK_P1: 0

BUGCHECK_P2: 2

BUGCHECK_P3: 8

BUGCHECK_P4: 0

READ_ADDRESS:  0000000000000000 

CURRENT_IRQL:  2

FAULTING_IP: 
+0
00000000`00000000 ??              ???

PROCESS_NAME:  nmap.exe

CPU_COUNT: 2

CPU_MHZ: a29

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 5e

CPU_STEPPING: 3

CPU_MICROCODE: 6,5e,3,0 (F,M,S,R)  SIG: 23'00000000 (cache) 23'00000000 (init)

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0xD1

ANALYSIS_SESSION_HOST:  DESKTOP-AKQG651

ANALYSIS_SESSION_TIME:  06-23-2016 13:56:03.0297

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

TRAP_FRAME:  fffff88006aa5680 -- (.trap 0xfffff88006aa5680)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffffa80018ede30 rbx=0000000000000000 rcx=fffffa8001a13390
rdx=fffffa800108de20 rsi=0000000000000000 rdi=0000000000000000
rip=0000000000000000 rsp=fffff88006aa5818 rbp=fffff88008565d06
 r8=fffff880017684e8  r9=fffff8800164f030 r10=0000000000000000
r11=fffff88006aa5480 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
00000000`00000000 ??              ???
Resetting default scope

IP_IN_FREE_BLOCK: 0

LAST_CONTROL_TRANSFER:  from fffff80002a7bfe9 to fffff80002a7ca40

FAILED_INSTRUCTION_ADDRESS: 
+0
00000000`00000000 ??              ???

STACK_TEXT:  
fffff880`06aa5818 fffff880`0173d917 : fffffa80`0108df50 fffffa80`0108df50 00000000`00000018 00000000`00000018 : 0x0
fffff880`06aa5820 fffff880`0173fe02 : fffffa80`026cc080 fffffa80`01d89080 00000000`00000087 00000000`00000000 : tcpip!Ipv6pHandleNeighborSolicitation+0x257
fffff880`06aa58e0 fffff880`0165bf9e : 00000000`00000000 00000000`00000000 fffff880`01769800 fffffa80`026cc1c0 : tcpip!Icmpv6ReceiveDatagrams+0x342
fffff880`06aa5980 fffff880`0165baaa : 00000000`00000000 fffff880`01769800 fffff880`06aa5b30 00000000`00000001 : tcpip!IppDeliverListToProtocol+0xfe
fffff880`06aa5a40 fffff880`0165b0a9 : 00000000`00000003 fffffa80`026cc100 fffff880`06aa5a03 fffff880`06aa5b30 : tcpip!IppProcessDeliverList+0x5a
fffff880`06aa5ae0 fffff880`0163e28f : fffff880`01769800 00000000`00000000 00000000`00000000 fffff880`06aa5c78 : tcpip!IppReceiveHeaderBatch+0x23a
fffff880`06aa5bc0 fffff800`02a893d8 : fffff880`01769800 00000000`00000000 00000000`00000000 00000000`00000000 : tcpip!IppLoopbackTransmit+0x38f
fffff880`06aa5c70 fffff880`0163e92f : fffff880`016916fc fffffa80`01a0f490 fffff880`06aa5e02 00000000`00000000 : nt!KeExpandKernelStackAndCalloutEx+0xd8
fffff880`06aa5d50 fffff880`0165d4ca : fffffa80`026cc1c0 00000000`00000000 fffffa80`01a0f400 fffffa80`0195e820 : tcpip!IppLoopbackEnqueue+0x22f
fffff880`06aa5e00 fffff880`0165ebf5 : 00000000`00000000 fffffa80`036f4900 fffffa80`019ae400 00000000`000000fa : tcpip!IppDispatchSendPacketHelper+0x38a
fffff880`06aa5ec0 fffff880`0165de7e : fffffa80`019ae4fa fffff880`06aa6200 00000000`00000028 fffffa80`00000000 : tcpip!IppPacketizeDatagrams+0x2d5
fffff880`06aa5fe0 fffff880`0166079e : 00000000`00000000 fffffa80`019b4204 fffff880`01623790 fffffa80`0195e820 : tcpip!IppSendDatagramsCommon+0x87e
fffff880`06aa6180 fffff880`01624248 : fffffa80`019b42f0 fffff880`06aa6700 00000000`00000000 00000000`000007ff : tcpip!IpNlpSendDatagrams+0x3e
fffff880`06aa61c0 fffff880`0162462d : 00000000`00000103 fffff880`01730470 fffffa80`0279c0e0 fffff880`00000001 : tcpip!RawSendMessagesOnPathCreation+0x238
fffff880`06aa63f0 fffff880`03afe69e : fffffa80`00ebc8a0 00000000`00000001 fffffa80`031ea580 fffff880`05a0a7e8 : tcpip!RawSendMessages+0x2bd
fffff880`06aa66e0 fffff880`05a01fb0 : fffffa80`02c77d48 00000025`02a80f78 fffff880`05a0a7e8 00000000`00000000 : afd!WskProIRPSendTo+0x11e
fffff880`06aa6790 fffff880`05a01bdb : 00000000`c0000001 fffffa80`033d8350 fffffa80`03cede20 fffffa80`03cede20 : npcap!WSKSendTo_NBL+0xd4 [j:\npcap\packetwin7\npf\npf\lo_send.c @ 858]
fffff880`06aa6820 fffff880`05a06a0c : fffffa80`03cede20 fffffa80`033d8420 00000000`00000001 fffffa80`03e49318 : npcap!NPF_WSKSendPacket_NBL+0x93 [j:\npcap\packetwin7\npf\npf\lo_send.c @ 366]
fffff880`06aa6860 fffff880`05a06e4b : 00000000`00000000 fffffa80`033d8350 fffffa80`03e40000 00000000`00000000 : npcap!NPF_LoopbackSendNetBufferLists+0x18 [j:\npcap\packetwin7\npf\npf\write.c @ 1019]
fffff880`06aa6890 fffff800`02d8530b : 00000000`00000001 fffffa80`00000000 fffffa80`033d8420 fffffa80`033d8350 : npcap!NPF_Write+0x243 [j:\npcap\packetwin7\npf\npf\write.c @ 328]
fffff880`06aa6900 fffff800`02d90b13 : fffffa80`033d8468 00000000`00000000 fffffa80`0269c9b0 fffffa80`033d8468 : nt!IopSynchronousServiceTail+0xfb
fffff880`06aa6970 fffff800`02a7bcd3 : 00000000`75192401 00000000`00000000 00000000`00000000 00000000`00000000 : nt!NtWriteFile+0x7e2
fffff880`06aa6a70 00000000`75192e09 : 00000000`751929f5 00000000`778201b4 00000000`74ea0023 00000000`00000246 : nt!KiSystemServiceCopyEnd+0x13
00000000`0010e4f8 00000000`751929f5 : 00000000`778201b4 00000000`74ea0023 00000000`00000246 00000000`0030f8fc : wow64cpu!CpupSyscallStub+0x9
00000000`0010e500 00000000`74ead286 : 00000000`00000000 00000000`75191920 ffffffff`fc630000 00000000`7765e021 : wow64cpu!ReadWriteFileFault+0x31
00000000`0010e5c0 00000000`74eac69e : 00000000`00000000 00000000`00000000 00000000`74ea4b10 00000000`7ffe0030 : wow64!RunCpuSimulation+0xa
00000000`0010e610 00000000`77671736 : 00000000`00472e50 00000000`00000000 00000000`7775d670 00000000`77730920 : wow64!Wow64LdrpInitialize+0x42a
00000000`0010eb60 00000000`776cca90 : 00000000`00000000 00000000`77670e41 00000000`0010f110 00000000`00000000 : ntdll!LdrpInitializeProcess+0x17e3
00000000`0010f050 00000000`7765b69e : 00000000`0010f110 00000000`00000000 00000000`7efdf000 00000000`00000000 : ntdll! ?? ::FNODOBFM::`string'+0x25cf0
00000000`0010f0c0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe


STACK_COMMAND:  .trap 0xfffff88006aa5680 ; kb

THREAD_SHA1_HASH_MOD_FUNC:  dbfd1c8718001d6bf1bf4c8614036f99d76c5b23

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  bb2b8033b6c74e4069a0f00b4027a4e6f51f03e3

THREAD_SHA1_HASH_MOD:  b7fd3d0a19cb3a2bbc48aa7b577ad71c3bba8ecf

FOLLOWUP_IP: 
npcap!WSKSendTo_NBL+d4 [j:\npcap\packetwin7\npf\npf\lo_send.c @ 858]
fffff880`05a01fb0 3d03010000      cmp     eax,103h

FAULT_INSTR_CODE:  1033d

FAULTING_SOURCE_LINE:  j:\npcap\packetwin7\npf\npf\lo_send.c

FAULTING_SOURCE_FILE:  j:\npcap\packetwin7\npf\npf\lo_send.c

FAULTING_SOURCE_LINE_NUMBER:  858

FAULTING_SOURCE_CODE:  
   854:         RemoteAddress,
   855:         0,
   856:         NULL,
   857:         Irp);
>  858:     if (Status == STATUS_PENDING)
   859:     {
   860:         KeWaitForSingleObject(&CompletionEvent, Executive, KernelMode, FALSE, NULL);
   861:         Status = Irp->IoStatus.Status;
   862:     }
   863: 


SYMBOL_STACK_INDEX:  10

SYMBOL_NAME:  npcap!WSKSendTo_NBL+d4

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: npcap

IMAGE_NAME:  npcap.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5767b816

FAILURE_BUCKET_ID:  X64_0xD1_CODE_AV_NULL_IP_npcap!WSKSendTo_NBL+d4

BUCKET_ID:  X64_0xD1_CODE_AV_NULL_IP_npcap!WSKSendTo_NBL+d4

PRIMARY_PROBLEM_CLASS:  X64_0xD1_CODE_AV_NULL_IP_npcap!WSKSendTo_NBL+d4

TARGET_TIME:  2016-06-23T05:50:07.000Z

OSBUILD:  7601

OSSERVICEPACK:  1000

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 7

OSEDITION:  Windows 7 WinNt (Service Pack 1) TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2015-03-17 12:02:04

BUILDDATESTAMP_STR:  150316-1654

BUILDLAB_STR:  win7sp1_gdr

BUILDOSVER_STR:  6.1.7601.18798.amd64fre.win7sp1_gdr.150316-1654

ANALYSIS_SESSION_ELAPSED_TIME: 124e

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:x64_0xd1_code_av_null_ip_npcap!wsksendto_nbl+d4

FAILURE_ID_HASH:  {4a65a334-abd9-00b8-4b67-6fff67ae90f0}

Followup:     MachineOwner
---------

The faulty code line is here: https://github.com/nmap/npcap/blob/4325bdac9e8434186dca295f3b2ae893047b818f/packetWin7/npf/npf/Lo_send.c#L850-L857

The raw socket is created in the NPF_WSKInitSockets function.

I see that when you create your socket, you're using a custom protocol identifier of `IPPROTO_NPCAP_LOOPBACK` which you've defined to be 250. Why do you think that's safe/correct? — Peter Brittain, Jun 29 '16 at 12:02
@PeterBrittain I just want to define a private tag for my own packets. Based on https://opensource.apple.com/source/tcpdump/tcpdump-16/tcpdump/ipproto.h, it seems that the value 250 is unused by any public protocols, so I used it. Is that unsafe? — hsluoyz, Jun 29 '16 at 18:16
I don't know... I was suspicious given that the stack says it was trying to dispatch the packet after the loopback and dying inside the processing of that packet, but it looks like ICMPv6 has its own value and doesn't match 250, so it's probably a red herring. — Peter Brittain, Jun 30 '16 at 17:26

score 3 · Accepted Answer · answered Jun 30 '16 at 22:30

from stack view - you send Icmpv6 datagram to in6LoopbackAddr - and all here correct, no mistakes. because it to in6LoopbackAddr tcpip.sys just Icmpv6ReceiveDatagrams called. in function Icmpv6ReceiveDatagrams exist switch, how packet process, based on 1 byte from packet:

switch (cl)
{
case 0x80: Icmpv6pHandleEchoRequest();break;
case 0x81: Icmpv6pHandleEchoReplyAndError();break;
case 0x82: Ipv6pHandleMldQuery();break;
case 0x83: Ipv6pHandleMldReport();break;
case 0x85: Ipv6pHandleRouterSolication();break;
case 0x86: Ipv6pHandleRouterAdvertisement();break;
case 0x87: Ipv6pHandleNeighborSolicitation();break;
case 0x89: Ipv6pHandleRedirect();break;
}

our case is (87) - Ipv6pHandleNeighborSolicitation(x,y) . and in Ipv6pHandleNeighborSolicitation crash at next line -

call    qword ptr [r8+50h] // 0 at r8+50h

so tcpip try call some callback, but it is zero. i look, what at memory to which r8 point, here some callbacks table. all functions from tcpip.sys (so this not your WSK callbacks):

08 FllQueryInterface
10 WfpInbuiltCalloutNotifyNull
18 FlQuerySubInterface
20 WfpInbuiltCalloutNotifyNull
28 IppCleanupNlp
30 FllMapAddress
38 FllSendPackets
40 FllFastSendPackets
48 FllCancelSendPackets
50 0 - and this 0 and called !

this is on win7. but if look on win8.1 and win10 in same place - already no any callback called - this code is removed. so i guess this is faster win7 bug than your - no memory corruption, wrong calls, not init structs.. but same zero callback, and think not you must init it. and no this callbacks on later windows versions. from another side - i dont sure, are Ipv6pHandleNeighborSolicitation() - function,that you want to be called on packet. may be wrong icmp packet format ? of course this not full response, but something

some place on win8.1 and on win10

@hsluoyz I think this is the right answer: our IPv6 OS detection sends [the NS probe](https://nmap.org/book/osdetect-ipv6-methods.html#idm140159082985520), which appears to exercise this code. Win8.1 did not respond to it, so there shouldn't be a problem removing it. I'll do some more tests to confirm, and would appreciate if you do the same. — bonsaiviking, Jul 01 '16 at 03:57
- now i understand switch in Icmpv6ReceiveDatagrams. Crash only in case NeighborSolicitation packet on win7, however interesting what must be is callback at [r8+50], which zero now — RbMm, Jul 01 '16 at 07:10

Winsock Kernel's "WskSendTo" function causes "DRIVER_IRQL_NOT_LESS_OR_EQUAL" BSOD on Win7 SP1

1 Answers1