4

I'd like to check the option to debug my kernel driver installed over remote physical machine (since I don't have firewire cables). Reading the relevant documentation, I haven't seen any limitation about remote physical debugging medium, so I deduced both firewire cables and ip over wireless network should work.

I thought that lldb remote connection using kdp-remote <machine-ip> would do the trick, but I don't get any response.

From remote VM however, it succeed even though the VM can be located on remote physical machine.

My boot-args configuration are keepsyms=1 debug=0x144 -v

Zohar81
  • 4,554
  • 5
  • 29
  • 82
  • 1
    Can you clarify your setup please? What exactly works, what doesn't? The target machine needs to use either physical ethernet (PCIe/Thunderbolt based) or FireWire. The client machine (where lldb runs) just needs to be able to reach the target machine via IPv4 (wifi this end is fine). For FireWire, the lldb machine needs to either run the `fwkdp` proxy locally with the other end of the firewire cable, or connect to another machine with the firewire cable running the proxy. – pmdj May 08 '18 at 13:29
  • My target machine uses physical standard ethernet cable connected to macPro using ethernet to thunderbolt adaptor. The client machine connected through wifi on the same subnet (I've checked ping before getting the panic). However, when the panic occurs, the ping stop responding .. and finally the boot-args configuration is mentioned above. – Zohar81 May 08 '18 at 14:21
  • Just a thought, my ethernet connection in target machine uses DHCP. perhaps causes the IP to change when the machine goes to panic ? – Zohar81 May 08 '18 at 14:35
  • 1
    Thunderbolt Ethernet usually means the device isn't en0 - you can check with the ifconfig command. You need to select the device in the boot args, I'll find you the exact setting – pmdj May 08 '18 at 14:39
  • I actually tried to set the kdp communication from the correct interface by adding `kdp_match_name=en4` to boot-args (now it contains the following params, not sure if the order matters `keepsyms=4 debug=0x144 kdp_match_name=en4 -v`), but it still doesn't work – Zohar81 May 08 '18 at 14:50
  • 1
    The order doesn't matter. `keepsyms` is a boolean, so really should be 1; I'm not sure if specifying 4 has any negative effects, I don't think it should though. I guess a few other things you could check: 1. The protocol uses UDP, not TCP. Have you got a firewall running on your lldb machine which might be blocking UDP packets? (You could try sending udp packets from target to client with `nc` when the machine is not crashed.) 2. Is the ARP entry correct on the client machine? `arp ` should yield the target interface's MAC address. – pmdj May 08 '18 at 15:51
  • 1
    3. You mention the target is a Mac Pro. Can you try with the built-in ethernet device rather than a Thunderbolt ethernet adapter? 4. The crashed Mac will NOT respond to pings, only to KDP packets via UDP. So not getting pings back doesn't mean anything. 5. As far as I'm aware the machine won't request a new DHCP lease when it crashes, so that shouldn't be the problem. 6. What does `ioreg -rtc IOKernelDebugger` output? – pmdj May 08 '18 at 15:54
  • 1
    (Re: 5/DHCP - you could always set a static IP as a test just to rule this out.) **7. Did you reboot after setting the `boot-args`?** Without a reboot, it won't work and may even forget the settings on crash. You can check with `nvram boot-args` whether the settings were properly remembered. 8. If SIP is active, you can only set nvram variables from the recovery environment from 10.11 onwards. You'll get an error otherwise though, so you'd have noticed that so it shouldn't be the problem. – pmdj May 08 '18 at 16:07
  • 1
    It turns out that that my firewall blocked UDP communication. thank you for all the insights. Now it's working, although it take 10-20 seconds after the crash and before the kdp start worked. – Zohar81 May 09 '18 at 08:26
  • In my experience, using Firewire for kdp is much faster, and the other big advantage is being able to use kprintf. So if you can, consider getting Thunderbolt to Firewire adapters or Thunderbolt dock with a Firewire port. It's a little tricky with TB3 macs as I don't think there's a straight adapter for those, so you'd need to use a TB2->TB3 adapter in that case. Or get a TB3 dock with a Firewire port, if that exists. Or get a refurbished/used Mac Mini from 2012 as a test machine, those still had FW ports built in. – pmdj May 09 '18 at 10:18

1 Answers1

2

We figured out the problem in the comments (item 2 below), but for posterity, here's a list of things to check if xnu kernel debugging isn't working:

  1. The target machine must have a physical ethernet port which is connected via PCIe or Thunderbolt, or you must use a direct firewire connection (optionally via Thunderbolt). USB to ethernet adapters will not work on the target end. The client machine is less fussy, you can use wifi or USB-ethernet there.
  2. The protocol uses UDP, not TCP. Have you got a firewall running on your lldb machine which might be blocking UDP packets? (You could try sending udp packets from target to client with the nc (netcat) tool while the machine is not crashed.)
  3. Is the ARP entry correct on the client machine? arp <target ip> should yield the target interface's MAC address.
  4. The crashed Mac will NOT respond to pings, only to KDP packets via UDP. So not getting pings back doesn't mean anything.
  5. As far as I'm aware the machine won't request a new DHCP lease when it crashes, so that shouldn't be the problem, but you can always try setting a static IP address just to be sure.
  6. Did you reboot after setting the boot-args? They only take effect on a fresh boot.
  7. If SIP is active, you can only set nvram variables from the recovery environment from OS X/macOS 10.11 onwards. You can run nvram boot-args to verify that the settings stuck.

My personal recommendation is to use FireWire for kernel debugging if possible, it seems to be the fastest and most reliable in my experience.

pmdj
  • 22,018
  • 3
  • 52
  • 103