8

I have a NIC card and a HDD both connected on PCIe slots in a Linux machine. Ideally, I'd like to funnel incoming packets to the HDD without involving the CPU, or involving it minimally. Is it possible to set up direct communication along the PCI bus like that? Does anyone have pointers as to what to read up on to get started on a project like this?

Thanks all.

Trev
  • 222
  • 4
  • 9
  • Are you using any programming for this? – Jonas May 12 '10 at 20:46
  • Yes. Nothing's been written yet, but I'm assuming it will be in C. I'm just wondering if it's even possible to directly communicate like that. – Trev May 13 '10 at 15:37

2 Answers2

10

Not sure if you are asking about PCI or PCIe. You used both terms, and the answer is different for each.

If you are talking about a legacy PCI bus: The answer is "yes". Board to board DMA is doable. Video capture boards may DMA video frames directly into your graphics card memory for example.

In your example, the video card could DMA directly to a storage device. However, the data would be quite "raw". Your NIC would have no concept of a filesystem for example. You also need to make sure you can program the NIC's DMA engine to sit within the confines of your SATA controller's registers. You don't want to walk off the end of the BAR!

If you are talking about a modern PCIe bus: The answer is "typically no, but it depends". Peer-to-peer bus transactions are a funny thing in the PCI Express Spec. Root complex devices are not required to support it.

In my testing, peer-to-peer DMA will work, if your devices are behind a PCIe switch (not directly plugged into the motherboard). However, if your devices are connected directly to the chipset (Root Complex), peer-to-peer DMA will not work, except in some special cases. The most notable special case would be the video capture example I mentioned earlier. The special cases are mentioned in the chipset datasheets.

We have tested the peer-to-peer PCIe DMA with a few different Intel and AMD chipsets and found consistent behavior. Have not tested the most recent generations of chipsets though. (We have discussed the lack of peer-to-peer PCIe DMA support with Intel, not sure if our feedback has had any impact on their Engineering dept.)

myron-semack
  • 6,259
  • 1
  • 26
  • 38
  • If we control which two boards we plug in, and both are programmed to talk to each other, such that "raw" data is fine.. and the device drivers are aware of the desire for direct communication.. can you see a way for direct data transfer? Getting creative? The cards are both custom, under our control, and so are the device drivers. The only thing outside our control is the chipset in the Server that the cards are plugged into..? – seanhalle Dec 17 '18 at 19:08
  • @seanhalle Use a server with a PCIe-to-PCIe packet switch, or design your own PCIe backplane/riser card with a packet switch, or put a sideband signaling connector on your boards and connect them directly. – myron-semack Dec 17 '18 at 19:11
2

Assuming that both the NIC card and the HDD are End Points (or Legacy Endpoints) you cannot funnel traffic without involving the Root Complex (CPU).

PCIe, unlike PCI or PCI-X, is not a bus but a link, thus any transaction from an Endpoint device (say the NIC) would have to travel through the Root Complex (CPU) in order to get to another branch (HDD).

Jacko
  • 12,665
  • 18
  • 75
  • 126
Nadav
  • 563
  • 3
  • 17