1

We saw this answer: Direct communication between two PCI devices

Which goes a long way towards answering the question. But we wanted to poke a bit at it, to see if there's any wiggle room.

So, we are making custom PCIe cards, with custom drivers. The only thing outside our control is the chipset in the Server that the cards are plugged into. if we control which two boards we plug in, and both are programmed to talk to each other, such that "raw" data is fine.. and the device drivers are aware of the desire for direct communication.. can you see a way for direct data transfer? Getting creative?

seanhalle
  • 973
  • 7
  • 27
  • The answer you linked is mistaken, or out of date. Root complexes in current Intel server platforms support peer-to-peer. Note that there is an interaction with VT-d: if address translation is enabled, then the peer's MMIO space needs to be mapped. I'm not sure whether peer-to-peer traffic is enabled by default; I haven't found any documentation that says how to enable it.. – prl Dec 17 '18 at 21:31
  • Thanks for the answer :-) It sounds like this is highly dependent upon which particular chipset, and which particular configuration. So, what we need to do is test with specific model numbers, and create a list of configurations that work. Then assume that if it hasn't been tested, that it does not work. – seanhalle Dec 20 '18 at 20:20

1 Answers1

0

If it is one root complex with multiple ports, then it's possible to have direct endpoint to endpoint communication.

If it's a multi-rooted root complex, then it's not covered by the PCIe specification and vendor-dependent if that feature is implemented. So notice multi-rooted PCIe subsystems in many cases if the bus IDs aren't consecutively numbered. It also depends on the implementation of the enumeration procedures if additional root complexes start their first bus ID by multiples of e.g. 64.

Paebbels
  • 15,573
  • 13
  • 70
  • 139