5

Here are some specifics.

When a process calls ibv_post_send(), what happens at the PCI interface to the HCA? Is the WQE encapsulated inside the PCIe doorbell and written via Programmed IO? Or is the WQE fetched in a separate DMA read by the HCA?

What happens When a process calls ibv_poll_cq()? How does the HCA push CQEs to the system memory? Or, if it's pull based, how does the CPU detect new CQEs in the HCA?

Anuj Kalia
  • 803
  • 8
  • 16

1 Answers1

4

Is the WQE encapsulated inside the PCIe doorbell and written via Programmed IO? Or is the WQE fetched in a separate DMA read by the HCA?

It can be either. The usual way is writing WQE and ringing the doorbell, in which case the HCA will fetch the WQE through DMA. The less common way is what is called "Blue Flame" - part of the PCIe BAR is used for WQEs, and the WQE is written to the HCA the same way the doorbell is written.

How does the HCA push CQEs to the system memory?

HCA uses DMA to write CQE to system memory. Your application has two ways to know about the new completion: poll-based or event-based. ibv_poll_cq() is used in a poll-based approach.

kliteyn
  • 1,917
  • 11
  • 24
  • 2
    Thanks! That's very helpful. Is there a document online that explains this interaction (and provides other useful information about Infiniband hardware?). I found one "verbs implementation guide" but even that didn't tell how the CQE is pushed. – Anuj Kalia Jan 30 '14 at 20:50
  • 2
    Handling of CQEs/WQEs and other things - all this is implementation dependent. Every vendor can do whatever he wants. If you want to understand how verbs are implemented, you just need to check the relevant source code - everything is open. For instance, for Mellanox ConnectX family of HCAs: http://lxr.free-electrons.com/source/drivers/infiniband/hw/mlx4/ If you want to know more than that, you need device PRM (Programming Reference Manual). You need to contact your vendor's support and ask for this doc. – kliteyn Feb 02 '14 at 08:23