Stack Overflow
Stack Overflow

Questions tagged [rdma]

RDMA refers to "Remote Direct Memory Access," which is a set of networking technologies typically used for high performance, low latency communication.

RDMA refers to "Remote Direct Memory Access," which is a set of networking technologies typically used for high performance, low latency communication.

RDMA networks have three main attributes:

  1. Asynchronous queueing: network operations are submitted by adding requests "work queues" which are asynchronously executed by hardware; when complete, the result is returned to software in a "completion queue."
  2. Kernel bypass: userspace processes can submit operations directly to the network adapter hardware without any system calls.
  3. One-sided operations: one system can read from or write into the memory of a remote system without involving any software on the remote system.
177 questions
11
votes
1 answer

RDMA memory sharing

I have a few multi-core computers connected by Infiniband network. I would like to have some low-latency computation on a pool of shared memory, with remote atomic operations. I know RDMA is the way to go. On each node I would register a memory…
Kinson Chan
  • 131
  • 1
  • 4
11
votes
3 answers

"Local" RDMA for development

I'm trying to build and run the RDMA examples here. But because I'm just exploring, I don't have any hardware capable of managing RDMA. I get errors like this when I try to run the example code. librdmacm: couldn't read ABI version. librdmacm:…
brooks94
  • 3,836
  • 4
  • 30
  • 57
10
votes
1 answer

InfiniBand: transfer rate depends on MPI_Test* frequency

I'm writing a multi-threaded OpenMPI application, using MPI_Isend and MPI_Irecv from several threads to exchange hundreds of messages per second between ranks over InfiniBand RDMA. Transfers are in the order of 400 - 800KByte, generating about 9…
Jan David Mol
  • 103
  • 4
10
votes
6 answers

Packet capture in RDMA?

Is there any utility like tcpdump in Linux for capturing the traffic which is going over RDMA channel? (Infiniband/RoCE/iWARP)
dhavale
  • 123
  • 1
  • 6
10
votes
1 answer

GPUDirect RDMA transfer from GPU to remote host

Scenario: I have two machines, a client and a server, connected with Infiniband. The server machine has an NVIDIA Fermi GPU, but the client machine has no GPU. I have an application running on the GPU machine that uses the GPU for some calculations.…
DaoWen
  • 32,589
  • 6
  • 74
  • 101
8
votes
6 answers

How to use DMA or RDMA in java?

"DMA" here means: Direct memory access, and "RDMA" is: remote direct memory access. I used Java to created an application to transfer stock data, but I found the latency is bigger than I expected. I heard someone developed same type application used…
Freewind
  • 193,756
  • 157
  • 432
  • 708
8
votes
1 answer

how is tcp(kernel) bypass implemented?

Assuming I would like to avoid the overhead of the linux kernel in handling incoming packets and instead would like to grab the packet directly from user space. I have googled around a bit and it seems that all that needs to happen is one would use…
Palace Chan
  • 8,845
  • 11
  • 41
  • 93
7
votes
2 answers

Cannot create queue pair with ib_create_qp

I am writing an RDMA (InfiniBand) kernel module. Up to now I have been successful creating the protection domain, completion queues for send and receive queues. But whenever I try to create the Queue Pair by invoking ib_create_qp, it is failing to…
user3243499
  • 2,953
  • 6
  • 33
  • 75
6
votes
3 answers

Are one-sided RDMA reads atomic for single cache lines?

My group (a project called Isis2) is experimenting with RDMA. We're puzzled by the lack of documentation for the atomicity guarantees of one-sided RDMA reads. I've spent the past hour and a half hunting for any kind of information at all on this…
Ken Birman
  • 1,088
  • 8
  • 25
6
votes
3 answers

How do I take advantage of RDMA in windows

How can I copy blocks of memory from one server to another under windows using RDMA? We don't have infiniband, but we do have 10gb network switches. All I need is an example, but I'm not having much luck with google. EDIT: Well no one has…
johnnycrash
  • 5,184
  • 5
  • 34
  • 58
5
votes
1 answer

C++ atomics and memory_order with RDMA

When using one-sided RDMA on modern memory, lock-free, the question arises of how a remote reader can safely view their incoming data if the data objects span multiple cache lines. In the Derecho open-source multicast and replicated logging library…
Ken Birman
  • 1,088
  • 8
  • 25
5
votes
1 answer

What are the PCIe operations involved in Infiniband verbs?

Here are some specifics. When a process calls ibv_post_send(), what happens at the PCI interface to the HCA? Is the WQE encapsulated inside the PCIe doorbell and written via Programmed IO? Or is the WQE fetched in a separate DMA read by the…
Anuj Kalia
  • 803
  • 8
  • 16
5
votes
2 answers

infiniband rdma poor transfer bw

In my application I use an infiniband infrastructure to send a stream of data from a server to another one. I have used to easy the development ip over infiniband because I'm more familiar with socket programming. Until now the performance (max bw)…
Gaetano Mendola
  • 1,344
  • 3
  • 12
  • 27
4
votes
1 answer

How to use the RDMA on Azure

I purchased 1 H16r Virtual Machine on Azure, and I tried the setting method of RDMA on "About H-series and compute-intensive A-series VMs"： azure config mode arm azure vm extension set RDMAUpdateForLinux…
ZhouYoung
  • 41
  • 2
4
votes
1 answer

What happens when RDMA operates over mmaped region?

RDMA is an efficient way to bypass the useless data copies between application and OS kernel. Mmap is an efficient way to deal with a large file as if it is just an array of bytes. I am working with MPI over Infiniband which supports for RDMA…
syko
  • 3,477
  • 5
  • 28
  • 51
1
2 3
…
11 12