2

I have the following scenario: My server allocates a buffer of 1MB, which is periodically updated and written to (about every 50ms). The client is connected to the the server via Infiniband and periodically reads that buffer via RDMA Read (potentially even faster than it is updated).

My question is: Is there any way to ensure that the local write operation is atomic so that the RDMA read can only read valid and contiguous memory?

Did I correctly understand that the memory regions are only protected from parallel RDMA operations, but not local memory operations? Is there any way to accomplish this behaviour natively with the ibverbs API?

hey0
  • 43
  • 6

1 Answers1

2

The answer may depend on the device. In RDMA Verbs, there are two different modes for atomic operation (which can be checked using the ibv_query_device verb). With IBV_ATOMIC_HCA, atomic operations are only atomic with respect to other operations from the same device, while IBV_ATOMIC_GLOB means that they are atomic also with respect to CPU operations and other RDMA devices.

If a device uses IBV_ATOMIC_HCA, you need to synchronize the RDMA reads with the CPU writes to ensure data consistency.

haggai_e
  • 4,689
  • 1
  • 24
  • 37
  • Hi, thank you very much for the answer. Just for clarification: This means that I can safely call memcpy(a, b), where b is the registered memory region and a is some other memory location, and the copy process would not be interrupted by another RDMA Read? – hey0 Apr 27 '22 at 14:05
  • 1
    @hey0 Almost certainly **no**, because memcpy makes many small accesses, not one big access – user253751 Apr 28 '22 at 09:05
  • 1
    Yes, in addition, an RDMA read operation may cause the target NIC to break the operation into multiple memory reads. Still, if you are doing an RDMA read from b and also have the CPU read from b at the same time, there shouldn't be any consistency issues. – haggai_e Apr 29 '22 at 06:37