4

RDMA is an efficient way to bypass the useless data copies between application and OS kernel. Mmap is an efficient way to deal with a large file as if it is just an array of bytes.

I am working with MPI over Infiniband which supports for RDMA network operations between processes. Each MPI process has a very large file to share with others.

Can each MPI process make mmap region over each large file and share it with others? I want to allow each process to read any file of any process as if it reads their memory via RDMA (MPI's one-sided communication).

As far as I know, when an application invokes RDMA operation, it passes the 'virtual memory address' to NIC directly. NIC will deal with the translation from the virtual memory address to its physical memory address. If RDMA driver pins the page of the interests before it issues the request to NIC, I think it will work. Is there any one who have any experience on it? :D

thanks

syko
  • 3,477
  • 5
  • 28
  • 51

1 Answers1

5

Yes, you can register with RDMA a region of memory mapped using mmap(). See the docs for ibv_reg_mr (http://www.rdmamojo.com/2012/09/07/ibv_reg_mr/) which say:

Every memory address in the virtual space of the calling process can be registered, including, but not limited to:

  • Local memory (either variable or array)
  • Global memory (either variable or array)
  • Dynamically allocated memory (using malloc() or mmap())
Community
  • 1
  • 1
John Zwinck
  • 239,568
  • 38
  • 324
  • 436
  • thanks for the nice answers :D I hope Intel's MPI Library is using the concept both in Linux and Windows. (It seems necessary to pin the pages of interests, so that RDMA can read the pages safely.) – syko Nov 28 '15 at 14:46