Stack Overflow
Stack Overflow

Questions tagged [infiniband]

InfiniBand is a high-speed switched fabric communications link technology used in high-performance computing and enterprise data centers.

InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include scalability, high throughput, low latency, quality of service and failover. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. Infiniband host bus adapters and network switches are commonly manufactured by both Mellanox and Intel.

178 questions
44
votes
1 answer

Difference between IPoIB and TCP over Infiniband

Can someone explain the concepts of IPoIB and TCP over infiniband? I understand the overall concept and data rates provided by native infiniband, but dont quite understand how TCP and IPoIB fit in. Why do u need them and what do they do? What is the…
Korizon
  • 3,677
  • 7
  • 37
  • 52
34
votes
2 answers

Having windows Azure A8 nodes with InfiniBand support how to send N bytes from one and receive on another?

I like InfiniBand promise of 40Gbit/s network. My needs do not map onto the MPI model with one core node + slaves, and if possible I would prefer not to use MPI at all. I need simple connect/send/receive/close (or its async versions) API. Yet…
DuckQueen
  • 772
  • 10
  • 62
  • 134
19
votes
3 answers

InfiniBand explained

Can anybody explain what is InfiniBand? What is the key differences in comparison with Ethernet, how these differences allow for it to be faster than Ethernet? In the official description from mellanox it is written that Introduce InfiniBand, a…
DimanNe
  • 1,791
  • 3
  • 12
  • 19
11
votes
1 answer

RDMA memory sharing

I have a few multi-core computers connected by Infiniband network. I would like to have some low-latency computation on a pool of shared memory, with remote atomic operations. I know RDMA is the way to go. On each node I would register a memory…
Kinson Chan
  • 131
  • 1
  • 4
11
votes
3 answers

"Local" RDMA for development

I'm trying to build and run the RDMA examples here. But because I'm just exploring, I don't have any hardware capable of managing RDMA. I get errors like this when I try to run the example code. librdmacm: couldn't read ABI version. librdmacm:…
brooks94
  • 3,836
  • 4
  • 30
  • 57
10
votes
1 answer

How do I use an InfiniBand network with Dask?

I have a cluster with a high performance network (InfiniBand). However when I set up my Dask scheduler and workers, performance doesn't seem to be as fast as I would expect. How can I tell Dask to use this network? Disclaimer: I'm just asking this…
MRocklin
  • 55,641
  • 23
  • 163
  • 235
10
votes
1 answer

InfiniBand: transfer rate depends on MPI_Test* frequency

I'm writing a multi-threaded OpenMPI application, using MPI_Isend and MPI_Irecv from several threads to exchange hundreds of messages per second between ranks over InfiniBand RDMA. Transfers are in the order of 400 - 800KByte, generating about 9…
Jan David Mol
  • 103
  • 4
10
votes
6 answers

Packet capture in RDMA?

Is there any utility like tcpdump in Linux for capturing the traffic which is going over RDMA channel? (Infiniband/RoCE/iWARP)
dhavale
  • 123
  • 1
  • 6
10
votes
1 answer

GPUDirect RDMA transfer from GPU to remote host

Scenario: I have two machines, a client and a server, connected with Infiniband. The server machine has an NVIDIA Fermi GPU, but the client machine has no GPU. I have an application running on the GPU machine that uses the GPU for some calculations.…
DaoWen
  • 32,589
  • 6
  • 74
  • 101
7
votes
2 answers

Cannot create queue pair with ib_create_qp

I am writing an RDMA (InfiniBand) kernel module. Up to now I have been successful creating the protection domain, completion queues for send and receive queues. But whenever I try to create the Queue Pair by invoking ib_create_qp, it is failing to…
user3243499
  • 2,953
  • 6
  • 33
  • 75
7
votes
1 answer

Infiniband addressing - host names to IB address without IBoIP

I've just started getting familiar with infiniband and I'm wanting to understand the methods you can use to address the infiniband nodes. Based on the code is the example from: RDMA read and write with IB verbs I can address individual nodes by IP…
hookenz
  • 36,432
  • 45
  • 177
  • 286
7
votes
2 answers

MPI_SEND takes huge part of virtual memory

Debugging my program on big counts of kernels, I faced with very strange error of insufficient virtual memory. My investigations lead to peace of code, where master sends small messages to each slave. Then I wrote small program, where 1 master…
vovo
  • 397
  • 1
  • 3
  • 15
5
votes
1 answer

Spark and InfiniBand

I am trying to use Spark in a HPC focused cluster that has infiniband interconnections. This cluster does not provide support for IPoIB. I saw the Spakr-RDMA project from ohio state university in here. I cannot find anyone else working on this, or…
M.Rez
  • 1,802
  • 2
  • 21
  • 30
5
votes
1 answer

What are the PCIe operations involved in Infiniband verbs?

Here are some specifics. When a process calls ibv_post_send(), what happens at the PCI interface to the HCA? Is the WQE encapsulated inside the PCIe doorbell and written via Programmed IO? Or is the WQE fetched in a separate DMA read by the…
Anuj Kalia
  • 803
  • 8
  • 16
5
votes
2 answers

infiniband rdma poor transfer bw

In my application I use an infiniband infrastructure to send a stream of data from a server to another one. I have used to easy the development ip over infiniband because I'm more familiar with socket programming. Until now the performance (max bw)…
Gaetano Mendola
  • 1,344
  • 3
  • 12
  • 27
1
2 3
…
11 12