GPUDirect peer-to-peer access and memory transfer between two Titan x GPUS

Question

I want to know if it is possible to use the peer-to-peer memory transfer and memory access between two Titan X GPUs. Currently I have only one GPU, and if it is possible I may get another one. I know that similar question had been asked before here and here, but those questions are more than 2 years old. I'm asking if there is anything new about that.

Answer 1

It should be possible with Titan X. There are some considerations, however:

1. Operating system/driver model - On Windows, GPUs can be either in WDDM mode or TCC mode. P2P is not supported in WDDM mode AFAIK. So you should put your GPUs in TCC mode, which is supported for Titan X (although not for most other GeForce products) in recent drivers. CUDA supported Linux OS'es should not have any additional considerations in this vein.

2. System topology - GPU Direct P2P depends on both source and destination device being on the same PCIE fabric, which effectively means they must be attached to the same PCIE root complex. If your system topology is such that one PCIE slot originates from one CPU socket, and another PCIE slot originates from another CPU socket (e.g. in a dual-socket motherboard) then those PCIE fabrics will be logically separate (they are serviced by separate root complexes) and P2P transfer between the two will not be possible. Furthermore, the topology decision depends on the GPU driver being able to discover it and recognize it properly. If this does not happen for some reason, then an otherwise compatible system may still not work for P2P. For enterprise/production use, then, the general recommendation is to purchase a system that has been certified by the manufacturer for P2P access between the GPUs. As such, this answer is by no means a guarantee that your particular set of components will work properly.

Data transfer between two GPUs that are not in a proper P2P environment is still possible, of course, but the data will flow through a path that more directly involves the CPU.

Note that in general, P2P support may vary by GPU or GPU family. The ability to run P2P on one GPU type or GPU family does not necessarily indicate it will work on another GPU type or family, even in the same system/setup. The final determinant of GPU P2P support are the tools provided that query the runtime via cudaDeviceCanAccessPeer. So the statement here "should be possible with Titan X" should not be construed to refer to any other GPU type. P2P support can vary by system and other factors as well. No statements made here are a guarantee of P2P support for any particular GPU in any particular setup.