3

I am trying to measure the PCIe bandwidth of NIC devices using Intel® Performance Counter Monitor (PCM) tools. But, I am not able to understand the output of it.

To measure the PCIe bandwidth, I executed the binary pcm-iio. This binary helps to measure the monitor PCIe bandwidth per PCIe device. After executing the binary I got the following output.


|IIO Stack 2 - PCIe1          |IB write|IB read|OB read|OB write|TLB Miss|VT-d L3 Miss|VT-d CTXT Miss|VT-d Lookup|
|_____________________________|________|_______|_______|________|________|____________|______________|___________|
| Part0 (1st x16/x8/x4)       |4498 M  |9003 M |   0   |3256 K  |   0    |   0        |   0          |   0       |
| Part1 (2nd x4)              |   0    |   0   |   0   |   0    |
| Part2 (2nd x8/3rd x4)       |   0    |   0   |   0   |   0    |
| Part3 (4th x4)              |   0    |   0   |   0   |   0    |
|_____________________________|________|_______|_______|________|________|____________|______________|___________|

I want to understand what is the meaning of IB read, IB write, OB read and OB write.

Anubhav Choudhary
  • 131
  • 1
  • 7

1 Answers1

6

IB write, short for inbound write, is the number of bytes that the PCIe device (specified in the first column) requested to write to main memory through DMA. IB read is the number of bytes that the PCIe device requested to read from main memory through DMA(“Direct Memory Access”). Whether it's a read or write request, it's the device that is issuing the request to the socket (the IIO stack to which the device is connected, to be specific). So it's inbound traffic from the perspective of the socket.

These metrics are measured using the DATA_REQ_OF_CPU uncore event. In particular, the event with umask 0x1 represents a 4-byte inbound write request and the event with umask 0x4 represents a 4-byte inbound read request.

OB write, short for outbound write, is the number of bytes that the processor socket requested to write to the PCIe device through MMIO("Memory-mapped I/O"). OB read is the number of bytes that the processor socket requested to read from the PCIe device through MMIO.

These metrics are measured similarly to the inbound ones, except that the DATA_REQ_BY_CPU uncore event is used instead.

In older versions of pcm-iio, the full form of IB and OB were used.

It's worth noting that DATA_REQ_OF_CPU doesn't measure memory bandwidth, but rather I/O bandwidth for each device that is connected to each IIO stack.

Robert Houghton
  • 1,202
  • 16
  • 28
Hadi Brais
  • 22,259
  • 3
  • 54
  • 95
  • When you say "perspective of the socket" is this a program? I've heard of socket programming but I'm wondering if the socket has a physical/hardware component I can visualize to help understanding. For example the day I realized the "register" is just a set of pins on the chip my mind was blown! Is there something like that for the concept of sockets? – Robert Houghton Jul 25 '19 at 09:55
  • 1
    @InstructionPointer I was referring to the CPU socket. – Hadi Brais Jul 25 '19 at 09:57
  • 2
    @InstructionPointer Registers are not pins on the chip; they are memory components inside the chip. – Hadi Brais Jul 25 '19 at 09:59
  • Wow I was overthinking this, thank you. Since you mentioned pcie I started to imagine NICs with ip:port and sending data back and forth but I'm on the wrong side, the physical port (say ethernet) side. You're describing the side sending data through the pcie bus to the processor which then IB Writes to memory correct? Is that essentially the IIO stack? – Robert Houghton Jul 25 '19 at 10:05
  • 1
    @InstructionPointer Each register has wires so that its contents can be accessed. The IIO stack is an on-chip unit that connects a PCIe bus to the on-chip interconnect (i.e., the mesh in the Intel Xeon SP). So all inbound and outbound traffic for that PCIe bus goes through the IIO stack. The performance events I mentioned in the answer are maintained in the IIO stack itself. – Hadi Brais Jul 25 '19 at 10:11
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/196979/discussion-between-instruction-pointer-and-hadi-brais). – Robert Houghton Jul 25 '19 at 10:36
  • I was reading "What every programmer should know about memory" and Drepper explains that cpu caches use SRAM, is that what the registers use instead of my misconceived "pin" visual I had been using? – Robert Houghton Aug 02 '19 at 02:34
  • 1
    @InstructionPointer Yes, this is the type of memory technology registers are made of. – Hadi Brais Aug 02 '19 at 08:27