1

I'm always running XDP applications on servers with Intel Xeon Gold CPU's, performance was always good and was not a problem - up to 125Mpps with 100 GbE MCX515A-CCAT network card and 2 CPU's inside 1U server.

Today I was trying to make it work on AMD EPYC 7371 and for some reason the performance results were very low — maximum I've got is 13Mpps with the same network card and 1 AMD EPYC 7371. All cores are pushed to their max (100%).

Testing was done on ubuntu 18.04 (and 20.04 as well afterwards). I have installed the same Mellanox drivers as I do for intel (OFED), made other configurations as usual.

I run XDP in Driver mode, JIT enabled. Other tuning as follows:

sudo mlnx_tune -p HIGH_THROUGHPUT
sudo ethtool -G enp65s0 rx 512 tx 512
sudo ethtool -L enp65s0 combined 8
sudo ethtool --show-priv-flags enp65s0
sudo ethtool --set-priv-flags enp65s0 rx_cqe_compress on

Is there anything else I should do to run it on AMD CPU? Because performance just can't be so low and I guess I misconfigured something throughout?

A bit more info about our config: kernel 5.4 for 20.04 and 4.5 for 18.04, we generate traffic from our 2nd machine (TRex) which also has MCX515A-CCAT network card and is connected to 1st machine (AMD one) with Mellanox 100 GbE cable

Qeole
  • 8,284
  • 1
  • 24
  • 52
ov3rdr1v337
  • 89
  • 4
  • Have you recompiled to AMD code? – Surt Nov 18 '20 at 20:36
  • There are some instructions that the other either doesn't define or runs as emulation, that could be the problem. Or you have discovered a bug in the current code. – Surt Nov 18 '20 at 20:46
  • 120Mpps vs 13Mpps, that means x10 decrease on performance just because of that? AMD is using x86_x64 arch (same as intel). When I compiled I also didn't use any -march flags. You sure it can influence the performance so much? – ov3rdr1v337 Nov 18 '20 at 20:48
  • Tried recompiling purely for AMD CPU, didn't help. Zero changes to the performance.but by setting up some options - performance increased up to 25Mpss... – ov3rdr1v337 Nov 18 '20 at 21:47
  • Have you read `https://github.com/pevma/SEPTun-Mark-II/blob/master/SEPTun-Mark-II.rst` the AMD part, might help. – Surt Nov 18 '20 at 21:54
  • 1
    Sanity check: Even with JIT enabled, and unless your kernel has `CONFIG_BPF_JIT_ALWAYS_ON`, the verifier may [fall back on the eBPF interpreter](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/bpf/verifier.c?h=v5.9#n10151) if it fails to JIT-compile for some reason. Have you checked if this might be happening in your case? `sudo bpftool prog` would tell whether the program is JIT-compiled. – Qeole Nov 19 '20 at 09:41
  • interesting issue @ov3rdr1v337 can please cross check the following information 1) BIOS configuration for the NPS mode? 2) NIC NUMA ? 3) cores receiving the NIC packets are on the same numa or not? Please remember Epyc cache hierarchy is different from Intel icelake, skylake, broadwell etc. Also there is no DDIO, but regressing to 10% is way off so please check the configuration and update the ticket – Vipin Varghese Feb 03 '22 at 15:55

0 Answers0