6

I am running jobs on a cluster composed of machines with different architectures: gcc -march=native -Q --help=target | grep -- '-march=' | cut -f3 gives me one of these: broadwell, haswell, ivybridge, sandybridge or skylake.

The executable needs to be the same, so I cannot use -march=native but at the same time the architectures have things in common (I think they all support AVX?).

I am aware that gcc (contrary to Intel icc) does not allow for multiple archictures in a single executable. What I would like to know is if there is a way to ask gcc for the highest set of instructions compatible with all the architectures listed above.

gcc version: 8.1.1

styko
  • 641
  • 3
  • 14
  • And why don't you just compile for generic x64 archutecture with `-march=x86-64`? I guess you need to find compatible instruction set between processor on your own. – KamilCuk Jun 28 '18 at 08:49
  • are you looking for the instruction set intersection for all target architectures, or for a [fat binary](https://en.wikipedia.org/wiki/Fat_binary). Either way (afaik) - assuming you're using gcc on linux - you're out of luck if you want it out of the box. You can do the former by determining the intersection yourself, and the latter has had some experimentation with FatELF. – Sander De Dycker Jun 28 '18 at 09:16

3 Answers3

3

Comments suggested me to look by myself at the 'intersection' between the architectures. The following bash script seems to do the job.

#!/usr/bin/env bash

archs=("broadwell" "haswell" "ivybridge" "sandybridge" "skylake")

for ar in ${archs[@]}; do
    gcc -march=$ar -Q --help=target | grep -- "  -m" > "$ar.log"
done

cp "${archs[0]}.log" all.log
for ar in ${archs[@]:1}; do
    join all.log "$ar.log" > tmp.log
    mv tmp.log all.log
done

cat all.log | grep "\[activé]" | grep -v "\[désactivé]" | cut -d' ' -f1 | tr '\n' ' '

(Computer in French: "activé" => "enabled", "désactivé" => "disabled")

The output is

-m128bit-long-double -m64 -m80387 -maes -malign-stringops -mavx -mcx16 -mfancy-math-387 -mfp-ret-in-387 -mfxsr -mglibc -mhard-float -mieee-fp -mlong-double-80 -mmmx -mpclmul -mpopcnt -mpush-args -mred-zone -msahf -msse -msse2 -msse3 -msse4 -msse4.1 -msse4.2 -mssse3 -mstv -mtls-direct-seg-refs -mvzeroupper -mxsave -mxsaveopt

As I expected all the architectures support both SSE and AVX.

styko
  • 641
  • 3
  • 14
  • Aren't you missing the broadwell architecture due to the zero-indexed arrays of bash? This shouldn't be a concern however since broadwell is pretty much a superset of haswell. If the systems are heterogeneous, [consider using a more portable shebang](https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang) :-) – Margaret Bloom Jun 28 '18 at 15:39
  • You left out `-mtune=haswell`, so you'll still tune for generic CPUs while making a binary that can only run on SnB or later. I'd recommend `-mtune=haswell` (not sandybridge), and make sure your arrays are 32-byte aligned for AVX auto-vectorization. – Peter Cordes Jun 28 '18 at 17:26
3

Intel hasn't ever removed instruction sets in future versions of the same CPU. i.e. a binary that works on an old Intel CPU always works on a newer Intel CPU.

(The one exception to this is first-gen Xeon Phi: Knight's Corner used an incompatible variant of AVX512 called KNI, but later Xeon Phi accelerator cards / computers use AVX512.)

If you must use the same binary on all CPUs, use gcc -march=sandybridge -mtune=haswell, and make sure your important arrays are aligned by 32 bytes.

Maybe worth benchmarking with gcc -march=sandybridge (i.e. with tune=sandybridge) as well, to see which works better for your code. -mprefer-avx128 or -mprefer-vector-width=256 might be interesting to try: some loops get messy when gcc auto-vectorizes with 256-bit vectors.

SnB/IvB have inefficient misaligned AVX loads/stores, so tune=sandybridge sets -mavx256-split-unaligned-load, which sucks a lot if your data is aligned at runtime but the compiler didn't know that. The extra instructions and shuffles aren't helpful on Haswell, so -mtune=haswell includes -mno-avx256-split-unaligned-load.

Unfortunately gcc doesn't have a "tune=avx2" option to tune for all CPUs which have AVX2, or an option to tune for the average CPU which supports the instruction sets you enabled. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80568. Your only choices are tune for a specific CPU, or tune for the generic baseline, or use specific tuning options.

Gcc does has some support for runtime dispatch with ifunc

You have to activate it in the source for specific functions. See https://lwn.net/Articles/691932/ for more about function multi-versioning.

Best option: build separate binaries for SnB / Haswell, and dispatch with a script or $PATH setting

On each cluster node, create a /etc/host-type or whatever, which has sandybridge or haswell or whatever. Any per-node filesystem is fine, or re-detect it at run time with gcc or something cheaper. In your job script:

#!/bin/sh

bin_dir="./bin-$(</etc/node-type)"
exec "$bin_dir/my_prog"  "$@"

Create symlinks as necessary to make bin-skylake and bin-broadwell use the Haswell binaries.

Haswell introduced AVX2 and FMA, and BMI1/2. If you're number-crunching, you really want FMA. BDW/SKL didn't introduce any significant ISA extensions that compilers can use to make your code run faster. Tuning for BDW/SKL is not different either.

If you have any Skylake-avx512 CPUs, that's different.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
1

What I would like to know is if there is a way to ask gcc for the highest set of instructions compatible with all the architectures listed above.

That's a NO.

If you want optimal performance look into fat binaries as Saner De Dycker commented.

An alternative solution though is to compile binaries and libraries for each instruction set and set PATH and LD_LIBRARY_PATH on each system to pick the best instruction set.

Goswin von Brederlow
  • 11,875
  • 2
  • 24
  • 42