It's determined by the assemblers, that is, software.
For x86, there exist two conventional syntaxes: AT&T and Intel syntax. While the former is used in GNU as and the GNU toolchain and programs built with it in general, the Intel syntax is the syntax described in the Intel Manual and used in products by Intel.
These syntaxes are determined by the assemblers, which translate instructions to machine code. Theoretically speaking, I could invent a new syntax, write an assembler for it (or alter the grammar of an already existing one), and use it.
The hardware does not know about the assembly syntax. It can only process the bits and bytes an instruction represents. nop
, for example, corresponds to 0x90 = 0b10010000
. You could write entire programs like Counterstrike using just these binary codes, so no assembler is actually needed.