Opcodes for the 80x86 - like those of the predecessors the 8080 and 8085 - are best understood in octal, not hexadecimal. It's understood as 8 bits per byte with 2+3+3 bits, so the first octal digit ranges over 0-3, while the other two each range over 0-7.
The floating point escape has the octal form 33c xrm D, for Escape #c, with nominal register #r and operand mode (x,m), where D is one byte of the type int8_t for x = 1; 2 bytes of the type uint16_t for x = 2 or for (x,m) = (0,6); and 0 bytes otherwise.
For x = 3, m is nominally a second register: register #m, while for x = 0, 1 or 2 (except from (x,m) = (0,6)), m nominally denotes a combination of indexing registers (BX+SI, BX+DI, BP+SI, BP+DI, SI, DI, BP, BX) for m = (0,1,2,3,4,5,6,7) respectively. The indexing is inapplicable for x = 3 (since m denotes a register, not a memory address) and is 0 or (x,m) = (0,6). The displacement is Disp = 0, if D is 0 bytes, Disp = (int16_t)D, if D is 1 byte; Disp = D, if D is 2 bytes.
Opcodes that are not supported trigger an exception in the CPU, as interrupt 6 - starting with the 80186.
I'm not sure exactly what handshaking goes on with the 33c opcodes nor what happens with interrupt 6, if anything. There's a hardware issue there, with synchronizing access to the data bus. For this purpose, the WAIT opcode (233) is present in the 8086 to allow the CPU to defer. An 8087 hooked up to the 8086 would do its thing before the 8086 resumed.
But for the 017 opcode and the other opcode-holes in the 8086, exception interrupt 6 gets triggered.
Whoever programs the bare-to-the-metal program for the CPU (by definition: The Firmware) is responsible for writing an exception-handler for interrupt 6 (and for all other interrupts and exceptions). The return address points to the start of the invalid operation, and this is done to give a hook for firmware to implement its own extensions of the opcode set.
There's nothing in the 80x86 language that gives a firmware programmer direct access to the "effective address" interpretation of (x,m,D) or "the register" for r, so they'd have to be explicitly interpreted in the firmware's exception-handler. But I think (x,m,D) drives the data bus at the hardware level, so that part of the task of interpretation is passed on to the hardware engineer. The task of interpreting "r", however, I think still has to be handled in firmware.
By the way, other places where the understanding implied by the octal format gets lost in the translation when people use hexadecimal for the opcodes, include the cases where the octal digit in the middle of an opcode encodes an operation.
What gets lost is that it's the same encoding that appears in various places. For instance, the opcodes 0pq denote operations (add, or, adc, sbb, and, sub, xor, cmp) for p = (0, 1, 2, 3, 4, 5, 6, 7) respectively; with different addressing modes q = (0, 1, 2, 3, 4, 5); cases q = (0, 1, 2, 3) taking on extra xrm D bytes, q = 4 takes on an extra 1-byte uint_8 value, and q = 5 an extra 2-byte uint_16 value.
It's the same "p" that occurs in the opcodes 20c xpm D, for c = (0, 1, 2, 3). So these have no register #r, as the r in xrm is replaced by the operator p. (Instead, they take on extra numeric bytes: 2 bytes for c = 2 and one byte for c = 0, 1 or 3). These are the "p" operations between the register or memory denoted by (x,m,D) and the extra 1 or 2 bytes of numeric data appended to the operation.
When I brought up the issue of the octal legacy the 8086, on the USENET, that prompted the creation of NASM (Netwide Assembler) in direct response, which is why it used - and still uses - octal internally for the opcodes.