If x is always a power of 2, and n is always a power of 2, then you can you can compute it easily and quickly using bit operations on a byte array, which you can then reconstitute into a "number".
If 2^N is (binary) 1 followed by N zeroes, then (2^N)^2 is (binary) 1 followed by 2N zeros.
2^3 squared is b'1000000'
If you have a number 2^K (binary 1 followed by K zeroes), then 2^K - 2 will be K-1 1s (ones) followed by a zero.
eg 2^4 is 16 = b'10000', 2^4 - 2 is b'1110'
If you require "% 2^M" then in binary, you just select the last (lower) M bits, and disregard the rest .
9999 is b'10011100001111'
9999 % 2^8 is b'00001111'
'
Hence combining the parts, if x=2^A and n=2^B, then
(x^2 - 2 ) % n
will be: (last B bits of) (binary) (2*A - 1 '1's followed by a '0')