If you dig into digital electronic circuits; for bitwise boolean operations (AND, OR, XOR, XNOR) each bit can be done in parallel, and for addition/subtraction they can't be (the carry/borrow from the lowest pair of bits influences the result for the next highest pair of bits, which ...).
However; CPUs are complex and to keep all the pieces synchronized everything typically (excluding extremely rare asynchronous/clockless designs like GreenArrays' chip) gets rounded up to the nearest clock cycle; so an XOR that could be done in 0.5 cycles will cost 1 whole cycle and a SUB that could be done in 0.9 cycles will cost the same 1 whole cycle.
In other words; in general it's extremely unlikely that there will be any difference in performance between XOR (or AND) and SUB (but I'd still prefer XOR or AND because it's "simpler for the CPU in theory" and might cost slightly less power consumption).
A more important consideration (especially for high level languages where there's a compiler that's supposed to do micro-optimizations for you) is source code readability. For bit removal, AND is common practice and SUB is unusual (more likely to confuse the reader briefly). XOR is reasonable if it helps you avoid a NOT (e.g. a = b ^ c
isn't harder to understand than a = b & (~c)
); but you may be able to invert the terminology (e.g. the words "bits to remove" replaced with "bit mask") and end up with a = b & c
without the NOT, and if you can do that it's likely to improve readability.