This is not actually a precision issue.
In javascript bitwise operators convert their arguments to 32bit signed values. In 32bit signed form &ff000000 is a negative number, so when you shift it, the bits that come in on the left are 1s instead of 0s (this is to ensure that a twos complement negative number remains negative after shifting). You can still get the behaviour you want by bitwise anding with 0xff after the bitshift, which if you're pulling out the different colour components is probably the best thing to do anyway.
var rgba = 0xFFFFFFFF;
console.log((rgba>>24)&0xff);
var red = (rgba>>24) & 0xff;
var green = (rgba>>16) & 0xff;
var blue = (rgba>>8) & 0xff;
var alpha = rgba & 0xff;
As Marvin Smit mentions above, >>> is an unsigned right shift which always pulls in 0s, so in this specific case you could use that too. Wikipedia describes them as 'arithmetic' and 'logical' right shifts.