You'd better cite a source for such a claim, which is ridiculous on the face of it.
We're talking about x86_64, which means modern processors. These ALUs will complete integer addition, subtraction, and/or comparison in a single clock cycle (cache misses will of course take longer, but depend only on the size and memory layout of the data, not signed-ness). Or even less, with the SIMD coprocessor.
I'd more likely to believe that there's a slight power difference between the two types of comparison, but not a speed difference.
Now, it is possible that for a particular compiler, code generation is worse for one data type vs the other when targeting the x86_64 platform. But that would be a very specialized case, and not likely to apply to all x86_64 compilers. And still, I'd suspect cache effects or background processes were affecting the performance measurement (even performance counters that measure time spent per-process will be affected by a context switch invalidating caches).