The output is ALWAYS an integer. If you really want to use this method you only can covert in your case binary into an integer.
So if you try:
parseInt("10101",2) the result would be 21. And this is correct. But in your case you provide to the method absolute invalid arguments because you may only use 0 and 1 for a binary input. The conversion goes left to right. The first invalid value will force to return the result without further compute.
Your 1st case:
parseInt('123',2)
The number 2 is an invalid binary value. This will make to finish after the 1 as input, which is a 1 as integer. It is correct.
Your 2nd try:
parseInt('123', 3)
The last number 3 is invalid for a base of 3 with 0,1,2 as valid numbers. And if you convert the number 12 with the base of 3 the result is 5. From my point of view the method works as expected.