Let us take an example to understand your code.
1) Suppose you pass the character 'p' to your method toUpper().
2) Now the condition in your if-statement will be always true since any alphabet is between 'a' and 'z'.
3) Inside if-statement you have the following code
return (char) (c & ~('a' - 'A'));
4) In above statement this part
('a' - 'A')
will be always executed first since it is in parenthesis. Here you are just subtracting 'A' from 'a' i.e. 97-65 which are ASCII values. (A---> 65 and a----> 97). So the answer would be always 32 regardless of the character you pass to your toUpper() method.
5) How then there is the operator ~ i.e.
~('a' - 'A')
As I told the answer of ('a' - 'A') would be always 32, therefore the operator ~ is applied on the number 32 i.e.
~32
To predict the output of the operator ~, the formula is as shown below
~(number)
= -(number) - 1
Since here the number is 32 therefore the output of ~32 from above formula is
-(32) - 1
= -32 - 1
= -33
So the output of
~('a' - 'A')
would be always -33
5) Now you have
(c & ~('a' - 'A'))
i.e
(c & -33)
Here c has the alphabet which user passes, in our example it is 'p'
i.e.
p & -33
Since the ASCII value of 'p' is 112 i.e
112 & -33
i.e.
1110000 & 1011111
which are the corresponding value of 112 and -33 in binary
So after applying the operator & we get
1010000
6) Now convert 1010000 into decimal we get 80 which is the ASCII value of uppercase alphabet 'P'.
7) So in general we can say the operation that is going to be performed is
(ASCII value of user inputted alphabet) & -33
8) One more thing "Java goes for Unicode. But the first set of characters in Unicode are ASCII, also said by @paxdiablo. So I have mentioned ASCII in above answer.