It is a case-by-case basis, and you just have to use your knowledge of the problem you're solving to make that call. The goal is to avoid logic errors at runtime, not to stamp out warning messages.
Casting one or the other value to be the same as the other removes the warning, but won't prevent runtime problems due to logic errors.
e.g. if you had a large unsigned value and cast that to be signed, then it flips negative, which would mess up comparisions. Conversely, flipping -3 to unsigned will make it into a very large positive value, which would mess up the comparison you're trying. Sure, explicitly casting the values could avoid the messages but those messages are warning you about possible unexpected behaviour from your program, which you need to consider carefully regarding the possible & likely values that these variables can take.