When your macro is expanded, the code looks like this:
printf("%d\n", '3' - '0' * 5);
at which point, the usual arithmetic precedence kicks in, and the multiplication happens first. So you get 51-(48*5), which gives the answer you see.
You could avoid this be putting your macro expansion in parantheses like this:
#define toDigit(c) ((c) - '0')
Ideally, avoid macros for this purpose. Or for any other purpose, for that matter. They are error prone, and can easily be replaced by a function, like this:
int toDigit(char c) { return c - '0'; }
You can use this inside complicated expressions without any issue.