Your macro should be defined as
#define square(x) ((x)*(x))
It is necessary to enclose the x's in parenthesis to prevent any surprises about operator precedence. The outer parenthesis are for the same reason.
Please note that this macro even as corrected above will not work if the parameter is a self-modifying expression. So you may want to consider putting it in all-uppercase or something to alert the user of the macro that it will not behave equivalently to a function call.
The reason your expansion of 36/square(6) does not work as you expect is because of its expansion.
36/square(6)
36/6*6
6*6 <-- 36/6 evaluated
36 <-- 6*6 evaluated
The corrected macro would be expanded thus
36/((6)*(6))
36/(36)
1
Which is the answer you would expect. Also note that 5+1 would also work as an argument because of the inner parenthesis but y++ would not behave as you would expect if reading the macro as a function, hence the reason I recommend naming it SQUARE to alert the user that this is a macro not a function.
Macros only behave as functions if each of their parameters appears exactly once and the syntax is otherwise like an expression (i.e. no {}'s). Also the user cannot pass a pointer to a macro as they can to a function.