Assuming by "three diff parenthesis" do you mean three types, so "[", "{", "("...
Without using a stack I'd try this (pseudo code... yet to try to code it up)
OPEN_PARENS = [ "{", "[", "(" ]
Find last occurrence of one of OPEN_PARENS
Find next occurrence of corresponding closing paren.
If find another closer that does not correspond then error.
When closing paren found replace it with a space (and also the opening paren)
Repeat until no opening paren is found
Check there are no closing parens.
Just an idea, yet to test it. Also it assumes the original input string is mutable.
This does not use a stack and takes up no extra memory, but is now an O(n^2)-ish parse rather than O(n) parse.
EDIT: As Matt points out this still uses memory. It is the memory already allocated for a string but if the string were const, this wouldn't work. So... as Matt suggests just storing a pointer to the last removed open parenthesis would also work. I think you would have to add a count for each type of parenthesis seen too.
Err.. maybe something like this. I don't have time now but will re-check this logic in the evening.
numSq = 0;
numCurley = 0;
numCurve = 0;
OPEN_PARENS = [ "{", "[", "(" ]
Find last occurrence of one of OPEN_PARENS before the last seen (if any other wise search from end of string),
store pointer to it and increment a count for that type of parenthesis.
If none found exit success
Find next occurrence of corresponding closing paren.
If find another closer that does not correspond then decrement count
for that closer. If count reaches negative then there is an error in string.
When closing param found start again
It works on the principle that once you have the inner most matching brackets matched, matches futher out can "ignore" them, but need to track counts because there has to be some kind of "memory" in the system. Using one pointer and 3 counters however, would be O(1) memory. Still same O(n^2)-ish parse time.