The language you're trying to define with your regular expression unfortunately smells non-regular, i.e. regular expressions are not suitable for this type of expressions. (To be precise, "well balanced parenthesis" is not something you can define with regular expressions.)
If you however simply want to find the substring a * b
in your example, the following expressions should do:
Pattern p = Pattern.compile("\\(([^()]*)\\)");
Matcher m = p.matcher("(((a * b) * ) + c) * d)");
if (m.find())
System.out.println(m.group(1)); // prints "a * b"