First, you have to realize that double values are semantically different from integers. They are imprecise, so there is always ever-so-slight error in every double value. The error, however, is small, but in unfortunate case it could be on the other side of zero - technically correct, but not what you want. This understanding is essential; if you need digit-exact arithmetic, you shouldn't be using doubles anyway, but integers/longs. Part of IEEE specification for double values also defines "negative zero", "NaN", "infinity" and so on, so technically the software is correct, but you are not using it the right way for what you want to achieve.
Second, like other people already mentioned, never use string formatting for rounding. If you need 4 decimal places, a much better way is to multiply the number by 10000, take floor/round of it and divide it by 10000 again. However, due to the facts mentioned above, you might again get some small decimal off (such as the 15th decimal digit).
On the other hand, if you just want to get rid of "rounding noise" which is sufficiently close to zero, you can also use this approach which is very robust:
if (Math.abs(x) < 0.000001d) x = 0d;