This might already be something you tried but ... if you are running your unit tests as "JUnit plug-in tests" ... then your settings for Xmx and MaxPermSize in eclispe.ini are useless.
eclipse.ini settings allow you to change the settings for your current Eclipse. Any Eclipse instance spawned from this Eclipse will still have the defaults settings (something like Xms40m and Xmx512m). You also have to change these.
Open the launch configuration you use for your tests ("Run > Run Configurations...", select the one you use to launch your unit tests) and go to the "Arguments" tab. There, in the "VM Arguments" text area, input your new memory settings as you did in the eclipse.ini file (I use -Xms256m -Xmx1536m -XX:MaxPermSize=256m
for my most expensive test suites ... but you might need more).
The main issue though might be to try and fix the potential memory leaks of your unit tests. Are you sure you don't have more unloading/memory freeing/... in the tearDown()
? Running your tests under profiling might help (we use Yourkit java profiler to this end where I work... but it isn't free. JConsole might help you there, see also http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html).