Reducing the value -Xmx on \jdeveloper\bin\ant-sca-compile.xml worked for me:
<target name="scac" description="Compile and validate a composite">
<scac input="${scac.input}" outXml="${scac.output}" error="${scac.error}" appHome="${scac.application.home}" failonerror="true" displayLevel="${scac.displayLevel}">
<jvmarg value="-Xms128m"/>
#<jvmarg value="-Xmx1024m"/>
<jvmarg value="-Xmx700m"/>
<jvmarg value="-XX:PermSize=32m"/>
<jvmarg value="-XX:MaxPermSize=256m"/>
<!-- jvmarg value="-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005"/ -->
</scac>
</target>
If you change jdev.conf you may experience the error:
Unable to create instance of the Virtual Java Machine Located at Path:
C:\Program Files(x86)\Java\jdk1.6.0_45\jre\bin\client\jvm.dll