Try Run
menu -> Edit Configurations...
-> find your project in the tree of projects on the left, look for VM options:
in the panel on the right, and enter something there, according to information found here: What are the -Xms and -Xmx parameters when starting JVM?
That having been said, I should also add that if you are running out of memory without knowingly doing extremely memory hungry stuff, then what you have in your hands is a bug which is causing your program to do runaway memory allocation, which will always be resulting in out-of-memory errors no matter how much you increase your heap size. In that case, you will need to look at your code, not at your project options.