I had the same problem after changing the value of Xmx within studio64.exe.vmoptions (inside the directory:.AndroidStudioX.X in the user home directory). I did that after reading the official Google docs how to change the heap size of Android Studio to make it run faster:
https://developer.android.com/studio/intro/studio-config.html#adjusting_heap_size
Immediately after this change Android Studio could no longer start and displayed the same error as you posted. Well, soon I found lots of advices here on stackoverflow to delete the studio64.exe.vmoptions file, which worked, but it also changed back the Xmx value to default 1280MB. But I wanted to keep a larger value of the maximum heap size and after lots of searching and trying I finally found the solution. Originally I just uncommented the line starting with Xmx inside studio64.exe.vmoptions and changed the value to 3g. But I didn't delete the rest of that line, so it looked like this:
-Xmx3g #<------ increase this to most of your RAM
But this doesn't work, I had to move the rest of that line to a separate line like this:
# custom Android Studio VM options, see http://tools.android.com/tech-docs/configuration
-Xms512m
#<------ increase this to most of your RAM
-Xmx3g
And this finally worked. I also checked that the Xmx parameter is actually applied by running the jps -lvm command (as described at the Google docs link above).
I think, that the root of this problem was, that the original line (as created by Android Studio itself) was commented out as a whole, and it also contained that comment "#<------ increase this to most of your RAM" which also had another # characted to left of it. So I thought it can stay there, but it just can't, it must be on a separate line, if you uncomment and edit the Xmx parameter.