Simulate is a broad term however as your question tag indicates you want to do this on an avd, the following might help you out if you are using eclipse.( I am sure there is a way to do the same in other IDE's as well).
Define Eclipse use of Heap Size
You can always increase or decrease the amount of memory available to eclipse. Doing this you can reduce the amount of memory available for the avd to function. This link quite aptly describes how to manipulate the memory available for eclipse to use.
You can choose whatever max memory "size" eclipse is allowed to use, therefore your avd is also bounded by that limitation. This will make the whole process of development slow however and might even cause a java heap size error. However I think that's the error you want to "simulate" anyway.