I think you're seeing the effects of evicting useful programs from main memory to the disk.
Your intentionally-leaky program is trying to allocate all memory on the system. To satisfy your program's demand, Windows is finding other programs on the system, writing their memory to the page file, and reallocating their memory to your program. When you see the page file usage jump to maximum, it's because most other programs have been shoved there instead of main memory.
When your program exits, Windows reclaims all the program's memory (as noted by others). But all the other programs on your computer still have their memory saved in the page file on disk, not in main memory. So when they run, Windows has to go load their memory pages from disk, making the program appear slow. After a while, the programs will move back to main memory, and performance will look like normal.