Am I doing anything wrong?
No, this is just how the debugger works. You likely have the Visual Studio hosting process enabled (it is enabled by default). The settings manager uses the .exe as the key for the directory where the settings are stored.
The result of this is that you actually have two different settings files. One that's used when you run the program standalone, and one that's used when you run the program under the debugger, because in each case a different process .exe is actually what's running.
In many cases, you can live without the hosting process. It can be disabled in the project settings, under the Debug tab. Uncheck the checkbox labeled "Enable the Visual Studio hosting process". If you do so, then even when debugging the process .exe will be your actual program, and it will use the same settings file as when the program is used standalone.
Personally, I'd leave things as they are. I don't usually find it a problem to have two different settings files in use. I typically am not running the program standalone until debugging is, for the most part, over with.
See What is the purpose of vshost.exe file? for more details.