The old version of the setup was created with InstallScope="PerMachine". The new version is intended to have InstallScope="PerUser"; it also needs to use the same registry keys as the old version creates.
The problem is that whatever values are stored under these registry keys during the upgrade will be overwritten at the end with the initial values stored by the old version. Even deleting these keys manually before the installation will make them reappear (with the wrong values) after the installation process.
I have tried creating a custom action and specifically delete these keys, but the result is the same.
How can I ensure that the old version does not interfere with the installation process of the new version allowing to delete the old Registry Keys and re-create them?
What I found to be working:
Performing a REPAIR immediately after installing the new version will yield the correct results!
Uninstalling the old version manually before installing the new one will not remove the keys, but will allow to overwrite them with the correct values.