Deleting from the device didn't help with Xcode 4.5.1. The build process is caching the image. With a hint from Zorayr's answer to related question "Xcode, can't remove “Launch image” from project," I decided to track the cached image down and destroy it, so it wouldn't keep being built and showing up in my test App.
Eventually, I found that Xcode (at least 4.5) builds the images by default in the following path off your home directory:
~/Library/Developer/Xcode/DerivedData/<ProjectName>-<uniqueID>/Build/Products/<Scheme>-iphone{os,simulator}/<ProjectName>.app/
So in addition to removing the old App image from the device (or simulator,) you also need to remove the image file(s) from the build cache. But just doing something like
rm ~/Library/Developer/Xcode/DerivedData/MyProject-fpwnjlbayydgzxcdlyrwxrwntapr/Build/Products/Debug-iphoneos/MyProject.app/Default*.png
won't always solve your problem. Instead, you might get a strange error in a popup window saying something like "Signed product moved or deleted." (I can't reproduce it right now to get the exact text.) It doesn't show up in the log file, and it just hangs the build process (in Xcode 4.5.1.) To get around that, I deleted all of the files under the <ProjectName>.app/ directory tree and the sibling directory tree <ProjectName>.app.dSYM/ containing (seemingly) the symbol tables and signature files for the parallel products. The build process will then re-create everything with no more complaints.