I am not sure what this means: "Application is background only = YES" ?
I tried to replicate your problem with my app foo and the ...947 build of Pyinstaller (a recent, dev version.) I used --onefile --windowed args to Pyinstaller. I found that in the Finder, double clicking build/foo starts a console, and then the app window appears. While double clicking the bundle build/foo.app did not seem to start a console, and my app window appears. In other words, it works for me.
On the other hand, I tried another packager: Whitebox Packages In creating a package, I put the pyinstaller/foo/build/foo file (the executable) into the bundle. Etc. 'Packages' created a .mpkg file for me, which I installed. Then, when I invoked my app, a console did appear, and them my app's window. (Which is not good, I don't want a console.) I don't understand what attributes of the package make the app start without a console. I don't think it is the case that the executable include a shell that starts a console, it must be OSX that is starting the console.
Probably, it is not a high priority of the PyInstaller project to do 'packaging for distribution' correctly, since other tools exist.