You can do that by creating a new shortcut to the python.exe or pythonw.exe file, (option available in Window's explorer right-click menu), then right click the shortcut, select properties, and change target to:
"C:\Path\to\Python.exe" "Absolute\or\relative\path\to\file.py"
Then select change icon (still in the properties window), and select your .ico file. Executing the program from this shortcut will automaticaly set the selected icon as taskbar and window icon.
Note though that by executing set shortcut, instead of clicking directly your .py file, the current working directory will be the one of python.exe, and not of your file, you can change that too in the shortcut's properties window, in the "start in" entry field (underneath target), or by using the python methods os.getcwd()
to find the current working directory and os.chdir(path)
to set it