Building an AMI using Packer with Python & Python modules (using pip) installed via powershell script

Question

Using Packer, I am trying to create a Windows AMI with Python + the cryptography module installed. Here is the installation command I'm using for Python:

Invoke-Expression "python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0"

Standalone that works fine. If I launch an EC2 instance from the resulting AMI, I can open Powershell and execute python --version and it returns the Python version. This is to be expected since, according to Python documentation, PrependPath=1 will "Add install and Scripts directories to PATH" In addition, however, I want to install cryptography module so I add the following to the install script:

Invoke-Expression "python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0"
pip install --upgrade pip
pip install cryptography

Now Packer will fail when it gets to the pip command saying The term 'pip' is not recognized as the name of a amazon-ebs.windows: cmdlet, function, script file, or operable program. I tried adding pip's location to the system path in multiple different ways but nothing helped. What did work (as well as the addition to the system path) was adding a sleep after the Python install command. Seemingly Packer/Powershell doesn't wait for the Python installer to finish. So now my install script looks like this:

Invoke-Expression "python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0"
sleep 30
$env:Path += ";C:\Program Files\Python36\Scripts\"
pip install --upgrade pip
pip install cryptography

Now Packer executes no problem and creates the new AMI but when I launch the resulting AMI and run python --version I get 'python' is not recognized as the name of a cmdlet, function, script file, or operable program. Adding commands to the script to append the system path has not helped.

Can anyone shed any light on this predicament?

PS uses for the path the environmental variable PSMODULEPATH which can be modified by pressing start button and typing Edit Environmental Variable. Also see https://gis.stackexchange.com/questions/367816/python-error-when-run-qgis-couldnt-load-sip-module?force_isolation=true — jdweng, Mar 13 '23 at 12:38
As an aside, on a general note: [`Invoke-Expression` (`iex`) should generally be avoided](https://stackoverflow.com/a/51252636/45375); except in unusual circumstances, [don't use it to invoke an external program or PowerShell script / command](https://stackoverflow.com/a/57966347/45375). — mklement0, Mar 13 '23 at 14:08
@jdweng, `PSMODULEPATH` has nothing to do with this question, and neither is the link helpful. What matters with respect to execution of _external programs_ is the `Path` environment variable, which Jay already knows how to modify in-process. — mklement0, Mar 13 '23 at 14:13

mklement0 · Accepted Answer · 2023-03-14T15:54:10.043

As an aside:

Invoke-Expression (iex) should generally be avoided; except in unusual circumstances, don't use it to invoke an external program or PowerShell script / command.
The commands below use direct invocation of ./python-3.6.8-amd64.exe with individual arguments - no need to call via a command line stored in a string.

There are two problems with your code:

Your ./python-3.6.8-amd64.exe installer appears to be GUI-subsystem application, which therefore runs asynchronously by default.
- Your sleep 30 approach to await completion of the installer isn't reliable, as there's no guarantee that it will complete in that time frame, given that execution time may vary based on external factors (system load). Below are two reliable alternatives.
- A simple trick for making such a call synchronous - i.e. to wait for its completion - is to append an additional pipeline segment (what specific command you use in that last segment is irrelevant; here, Out-Host is used, which would print any stdout output (which GUI applications rarely produce) to the console; use Out-Null to suppress any such output); this approach has the additional advantage that the exit code of the process is reflected in the automatic $LASTEXITCODE variable (see this answer for background information):
```
# Note the ... | Out-Host at the end.
python-3.6.8-amd64.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0 | Out-Host
# Inspect $LASTEXITCODE to see if the installer signaled
# failure via a nonzero value.
```
- Alternatively, use Start-Process with the -Wait parameter:
```
Start-Process -Wait python-3.6.8-amd64.exe '/quiet InstallAllUsers=1 PrependPath=1 Include_test=0'
```
  - To obtain the process exit code in this case, use the -PassThru switch and examine the .ExitCode property value of the process-info object that -PassThru emits:
```
$exitCode = (Start-Process -PassThru -Wait python-3.6.8-amd64.exe '/quiet InstallAllUsers=1 PrependPath=1 Include_test=0').ExitCode
```
Whatever modifications the installer makes to the persistent, registry-based Path environment variable ($env:Path) are not immediately seen in the same session (see this answer for background information).
- You can use the following command to explicitly refresh your session's Path value - this should make the pip Python executable discoverable:
```
$env:PATH = [Environment]::GetEnvironmentVariable('Path', 'Machine'),
            [Environment]::GetEnvironmentVariable('Path', 'User') -join ';'
```
- However, there may be a better, Packer-specific approach, mentioned in a since-deleted answer by Paolo: use of a separate shell provisioner in which the modified Path value is automatically seen (I don't know Packer, so I cannot spell out this solution.)

I'm glad to hear the answer helped, and I appreciate the nice feedback as well as the coffee, @Jay. I've just updated the answer, which should address all your points. — mklement0, Mar 14 '23 at 14:10

Building an AMI using Packer with Python & Python modules (using pip) installed via powershell script

1 Answers1