Python: subprocess.run throws UnicodeDecodeError error

Question

I have a weird issue that I don't understand. I am running a Python FastAPI app with 4 different APIs. In the first one, I have implemented an error handling which after some trouble is working somehow fine. So I copied it's structure to the second API and suddenly it throws an error and won't work anymore.

All of my 4 APIs call a powershell script because it's easy to work with the Active Directory with powershell. After the first ps1 script is called, it calls another ps1 script under a priviledged user so that we get the permission to add groups to a specific OU (organizational unit in AD). So it's like this: Python app -> First ps1 -> 2nd ps1

In python, I'm using the subprocess.run command to call the ps1 script:

result = subprocess.run(["powershell.exe", 
    "./managead/ADMgmtInitial.ps1", 
    "-action",  item.action, 
    "-groupname", providedadgroup, 
    "-techuser", item.techuser,
    "-region", item.region],  
    text=True,  #text=True makes the function to return a string instead of bytes
    stderr=subprocess.PIPE) 

This works fine for the 1st API, but it fails for the 2nd API:

File "C:\dfb-apis\main.py", line 361, in manageadgroup
result = subprocess.run(["powershell.exe",
File "C:\Program Files\Python310\lib\subprocess.py", line 505, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "C:\Program Files\Python310\lib\subprocess.py", line 1144, in communicate
stderr = self.stderr.read()
File "C:\Program Files\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 55: character maps to <undefined>

Afterwards, I check the result object, to see if the ps1 script was executed without issues:

if result.returncode == 0: #0 stands for no returned error; everything was executed without errors
    returncodestring = str(result.returncode)
    
    print ("returncode in Python: " + returncodestring)
    
    stdoutmessage = result.stderr.strip()
    print ("stdoutmessage: ")
    print (stdoutmessage)

    #return specific messages depended on the action type add or remove
    if item.action == "add":
        print("User " + item.techuser + " has been added to the AD group" + providedadgroup)
        return {"message": "Success: User " + item.techuser + " has been added to the AD group " + providedadgroup}
    else: #action == "remove"
        print("User " + item.techuser + " has been removed from the AD group " + providedadgroup)
        return {"message": "Success: User " + item.techuser + " has been removed from the AD group " + providedadgroup}
    
else: # !=0 stands for an error; something went wrong
    returncodestring = str(result.returncode)
    print ("returncode in Python: " + returncodestring)
    errormessagestring= result.stderr.strip()
    print ("Python Error | errormessagestring: " + errormessagestring)
    raise HTTPException(
        status_code=500,
        detail="Failed to add user " + item.techuser + " to the AD group " + providedadgroup +". Details: " + errormessagestring,
        headers={"Error": "Could not add user " + item.techuser + " to the AD group " + providedadgroup},
    )

In the first ps1 script, I use a try-catch-logic to call the 2nd script which would actually add the AD user to the AD group:

try {                         
    #use the service account to trigger a new powershell script for the group creation
    $process = Start-Process -NoNewWindow powershell.exe -Credential $credential -ArgumentList $ArgumentList             
    $exitCode = $process.ExitCode        
    Write-Host "PS1 script ADMgmtInitial: exitCode: " + $exitCode
} catch {
    $errorMessage = $_.Exception.Message
    Write-Host "PS1 script ADMgmtInitial: Error: " + $errorMessage
    Write-Host "PS1 script ADMgmtInitial: The AD Management script has failed"
    $ErrorActionPreference = 'Stop'
    throw $errorMessage
}

Then, there is the 2nd script which actually add the user to the group:

Try {
    Add-ADGroupMember -Identity $groupname -Members $techuser;
} catch {
    $errorMessage = $_.Exception.Message
    Write-Host "PS1 script ADgroupmgmt: Error: " + $errorMessage
    Write-Host "PS1 script ADgroupmgmt: The AD Management script has failed"
    $ErrorActionPreference = 'Stop'
    throw $errorMessage
    Stop-Transcript
}  

At the python app, I tried different variants of subprocess.run and subprocess.Popen. I never was able to read stdout and stderr at the same time. So I was happy when I got it working with the stderr, which for me is more necessary then stdout. But still, the code fails for the 2nd API. Do you have any guesses what's wrong here?

Thanks for any help!

Answer 1

• Your error message implies that your Python script is expecting PowerShell's output to be UTF-8-encoded.

• However, PowerShell by default respects the current code page of the console window associated with it (such a console window is created on demand, if the caller is a GUI-subsystem application, i.e. has no attached console window), as reported by chcp, and which defaults to the system's legacy OEM code page, which is a fixed-single-byte encoding such as CP437 on US-English systems.

  • The bytes of output that is encoded with such a legacy code page that have the high bit set (i.e. byte values >= 128 / 0x80) are typically invalid when mistakenly decoded as UTF-8, resulting in the error you saw.

• For PowerShell to use UTF-8, the code page must be set to 65001, which translates to [Console]::OutputEncoding reflecting UTF-8 inside the PowerShell session.

---

Adapting the second solution from this answer to your use case:

result = subprocess.run(
    f'''\
    chcp 65001 >NUL & powershell.exe -File "./managead/ADMgmtInitial.ps1" -action "{item.action}" -groupname "{providedadgroup}" -techuser "{item.techuser}" -region "{item.region}"
    ''',
     shell=True,              # call via cmd.exe (on Windows)
     text=True,               # decode output as text
     stdout=subprocess.PIPE,  # capture stdout instead of printing to the console
     stderr=subprocess.PIPE   # capture stderr
)

# Print stdout and stderr output as well as the process exit code,
# for diagnostic purposes.
print(result.stdout)
print(result.stderr)
print(result.returncode)