Streaming objects from Python to PowerShell?

Question

After some searching and checking previous answers like here Passing objects from python to powershell, apparently the best way to send objects from a Python script to PowerShell script or command is going to be as JSON.

However, with something like this (dir_json.py):

from json import dumps
from pathlib import Path

for fn in Path('.').glob('**/*'):
    print(dumps({'name': str(fn)}))

You can do this:

python .\dir_json.py | ConvertFrom-JSON

And the result is OK, but the problem I'm hoping to solve is that ConvertFrom-JSON seems to wait until the script has completed before reading any of the JSON, even though the invidual JSON objects end on each line. This can easily be verified by adding a line like time.sleep(1) after the print.

Is there a better way to send objects from Python to PowerShell than using JSON objects? And is there a way to actually stream them as they are written, instead of passing the entire output of the Python script after the script completes?

I ran into jq, which was recommended by "people on the internet" as a solution to my type of problem, stating that ConvertFrom-JSON doesn't allow streaming, but jq does. However, this did nothing to improve my situation:

python .\dir_json_slow.py | jq -cn --stream 'fromstream(1|truncate_stream(inputs))' | ConvertFrom-JSON

To make jq play nice, I did change the script to write a list of objects instead of separate objects:

from sys import stdout
from time import sleep
from json import dumps
from pathlib import Path

first = True
stdout.write('[\n')
for fn in Path('.').glob('**/*'):
    if first:
        stdout.write(dumps({'name': str(fn)}))
        first = False
    else:
        stdout.write(',\n'+dumps({'name': str(fn)}))
    stdout.flush()
    sleep(.1)
stdout.write('\n]')

(note that the problem isn't ConvertFrom-JSON holding things up at the end, jq itself only starts writing output once the Python script completes)

mklement0 · Accepted Answer · 2021-11-11T05:03:40.370

As long as each line^[1] that your python script outputs is a complete JSON object by itself, you can use a ForEach-Object call to process each output line as it is being received by PowerShell and call ConvertFrom-Json for each:

python .\dir_json.py | ForEach-Object { ConvertFrom-JSON $_ }

A simplified example that demonstrates that streaming occurs, pausing between lines processed (waiting for a keypress):

# Prompts for a keystroke after each line emitted by the Python command.
python -c 'from json import dumps; print(dumps({''name'': ''foo''})); print(dumps({''name'': ''bar''}))' | 
  ForEach-Object { ConvertFrom-Json $_ | Out-Host; pause }

Note: The Out-Host call is only used to work around a display bug in PowerShell, still present as of PowerShell 7.2: Out-Host forces synchronous printing of the implicit table-formatting that is applied - see this answer.

ConvertFrom-Json - atypically for PowerShell cmdlets - collects all input up front before emitting the object(s) that the JSON input has been parsed into, which can be demonstrated as follows:

# Prompts for a keystroke first, and only after *both*
# strings have been emitted does ConvertFrom-Json produce output.
& { '{ "name": "foo" }'; pause; '{ "name": "bar" }' } |
  ConvertFrom-Json | Out-Host

^{[1] PowerShell relays output from external programs such as Python invariably line by line. By contrast, a PowerShell-native command is free to emit any object to the pipeline, including multiline strings.}

Streaming objects from Python to PowerShell?

1 Answers1