Call Python script per PowerShell & passing PSObject and return the parsed data

Question

some background: currently I am querying 4Mio rows (with 50 columns) from a MS SQL server with dbatools into a PSObject (in Batch 10.000 rows each query), processing the data with PowerShell (a lot of RegEx stuff) and writing back into a MariaDb with SimplySql. In average i get approx. 150 rows/sec. Had to use a lot of tricks (Net's Stringbuilder etc.) for this performance, its not that bad imho

As new requirements I want to detect the language of some text cells and I have to remove personal data (name & address). I found some good python libs (spacy and pycld2) for that purpose. I made tests with pycld2 - pretty good detection.

Simplified code for clarification (hint:I am a python noob):

#get data from MS SQL
$data = Invoke-DbaQuery -SqlInstance $Connection -Query $Query -As PSObject -QueryTimeout 1800
for ($i=0;$i -lt $data.length;$i++){
  #do a lot of other stuff here
  #...
  #finally make lang detection
  if ($LangDetect.IsPresent){
    $strLang = $tCaseDescription -replace "([^\p{L}\p{N}_\.\s]|`t|`n|`r)+",""
    $arg = "import pycld2 as cld2; isReliable, textBytesFound, details = cld2.detect('" + $strLang + "', isPlainText = True, bestEffort = True);print(details[0][1])"
    $tCaseLang = & $Env:Programfiles\Python39\python.exe -c $arg
  } else {
    $tCaseLang = ''
  }
}
#write to MariaDB
Invoke-SqlUpdate -ConnectionName $ConnectionName -Query $Query

This python call each time works, but it destroys the performance (12rows/sec) due the loop-call and importing pycld2 lib each time. So, this is a lame solution :) In addition, as mentioned above - I want to use spacy - where some more columns has to parsed for getting rid of the personal data.

I am not sure, if I have the mood to convert the whole PS Parser to python :|

I believe, a better solution might be to pass the whole PSObject from PowerShell to python (before the PS loop starts) and return it as well as PSObject - after it has been processed in python - but I don't know, how I can realize this with python / python function.

What would be your approach/suggestions, any other ideas? Thanks :)

It may actually be faster just to do the text removal with Powershell? — Scepticalist, Nov 28 '20 at 13:41
I already do a lot of parsing, removals etc. with Powershell. Removing emails with powershell is easy as well. But I did not find any data science library as like spacy for detection of names or detection of language as like with pycld2. That is the reason for Python. — TefoD, Nov 28 '20 at 13:48

mklement0 · Accepted Answer · 2020-11-29T21:30:03.163

The following simplified example shows you how you can pass multiple [pscustomobject] ([psobject]) instances from PowerShell to a Python script (passed as a string via -c in this case):

by using JSON as the serialization format, via ConvertTo-Json...
... and passing that JSON via the pipeline, which Python can read via stdin (standard input).

Important:

Character encoding:
- PowerShell uses the encoding specified in the $OutputEncoding preference variable when sending data to external programs (such as Python), which commendably defaults to BOM-less UTF-8 in PowerShell [Core] v6+, but regrettably to ASCII(!) in Windows PowerShell.
- Just like PowerShell limits you to sending text to an external program, it also invariably interprets what it receives as text, namely based on the encoding stored in [Console]::OutputEncoding; regrettably, both PowerShell editions as of this writing default to the system's OEM code page.
- To both send and receive (BOM-less) UTF-8 in both PowerShell editions, (temporarily) set $OutputEncoding and [Console]::OutputEncoding as follows:
  $OutputEncoding = [Console]::OutputEncoding = [System.Text.Utf8Encoding]::new($false)
If you want your Python script to also output objects, again consider using JSON, which on the PowerShell you can parse into objects with ConvertFrom-Json.

# Sample input objects.
$data = [pscustomobject] @{ one = 1; two = 2 }, [pscustomobject] @{ one = 10; two = 20 }

# Convert to JSON and pipe to Python.
ConvertTo-Json $data | python -c @'

import sys, json

# Parse the JSON passed via stdin into a list of dictionaries.
dicts = json.load(sys.stdin)

# Sample processing: print the 'one' entry of each dict.
for dict in dicts:
  print(dict['one'])

'@

If the data to pass is a collection of single-line strings, you don't need JSON:

$data = 'foo', 'bar', 'baz'

$data | python -c @'

import sys

# Sample processing: print each stdin input line enclosed in [...]
for line in sys.stdin:
  print('[' + line.rstrip('\r\n') + ']')

'@

score 0 · Answer 2 · answered Nov 29 '20 at 20:48

Based on the @mklement0's answer, i want to share the completed and tested solution with returning the JSON from python to Powershell with consideration of correct character encoding. I tried it already with 100k Rows on one batch - no issues, running flawlessly and superfast :)

#get data from MS SQL
$query = -join@(
                'SELECT `Id`, `CaseSubject`, `CaseDescription`, 
                `AccountCountry`, `CaseLang` '
                'FROM `db`.`table_global` '
                'ORDER BY `Id` DESC, `Id` ASC '
                'LIMIT 10000;'
                )
$data = Invoke-DbaQuery -SqlInstance $Connection -Query $Query -As PSObject -QueryTimeout 1800

$arg = @'
import pycld2 as cld2
import simplejson as json
import sys, re, logging

def main():
  #toggle the logging level to stderr
  # https://stackoverflow.com/a/6579522/14226613 -> https://docs.python.org/3/library/logging.html#logging.debug
  logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
  logging.info('->Encoding Python: ' + str(sys.stdin.encoding))
  # consideration of correct character encoding -> https://stackoverflow.com/a/30107752/14226613
  # Parse the JSON passed via stdin into a list of dictionaries -> https://stackoverflow.com/a/65051178/14226613
  cases = json.load(sys.stdin, 'utf-8')
  # Sample processing: print the 'one' entry of each dict.
  # https://regex101.com/r/bymIQS/1
  regex = re.compile(r'(?=[^\w\s]).|[\r\n]|\'|\"|\\')
  # hash table with Country vs Language for 'boosting' the language detection, if pycld2 is not sure
  lang_country = {'Albania' : 'ALBANIAN', 'Algeria' : 'ARABIC', 'Argentina' : 'SPANISH', 'Armenia' : 'ARMENIAN', 'Austria' : 'GERMAN', 'Azerbaijan' : 'AZERBAIJANI', 'Bangladesh' : 'BENGALI', 'Belgium' : 'DUTCH', 'Benin' : 'FRENCH', 'Bolivia, Plurinational State of' : 'SPANISH', 'Bosnia and Herzegovina' : 'BOSNIAN', 'Brazil' : 'PORTUGUESE', 'Bulgaria' : 'BULGARIAN', 'Chile' : 'SPANISH', 'China' : 'Chinese', 'Colombia' : 'SPANISH', 'Costa Rica' : 'SPANISH', 'Croatia' : 'CROATIAN', 'Czech Republic' : 'CZECH', 'Denmark' : 'DANISH', 'Ecuador' : 'SPANISH', 'Egypt' : 'ARABIC', 'El Salvador' : 'SPANISH', 'Finland' : 'FINNISH', 'France' : 'FRENCH', 'Germany' : 'GERMAN', 'Greece' : 'GREEK', 'Greenland' : 'GREENLANDIC', 'Hungary' : 'HUNGARIAN', 'Iceland' : 'ICELANDIC', 'India' : 'HINDI', 'Iran' : 'PERSIAN', 'Iraq' : 'ARABIC', 'Ireland' : 'ENGLISH', 'Israel' : 'HEBREW', 'Italy' : 'ITALIAN', 'Japan' : 'Japanese', 'Kosovo' : 'ALBANIAN', 'Kuwait' : 'ARABIC', 'Mexico' : 'SPANISH', 'Monaco' : 'FRENCH', 'Morocco' : 'ARABIC', 'Netherlands' : 'DUTCH', 'New Zealand' : 'ENGLISH', 'Norway' : 'NORWEGIAN', 'Panama' : 'SPANISH', 'Paraguay' : 'SPANISH', 'Peru' : 'SPANISH', 'Poland' : 'POLISH', 'Portugal' : 'PORTUGUESE', 'Qatar' : 'ARABIC', 'Romania' : 'ROMANIAN', 'Russia' : 'RUSSIAN', 'San Marino' : 'ITALIAN', 'Saudi Arabia' : 'ARABIC', 'Serbia' : 'SERBIAN', 'Slovakia' : 'SLOVAK', 'Slovenia' : 'SLOVENIAN', 'South Africa' : 'AFRIKAANS', 'South Korea' : 'Korean', 'Spain' : 'SPANISH', 'Sweden' : 'SWEDISH', 'Switzerland' : 'GERMAN', 'Thailand' : 'THAI', 'Tunisia' : 'ARABIC', 'Turkey' : 'TURKISH', 'Ukraine' : 'UKRAINIAN', 'United Arab Emirates' : 'ARABIC', 'United Kingdom' : 'ENGLISH', 'United States' : 'ENGLISH', 'Uruguay' : 'SPANISH', 'Uzbekistan' : 'UZBEK', 'Venezuela' : 'SPANISH'}
  for case in cases:
    #concatenate two fiels and clean them a bitfield, so that we not get any faults due line brakes etc.
    tCaseDescription = regex.sub('', (case['CaseSubject'] + ' ' + case['CaseDescription']))
    tCaseAccCountry = case['AccountCountry']
    if tCaseAccCountry in lang_country:
      language = lang_country[tCaseAccCountry]
      isReliable, textBytesFound, details = cld2.detect(tCaseDescription, 
            isPlainText = True, 
            bestEffort = True,
            hintLanguage = language)
    else:
      isReliable, textBytesFound, details = cld2.detect(tCaseDescription, 
            isPlainText = True, 
            bestEffort = True)

    #Take Value
    case['CaseLang'] = details[0][0]
    #logging.info('->Python processing CaseID: ' + str(case['Id']) + ' / Detected Language: ' + str(case['CaseLang']))
  #encode to JSON
  retVal = json.dumps(cases, 'utf-8')
  return retVal

if __name__ == '__main__':
  retVal = main()
  sys.stdout.write(str(retVal))
'@

$dataJson = ConvertTo-Json $data
$data = ($dataJson | python -X utf8 -c $arg) | ConvertFrom-Json

foreach($case in $data) {
    $tCaseSubject = $case.CaseSubject -replace "\\", "\\" -replace "'", "\'"
    $tCaseDescription = $case.CaseDescription -replace "\\", "\\" -replace "'", "\'"
    $tCaseLang = $case.CaseLang.substring(0,1).toupper() + $case.CaseLang.substring(1).tolower()
    $tCaseId = $case.Id
    
    $qUpdate = -join @(
        "UPDATE db.table_global SET CaseSubject=`'$tCaseSubject`', "
        "CaseDescription=`'$tCaseDescription`', "
        "CaseLang=`'$tCaseLang`' "
        "WHERE Id=$tCaseId;"
    )

    try{
        $result = Invoke-SqlUpdate -ConnectionName 'maria' -Query $qUpdate
      } catch {
        Write-Host -Foreground Red -Background Black ("result: " + $result + ' / No. ' + $i)
        #break
      }
}

Close-SqlConnection -ConnectionName 'maria'

Please apologize the unfortunate syntax highlighting; script block contains SQL, Powershell and Python..

Call Python script per PowerShell & passing PSObject and return the parsed data

2 Answers2