Automation of different parts of a framework

Question

I have a software framework consisting of multiple components which need to run sequentially. For example, I have a Python script which parses data, another Python script which POSTs the data through a REST API and persists it into a DB, and a Java program which retrieves the parsed data from the DB and performs some analysis on it. So one output is linked to the input of the other component.

I want to know how I can automate these tasks to run one after the other sequentially on a Windows platform. Is PowerShell a good tool to achieve this sort of sequential execution of programs with inter-related input/output flow?

If PowerShell is a suitable choice, it would be great if someone could give me an example of how to achieve this sort of sequential application execution. I have never used PowerShell before and the tutorials I looked up on PowerShell didn't specifically mention this sort of task.

If it is as simple as three (3) commands, why not just create a .bat file script? The same can be done my putting the three (3) commands into a .ps1 script. Simple is good. BTW, this is not really an SO kind of question. There is no code and it invites a variety of opinions. — lit, May 25 '19 at 19:33
Can you please post a example of how each program consumes the data? Ex `A.exe -input Datahere` — ArcSet, May 25 '19 at 20:02
@ArcSet, for example, `Parser.py ` -> parsed data -> `POSTClient.py ` ->persists data in db -> `Analysis.java ` -> Analysis output. This is the sequence. — , May 26 '19 at 12:04
@lit, I couldn't provide any code because I couldn't figure out where to start from in PowerShell. Could you please tell me how I can write the .ps1 script with three application which need to wait for the previous one to finish executing before the next one starts? — , May 26 '19 at 15:49
@robh - with regards to the question you deleted for running a python script from a PowerShel script: - `Start-Process` expects a filepath. Use `Invoke-Expression` instead. ... `$path = "f:\"; $file = "hello.py"; $cmd = "python " + $path + $file; Invoke-Expression $cmd` — wwii, May 27 '19 at 14:21

score 1 · Answer 1 · answered May 26 '19 at 18:56

1

It may be that I do not yet fully understand the complexity of the situation. I would start by writing a script that runs three (3) programs using the invocation operator &.

=== Run-Them.ps1

& prog1.exe
& prog2.exe
& prog3.exe

answered May 26 '19 at 18:56

lit

14,456
10
65
119

ArcSet · Answer 2 · 2019-05-28T00:41:27.673

0

So what I gather is you need to pass data after each script is run to the next script

So we can do that using RedirectStandardOutput from ProcessStartInfo and StandardOutput.ReadToEnd() from Process

function Run-Process($Location, $Arguments = $Null){
    $info = New-Object System.Diagnostics.ProcessStartInfo
    $info.FileName = $Location
    $info.Arguments = $Arguments
    $info.RedirectStandardOutput = $True
    $info.UseShellExecute = $false
    $info.WindowStyle = [System.Diagnostics.ProcessWindowStyle]::Hidden

    $proc = New-Object System.Diagnostics.Process
    $proc.StartInfo = $info
    $proc.Start() | Out-Null
    $proc.WaitForExit()
    return $proc.StandardOutput.ReadToEnd()
}

$Data = Run-Process Parser.py -Arguments "<filepath>"

$Data = Run-Process POSTClient.py -Arguments $Data

Run-Process Analysis.java -Arguments $Data

edited May 28 '19 at 00:41

answered May 26 '19 at 20:29

ArcSet

6,518
1
20
34

Just a clarification, where do I provide the respective paths of the .py and .java files? Sorry if it sounds stupid but I am not familiar with PowerShell syntax. – May 27 '19 at 12:14
I suspect from the explanation that the various pieces don't pass data around in a way that can be utilised by PS pipelines. I suspect that each component pushes it's data somewhere (a file, a database) and is picked up by the next process. – Nick.Mc May 28 '19 at 03:06
Only the POSTClient.py pushes it to a database and the Analysis.java picks it from the DB. But I still couldn't understand where to put all the filepaths of the files in the above script. – May 28 '19 at 10:28

Nick.Mc · Answer 3 · 2019-05-28T03:10:43.883

Why don't you start with a batch file for simplicity, then you can decide if you require the flexibility and control of powershell. Powershell is definitely worth learning in the long run.

First create your batch file. For Example, create a new text file and rename it MyBatchFile.CMD

Then open it notepad (my favourite editor is currently Visual Studio Code, I recommend it)

How do we run python from here? First we google it:

http://www.cs.bu.edu/courses/cs108/guides/runpython.html

This gives us this example:

C:\python27\python.exe Z:\code\hw01\script.py

Which you will obviously need to adjust for your setup. You'll probably also need to quote it to allow for spaces

Now add a pause to the end so that the window doesn't disappear:

"C:\python27\python.exe" "Z:\code\hw01\script.py"
PAUSE

Save this file and double click it. Did it work? No? adjust it till it does.

Now add your second python file and make sure that it doesn't start till the other one finishes:

"C:\python27\python.exe" "Z:\code\hw01\script.py"
"C:\python27\python.exe" "Z:\code\hw01\script2.py"
PAUSE

Now work out how to run java from the command line:

How do I run a Java program from the command line on Windows?

"C:\python27\python.exe" "Z:\code\hw01\script.py"
"C:\python27\python.exe" "Z:\code\hw01\script2.py"
"C:\Program Files\Java\jdk1.5.0_09\bin\java" "C:\myjars\filenamehere.java"
PAUSE

Now... you can decide if you want to make this batch file more sophisticated with variables, or consider using powershell instead

If you go ahead with a batch file and want to schedule this, make sure you remove the PAUSE from it

Automation of different parts of a framework

3 Answers3