executing multiline terminal python commands during code execution from within code body

Question

I would like to use some code line speed indicator related libraries such as scalene and tamppa to evaluate which code lines consume more times. We will need to run some command lines in the terminal, before and after the code execution, for seeing the results. For example using tamppa library, if we have the following code (test.py) and execute it in PyCharm:

from line_profiler import LineProfiler
import random

def do_stuff(numbers):
    s = sum(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

I must save the output log by using commands python test.py > mem_res_1.txt or python -m memory_profiler test.py > mem_res_1.txt in the terminal:

1. going to directory -------> cd C:\Users\Ali\Desktop
2. saving the output log ----> python -m memory_profiler test.py > mem_res_1.txt

and then import the saved file again to the python code by tamppa:

from tamppa import mem_parse
mem_parse("mem_res_1.txt")

note: saving the file could be achieved by line_profiler python module lp.dump_stats("mem_res_1.txt"), but it is incompatible with tamppa and …, and could not be readable by them. I would appreciate any useful recommends to use this module directly (it is not the answer of the issue).

After the code execution and for running one of the aforementioned commands, C:\Users\Ali\AppData\Local\Temp\test.py> address is written as default and must be changed to the working directory (which contains test.py) i.e. here C:\Users\Ali\Desktop>. I think this address could be achieved by os.getcwd(), but I don't know how to

1. going to the desired address directory, and then
2. applying the command

from within python code, to avoid doing the job manually. I have tried to do so using some codes like:

os.system("python -m memory_profiler " + os.getcwd() + r"\test.py" + " > mem_res_1.txt")

which will shows:

The process cannot access the file because it is being used by another process.

It is good to say that if I use C:\Users\Ali\Desktop directly, in one step, in the terminal command as python -m memory_profiler C:\Users\Ali\Desktop\test.py > mem_res_1.txt, it will save that file in the ex temp folder.
I know these codes are half-baked and wrong here, but I guess, perhaps, they must be used for this question in a correct way. I would be appreciated for any help to run these terminal command codes from within the python code execution, if it is possible of course.

One line code is more interested.

Answer 1

Don't worry about all these fancy python tools. It's all already built into bash. Install bash on Ubuntu on Windows here. And I will give you the script to run.

https://devblogs.microsoft.com/commandline/bash-on-ubuntu-on-windows-download-now-3/

#!/bin/bash
#Start time
START=$(date +%s)
$(python module.py)
#Call more stuff
END=$(date +%s)

#End time
DIFFERENCE=$(( END - START ))
echo 
echo Time Elapsed: $DIFFERENCE seconds.

Answer 2

note: saving the file could be achieved by line_profiler python module lp.dump_stats("mem_res_1.txt"), but it is incompatible with tamppa and …, and could not be readable by them.

A solution for using lp.dump_stats output together with tamppa is presented in a related line_profiler issue.

For the aforementioned terminal default address problem, perhaps it could be handled by a symbolic link in that directory pointing to the containing folder, which is mentioned in one of PyCharm issues. There are some SO links that have valuable answers help to change directory from within the code body. With inspiration from them and relating to the aforementioned libraries, the following subprocess codes can run the needed terminal commands from within the code body (applicable):

subprocess.getoutput("python -m memory_profiler test.py > mem_res.txt")  # tamppa
subprocess.call("python -m scalene --on test.py", stderr=subprocess.PIPE, shell=True)  # scalene

In my tests, these codes detect the working directory, automatically, but could use cwd=os.getcwd() subprocess method to ensure returning current working directory of the process, if any problem.