1

I have a Python script that is being executed within the anaconda prompt in a Windows environment. As this script analyses one variable at a time, the command line loops over the files in a folder, and for every file it executes the script once. It is done in this way, as looping within a script eventually results in a memory error. However, running the script for files one by one is too slow, and I could for sure run the script for five files at once, speeding up the proces five times.

I know that there are modules as subprocess and multiprocessing, but I have a feeling there might be a more simplistic way of doing it. Unfortunately I was not able to find one.

The code I am running at this moment is as follows

for /F %i in ('dir "Directory_containing_files" /b /s') do (python Executed_script.py -i %i -o "Folder_to_write_output_to")

I hope there is some piece of code that could look something like this:

for /F %i in ('dir "Directory_containing_files" /b /s') do (python Executed_script.py -i %i -o "Folder_to_write_output_to") -n 10

With the desired output that it runs 10 files in parallel.

Rivered
  • 741
  • 7
  • 27
  • Have you looked at Queues? https://docs.python.org/3/library/asyncio-queue.html – Jeff Aug 23 '19 at 17:07
  • So you want parallelism without using the python libraries to make it, but instead through the anaconda prompt? Does this help https://stackoverflow.com/questions/11010834/how-to-run-multiple-dos-commands-in-parallel – Chrismon Chin Aug 23 '19 at 17:10
  • Possible duplicate: [How do you run multiple programs in parallel from a bash script?](https://stackoverflow.com/q/3004811/570918) Not sure if that works in Anaconda Prompt, though. – merv Aug 23 '19 at 19:41
  • 1
    take a look at the [start](https://ss64.com/nt/start.html) command. – Stephan Aug 24 '19 at 08:10
  • I was not able to find anything useful in these commands. I solved it with a less elegant solution of running multiple command prompts, and dividing my initial files in multiple folders. – Rivered Aug 24 '19 at 18:53

0 Answers0