Use A Threading Wrapper Script
A treaded bash or python wrapper script may be all you need. A script that will split the work up and call it for you automatically. The bonus to this is that you would not have to rewrite too much to get it to work. The hypothetical below would potentially reduce runtime from 111 hours to 1.1 hours.
Say your current solution is this:
file_of_200k_uids.txt
ruby ruby_script.rb "file_of_200k_uids.txt"
So the ruby_script.rb runs through all 200K UIDs and performs the network task which takes say 2sec per equates to 400,000 seconds.
Proposed solution (write a wrapper thread with BASH4+):
file_of_200k_uids.txt
ruby ruby_script.rb "file_of_200k_uids.txt"
bash_thread_manager.sh
Contents of bash_thread_manager.sh would be something like this :
# -- Step one have the bash script break down the large file --
# and place the results in a /path/to/folder
cp file_of_200k_uids.txt /path/to/folder/temp_file_of_200k_uids.txt
split -d -b 10M file_of_200k_uids.txt uids_list
rm /path/to/folder/temp_file_of_200k_uids.txt
# -- Now run through the folders and launch the script you need to do the work --
# -- it will create instances of your script up to a max number (e.g. 100) --
child="$$"
for filename in /path/to/folder/*; do
num_children=$(ps --no-headers -o pid --ppid=$child | wc -w)
let num_children=num_children-1
if [[ $num_children -gt 100 ]] ; then
sleep 60
else
ruby ruby_script.rb "$filename" > /output/result-${RANDOM}.txt &
fi
done
wait
# -- final step would be a for loop that combines all of the files
cat /output/result-*.txt >> all.txt
The bash script will manage calling the UIDs from a file and collect the data as separate threads up to some number you define. In the example below we split temp_file_of_200k_uids.txt into smaller 10MB files max and then call 100 of these 10MB files at once using the bash script. Any time it drops below 100 threads it increases it back up to 100. Now you can do it 100x faster and so forth.
Further reading:
https://linoxide.com/linux-how-to/split-large-text-file-smaller-files-linux/
Multithreading in Bash
