How to randomly distribute the files across 3 folders using Bash script?

Question

I have many subdirectories and files in the folder mydata/files. I want to take files and copy them randomly into 3 folders:

train
test
dev

For example, mydata/files/ss/file1.wav could be copied into train folder:

train
  file1.wav

And so on and so forth, until all files from mydata/files are copied.

How can I do it using Bash script?

@0stone0 I implemented it in Python, but cannot do it in Bash because my knowledge of Bash is very limited. — Fluxy, Dec 29 '20 at 15:51
Don't reinvent [sklearn](https://stackoverflow.com/a/39210813/10248678) wheel. — oguz ismail, Dec 29 '20 at 16:47

score 1 · Accepted Answer · answered Dec 29 '20 at 16:31

Steps to solve this:

Need to gather all the files in the directory
Assign directories to a map
Generate random number for each file
Move the file to the corresponding directory

The script:

#!/bin/bash

original_dir=test/

## define 3 directories to copy into
# define an associative array (like a map)
declare -A target_dirs

target_dirs[0]="/path/to/train/"
target_dirs[1]="/path/to/test/"
target_dirs[2]="/path/to/dev/"

# recursively find all the files, and loop through them
find $original_dir -type f | while read -r file ; do
        # find a random number 0 - (size of target_dirs - 1)
        num=$(($RANDOM % ${#target_dirs[@]}))
        # get that index in the associative array
        target_dir=${target_dirs[$num]}
        # copy the file to that directory
        echo "Copying $file to $target_dir"
        cp $file $target_dir
done

Things you'll need to change:

Change the destination of the directories to match the path in your system
Add executable priviledges to the file so that you can run it.

chmod 744 copy_script_name
./copy_script_name

Notes:

This script should easily be extendable to any number of directories if needed (just add the new directories, and the script will adjust the random numbers.

If you need to only get the files in the current directory (not recursively), you can add -maxdepth 1 (see How to list only files and not directories of a directory Bash?).

Was able to leverage previous bash experience plus looking at bash documentation (it's generally pretty good). If you end up writing any scripts, be very careful about spaces

Dr Claw · Answer 2 · 2020-12-29T20:41:29.403

0

You can create a temp file, echo your destination folder to it, then use the shuf command.

dest=$(mktemp)
echo -e "test\ndev\ntrain" >> $dest
while IFS= read -r file; do
  mv "$file" "$(shuf -n1 < $dest)/."
done < <(find mydata/files -type f 2>/dev/null)
rm -f "$dest"

edited Dec 29 '20 at 20:41

answered Dec 29 '20 at 16:57

Dr Claw

636
4
15

How to randomly distribute the files across 3 folders using Bash script?

2 Answers2