Copy specific number of files from on directory to another using Linux

Question

I have a directory containing 8000+ fits files and I am wondering if there is a way to copy increments of them unto other directories such that I could have either 8 directories with about 1000 fits files in them each or 4 directories with 2000 fits files?

Is there any pattern in those file names to split them? Otherwise, you can loop through the list of files and move to `dir_$i`, being `$i` a var you increment when crossing a 1000 threshold. — fedorqui, Feb 21 '14 at 15:11
All the fits files start with the name spec. (spec-234-223-23.fits, spec-113-23-21.fits, etc). — user1821176, Feb 21 '14 at 15:16

score 4 · Accepted Answer · answered Feb 21 '14 at 15:23

4

Something like this could make it:

dir=1
counter=1

for file in spec*
do
   echo "cp $file dir_$dir"
   ((counter++))
   (( $counter%1000 == 1 )) && ((dir++))
done

Explanation

dir=1 and counter=1 are setting the variables.
for file in spec* loops through spec* pattern name files.
echo "cp $file dir_$dir" will output like cp spec123 dir_1 / dir_2, ... I used echo so that you can check the behaviour before going ahead and doing the proper cp.
((counter++)) increments the counter variable counter.
(( $counter%1000 == 1 )) && ((dir++)) if $counter is on a form 1000K+1, increment the value of $dir.

answered Feb 21 '14 at 15:23

fedorqui

275,237
103
548
598

Was I suppose to just use that code on my terminal? I did it but I cannot see the other directories that hold the files. – user1821176 Feb 21 '14 at 15:37
What do you mean? To perform this action you should have some knowledge about shell scripting. Not because it is difficult code, but because you should be able to personalize it to your own requirements. – fedorqui Feb 21 '14 at 15:39
@user1821176 note you can change `1000` for the number you like to group. Also, the dirs created will have the name `dir_1`, `dir_2`... You can change it in the `cp $file ...` part. Note also that I am `echo`ing, so you can test and see if it works. Then, remove the `echo` and just keep `cp $file dir_$dir`. To see how to execute it, check suenda's answer. – fedorqui Feb 24 '14 at 09:17

score 1 · Answer 2 · answered Feb 21 '14 at 16:02

To extend the script fedorqui wrote to your need, you can do the following:

create a file,

$ nano mycopy.sh (you can choose your own name)

paste the following and save the file:

source=$1
dir=1
counter=1

mkdir dir_$dir
for file in `ls $source`
do
   cp -r $source$file dir_$dir/
   ((counter++))
   (( $counter%1000 == 1 )) && ((dir++)) && ((`mkdir dir_$dir`))
done

make your file executable by

$ chmod u+x mycopy.sh

now execute the script by running

$ ./mycopy.sh MyDirectoryWithManyFiles/

where MyDirectoryWithManyFiles/ is the directory containing your files

The script will create sub directories and copy 1000 files max inside them

score 0 · Answer 3 · answered Feb 21 '14 at 16:07

Here's a way of doing it in Python:

Create a new file called "manage_files.py" or something, put this code into it:

# Hat-tip to http://stackoverflow.com/questions/3964681/find-all-files-in-directory-with-extension-txt-with-python
import os
import glob
from subprocess import call

# Initialise
my_dir = 'files_are_here'
num_files = 10

# Get a list of all files in my_dir
os.chdir(my_dir)
all_files = glob.glob('*')

for i in range((len(all_files) / num_files) + 1):
    files = all_files[i * num_files:(i + 1) * num_files]
    dirname = "dir_%s" % i
    call(["mkdir", dirname])
    file_names = ' '.join(files)
    arg = ['mv', '-t', '%s/' % dirname]
    arg.extend(files)
    call(arg)

Then, from the Linux shell type python manage_files.py and the magic will do its stuff.

Azam Zabihi · Answer 4 · 2022-08-19T10:08:07.593

I hope it helps you!

Here, rootfile folder contains several files, those names are included like ..om500.. or ..om501.. or ..om502..

So, I wanted to copy a specific number of each files name to a new folder inside rootfile folder, rootrate.

#!/bin/bash
mkdir rootfile/rootrate
for ((j=500;j<503;j++)); do
echo $j
file=(`ls rootfile/*om${j}_*.root`)
if [ "$j"  -eq "500" ];
then
  max=60
fi
if [ "$j"  -eq "501" ];
then
  max=8  
fi
if [ "$j"  -eq "502" ];
then
  max=120
fi

for ((i = 0; i<max ; i++)); do
  echo "${file[i]}"
  echo $i
  cp "${file[i]}" rootfile/rootrate/ 
done
done

Copy specific number of files from on directory to another using Linux

4 Answers4

Explanation