Find specific file extensions in recursive structure and extract them until no left

Question

I have a recursive multiple folder structures. Inside of there are multiple zip, img files. But when I extracted it, new zip,img files come out again. I want to write a loop for this.

Scan whole recursive folders, find multiple zip files and extract them. After extraction scan folders again if system found new zip files, extract it again and again continue until no left zip file in the folders.

sample.tar.gz
       1.img
           Thefolder
                 samplefolder
                         t.img
       2.img
           samplefolder2
                 different.gz
       3.img
       4.img

There are more folders also under other img files.

I tried to write for loop for this but couldn't work it. After extraction 1.img script getting error.

Error Output:

ID = 277952352

Everything is Ok

Folders: 3
Files: 1
Size:       345424152
Compressed: 671024244
+ rm -f /home/sample/1.img
+ recursiveExtract /home/sample/Thefolder
+ for file in "$path"/*
+ '[' -d /home/sample/Thefolder ']'
+ recursiveExtract /home/sample/Thefolder
Segmentation fault

The code:

#!/bin/bash
set -x


target_file=$1
path="$(realpath "$1")"
recursiveExtract () { # $1=directory
      for file in "$path"/*; do
        if [ -d "$file" ]; then
            recursiveExtract "$file"
        elif [ -f "$file" -a "${file##*.}" = 'img' ]; then
            7z x $file -o$path -r # variation 2
            rm -f "$file" # comment this if you want to keep the zip files.
            recursiveExtract "${file%.img}"
        fi
        b=$(ls $path | grep img | wc -l)
        if [[ "$b" -eq 0 ]]; then
            break
        fi
      done
      for file in "$path"/*; do
        if [ -d "$file" ]; then
            recursiveExtract "$file"
        elif [ -f "$file" -a "${file##*.}" = 'tar' ]; then
            7z x $file -o$path -r # variation 2
            rm -f "$file" # comment this if you want to keep the zip files.
            recursiveExtract "${file%.tar}"
        fi
        c=$(ls $path | grep tar | wc -l)
        if [[ "$c" -eq 0 ]]; then
            break
        fi
      done
      for file in "$path"/*; do
        if [ -d "$file" ]; then
            recursiveExtract "$file"
        elif [ -f "$file" -a "${file##*.}" = 'gz' ]; then
            7z x $file -o$path -r # variation 2
            rm -f "$file" # comment this if you want to keep the zip files.
            recursiveExtract "${file%.gz}"
        fi
        d=$(ls $path | grep gz | wc -l)
        if [[ "$d" -eq 0 ]]; then
            break
        fi
      done
}
recursiveExtract "$1"

Latest version of my code:

#!/bin/bash
# set -x

recursiveExtract () { # $1=directory
      path="$(realpath "$1")"
      echo "GZ------------------"
      for file in "$path"/*; do
        if [ -f "$file" -a "${file##*.}" = 'gz' ]; then
            echo "File:" $file
            echo "Path:" $path
            # 7z e "${file%.gz}" -o"$file" # variation 1
            7z x $file -o$path -r -aou # variation 2
            rm -f "$file" # comment this if you want to keep the zip files.
            recursiveExtract "${file%.gz}"
        fi
        # d=$(ls $path | grep gz | wc -l)
        d=$(find $path -type f -name "*.gz" | wc -l)
        echo "WC GZ-----------------:" $d
        if [[ "$d" -eq 0 ]]; then
            break
        fi
      done
      echo "IMG------------------"
      for file in "$path"/*; do
        if [ -f "$file" -a "${file##*.}" = 'img' ]; then
            echo "File:" $file
            echo "Path:" $path
            # 7z e "${file%.img}" -o"$file" # variation 1
            7z x $file -o$path -r -aou # variation 2
            rm -f "$file" # comment this if you want to keep the zip files.
            recursiveExtract "${file%.img}"
        fi
        # b=$(ls $path | grep img | wc -l)
        b=$(find $path -type f -name "*.img" | wc -l)
        echo "WC IMG-----------------:" $b
        if [[ "$b" -eq 0 ]]; then
            break
        fi
      done
}
while true
do
    d=$(find $1 -type f -name "*.gz" | wc -l)
    b=$(find $1 -type f -name "*.img" | wc -l)
    if [[ "$d" -eq 0 ]] && [[ "$b" -eq 0 ]]; then
        break
    else
        recursiveExtract "$1"
    fi
done

`unzip` doesn't remove the input file so there is no way this could ever finish. You need to somehow keep track of which files you have already processed. A recursive function seems like a very iffy way to approach this. I would extract each zip into a temporary directory and check whether the extracted tree contains any new zip files. — tripleee, May 20 '21 at 08:40
Hmm.. pretty similar to this [question](https://stackoverflow.com/questions/65889191/how-to-unzip-a-zip-file-inside-another-zip-file/65948329#65948329). — Darkman, May 20 '21 at 11:25
@Darkman I tried your solution and improve actually its worked. But one question how can I extract recursive zip file? For example I have got sample.tar.gz when I extracted this. I get 3 different files. 1.img, 2.img and 3.img. For loop starting to extract 1.img after extraction, system found more multiple folders inside of this file. Script completed extraction for 1.img but after that script get error before to start rest of them. I will update my question you can see more details. — hammer89, May 25 '21 at 14:36

score 0 · Answer 1 · answered May 21 '21 at 16:32

This shoud do the job

#!/bin/bash
find ./ -name "*.zip*" > current_zips.txt
while [[ `wc -l "current_zips.txt" | cut -d' ' -f1` > 0 ]]; do
    find ./ -name "*.zip*" -exec unzip {} \;
    while IFS= read file_; do rm $file_; done < "current_zips.txt" # for file_ in $(cat current_zips.txt);do rm $file_;done
    find ./ -name "*.zip*" > current_zips.txt
done

Use the following script to generate test data

#!/bin/bash
#create some data
echo "data 0" > file0.txt
mkdir folder1 folder2
echo "data 1" > folder1/file1.txt
echo "data 2" > folder2/file1.txt
#zip data
zip file0.zip file0.txt
zip -r folder1.zip folder1
zip -r folder2.zip folder2
zip -r data.zip *.zip
#delete original data
rm -rf file* folder*

Darkman · Answer 2 · 2021-05-27T08:31:44.170

I think you want something like this:

recursiveExtract.bash

#!/bin/bash

# efi,gz,img,tar

recursiveExtract () { # $1=file/directory
    local path="$(realpath "$1")" # This variable must be local.
    for file in "$path"/*; do
        if [ -d "$file" ]; then
            recursiveExtract "$file"
        elif [ -f "$file" -a "${file##*.}" = 'tgz' ]; then # file.tgz == file.tar.gz
            local anotherFile="${file%.*}"'.tar'
            echo 'Extracting: '"$file"
            7z x "$file" -o"${file%/*}" -aoa -bd -bso0
            7z x "$anotherFile" -o"${anotherFile%.*}" -aoa -bd -bso0
            #######################
            # -aoa   == Overwrite output files       #
            # -bd     == Don't show progress bar  #
            # -bso0 == Silent                                    #
            #######################
            rm -f "$anotherFile" # DO NOT COMMENT THIS LINE.
            rm -f "$file" # Comment this if you want to keep the archieves.
            recursiveExtract "${anotherFile%.*}"
        elif [ -f "$file" -a "${file##*.}" = 'gz' ]; then
            local anotherFile="${file%.*}"
            if [ -n "$anotherFile" -a \( "${anotherFile##*.}" = 'tar' -o "${anotherFile##*.}" = 'img' \) ]; then # file.tar.gz, file.img.gz
                echo 'Extracting: '"$file"
                7z x "$file" -o"${file%/*}" -aoa -bd -bso0
                7z x "$anotherFile" -o"${anotherFile%.*}" -aoa -bd -bso0
                rm -f "$anotherFile" # DO NOT COMMENT THIS LINE.
                rm -f "$file" # Comment this if you want to keep the archieves.
                recursiveExtract "${anotherFile%.*}"
            else # gz
                echo 'Extracting: '"$file"
                7z x "$file" -o"${file%.*}" -aoa -bd -bso0
                rm -f "$file" # Comment this if you want to keep the archieves.
                recursiveExtract "{file%.*}"
            fi
        elif [ -f "$file" -a \( "${file##*.}" = 'img' -o "${file##*.}" = 'tar' \) ]; then # file.img or file.tar
            echo 'Extracting: '"$file"
            7z x "$file" -o"${file%.*}" -aoa -bd -bso0
            rm -f "$file" # Comment this if you want to keep the archieves.
            recursiveExtract "${file%.*}"
        fi
    done
}
recursiveExtract "$1"

Give the script executable permission with chmod +x recursiveExtract.bash then you can run it like:

$ ./recursiveExtract.bash <directory/file>

In your case, maybe like this:

$ ./recursiveExtract.bash 'sample.img.gz'

My main artifact is sample.img.gz. In my case,I try to design generic extraction for some specific extension type. (tar, img, gz,efi) . Next time my main artifact can be tar.gz or img.gz when I extract it. So there is no sequential order. So I need to check all extension types in same time. Your code is really good but its working for gz or do this situation. Maybe next time I will face it with tar>gz>img>efi>img or img>gz>efi>img. I need to find a way for every extraction to check which type of extension came from before extract and extract it again until no left these type of files. — hammer89, May 26 '21 at 06:41
I mean I need a generic script because I don't know each extraction which type of extension will come out from main artifact. But I only know which type extension we have. Each extraction I need to check img, efi, tar, gz in every folder level until no left. — hammer89, May 26 '21 at 07:23
Updated my code, please review it from my question "Latest version of my code" part. I tried for only gz and img extensions if its work, I will add more extension conditions on it. — hammer89, May 26 '21 at 13:54
How do you extract **uefi** files? using 7z? About your script - Well, a lot of unnecessary ***for*** loops. Be careful with your loops especially within recursions. And you really don't need more than 1 loop. If you have another ***ext***, just copy-paste the last `elif` and change the `img` to whatever ***ext*** you might deal with, hence making another `elif`. — Darkman, May 27 '21 at 08:12
By the way, how do extract an `img` file using `7z` in the first place? I got an error like `Open ERROR: Can not open the file as [ELF] archive`. Please look at this [How would i extract a img file?](https://askubuntu.com/questions/986684/how-would-i-extract-a-img-file). — Darkman, May 27 '21 at 09:12
Like this `7z x $file -o$path -r` or `7z x $file -o$path -r -aou` Its also working for img files. — hammer89, May 27 '21 at 09:45

Find specific file extensions in recursive structure and extract them until no left

2 Answers2