diff two arrays each containing files paths into a third array (for removal)

Question

In the function below you will see notes on several attempts to solve this problem; each attempt has a note indicating what went wrong. Between my attempts there is a line from another question here which purports to solve some element of the matter. Again, I've added a note indicating what that is supposed to solve. My brain is mush at this point. What is the stupid simple thing I've overlooking?

function func_removeDestinationOrphans() { 
    readarray -d '' A_Destination_orphans < <( find "${directory_PMPRoot_destination}" -type f -print0 ) 
    for (( i = 0 ; i < ${#A_Destination_orphans[@]} ; i++ )) ; do 
        printf '%s\n' "→    ${A_Destination_orphans[${i}]}" # path to each track 
    done 
    printf '%b\n' "" 
    # https://stackoverflow.com/questions/2312762/compare-difference-of-two-arrays-in-bash 
    # echo ${Array1[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u ## original 
    # Array3=(`echo ${Array1[@]} ${Array2[@]} | tr ' ' '\n' | sort | uniq -u `) ## store in array 
    # A_Destination_orphans_diff=(`echo "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | tr ' ' '\n' | sort | uniq -u `) # drops file path after space 
    # printf "%s\0" "${Array1[@]}" "${Array2[@]}" | sort -z | uniq -zu ## newlines and white spaces 
    # A_Destination_orphans_diff=($( printf "%s\0" "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | sort -z | uniq -zu )) # throws warning and breaks at space but not newline 
    # printf '%s\n' "${Array1[@]}" "${Array2[@]}" | sort | uniq -u ## manage spaces 
    # A_Destination_orphans_diff=($( printf '%s\n' "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | sort | uniq -u )) # breaks at space and newline 
    # A_Destination_orphans_diff="($( printf '%s\n' "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | sort | uniq -u ))" # creates string surrounded by () 
    # A_Destination_orphans_diff=("$( printf '%s\n' "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | sort | uniq -u )") # creates string 
    # A_Destination_orphans_diff=($( printf '%s\n' ${A_Destination_dubUnders[@]} ${A_Destination_orphans[@]} | sort | uniq -u )) # drops file path after space 
    for (( i = 0 ; i < ${#A_Destination_orphans_diff[@]} ; i++ )) ; do 
        printf '%s\n' "→    ${A_Destination_orphans_diff[${i}]}" # path to each track 
    done 
    printf '%b\n' "" 
    for (( i = 0 ; i < ${#A_Destination_orphans_diff[@]} ; i++ )) ; do 
        echo # rm "${A_Destination_orphans_diff[i]}" 
    done 
    func_EnterToContinue 
}

score 1 · Answer 1 · edited Aug 08 '20 at 13:56

1

This throws warning and breaks at space but not newline because you build the array with direct assignment of syntax construct. When an entry contains spaces, it also splits break to a new entry.

A_Destination_orphans_diff=($( printf "%s\0" "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | sort -z | uniq -zu ))

To avoid the issue of the method above, you can mapfile/readarray a null delimited entries stream.

mapfile -t -d '' A_Destination_orphans_diff < <(
  printf "%s\0" "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" |
    sort -z |
      uniq -zu
)

In case your shell version is too old to support mapfile you can perform the same task with IFS=$'\37' read -r -d '' -a array.

$'\37' is shell's C-Style string syntax with octal code 37, which is ASCII 31 US for Unit Separator:

IFS=$'\37' read -r -d '' -a A_Destination_orphans_diff < <(
  printf "%s\0" "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" |
    sort -z |
      uniq -zu |
        xargs -0 printf '%s\37'
)

edited Aug 08 '20 at 13:56

oguz ismail

1
16
47
69

answered Aug 08 '20 at 11:22

Léa Gris

17,497
4
32
41

That seems legit. I've used mapfile/readarray like this elsewhere for different purposes, and the output here seems to check correctly. You'd say that those contributors in that thread are simply mistaken and that redirection is required? They seemed so sure of themselves. – JamesIsIn Aug 08 '20 at 16:44
Damn it. The machine is use for the sync is running an older Ubu with bash 4.3.xx. – JamesIsIn Aug 08 '20 at 17:53
Sorry, that \37 version doesn't seem to be working for me. I have added a second version (now current) of my function above. – JamesIsIn Aug 08 '20 at 19:56

score 1 · Answer 2 · answered Aug 08 '20 at 22:54

To remove all files not present in A_Destination_dubUnders array you could:

func_removeDestinationOrphans() { 
    find "${directory_PMPRoot_destination}" -type f -print0 |
    sort -z |
    join -z -v1 -t '' - <(printf "%s\0" "${A_Destination_dubUnders[@]}" | sort -z) |
    xargs -0 echo rm
}

Use join or comm to find elements not present in one list and present in another list. I am usually wrong about -v1, so try with -v2 if it echoes the elements from wrong list (I do not understand if you want to remove files present in A_Destination_dubUnders list or not present, you did not specify that).

Note that function name() is a mix of ksh and posix function definition. Just name() {. See bash hackers wiki obsolete

score 0 · Accepted Answer · answered Oct 15 '20 at 19:16

Here is the working version with modifications thanks to suggested input from the first two respondents (thanks!).

function func_removeDestinationOrphans() { 
    printf '%s\n' " →   Purge playlist orphans:  " "" 
    printf '%b\n' "First we will remove any files not present in your proposed playlist.  " 
    func_EnterToContinue 
    bash_version="$( bash --version | head -n1 | cut -d " " -f4 | cut -d "(" -f1 )" 
    if printf '%s\n' "4.4.0" "${bash_version}" | sort -V -C ; then 
        readarray -d '' A_Destination_orphans < <( find "${directory_PMPRoot_destination}" -type f -print0 ) # readarray or mapfile -d fails before bash 4.4.0 
        readarray -t -d '' A_Destination_orphans_diff < <( 
            printf "%s\0" "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | 
            sort -z | 
            uniq -zu 
        ) 
    else 
        while IFS=  read -r -d $'\0'; do 
            A_Destination_orphans+=( "$REPLY" ) 
        done < <( find "${directory_PMPRoot_destination}" -type f -print0 ) 
        IFS=$'\37' read -r -d '' -a A_Destination_orphans_diff < <( 
        printf "%s\0" "${A_Destination_dubUnders[@]}" "${A_Destination_dubUnders[@]}" "${A_Destination_orphans[@]}" | 
            sort -z | 
            uniq -zu | 
            xargs -0 printf '%s\37' 
        ) 
    fi 
    if [[ ! "${A_Destination_orphans_diff[*]}" = '' ]] ; then 
        for (( i = 0 ; i < ${#A_Destination_orphans_diff[@]} ; i++ )) ; do 
            rm "${A_Destination_orphans_diff[i]}" 
        done 
    fi 
}

If you would like to see the entire Personal Music Player sync script, you can find that via my GitHub.

diff two arrays each containing files paths into a third array (for removal)

3 Answers3

Linked