0

I want to loop through 40 netCDF files. There are 20 files with the variable PRECC and 20 files with the variable PRECL("modelmember001.PRECC.192001-200512.nc", "modelmember002.PRECC.192001-200512.nc", ... ,"modelmember020.PRECC.192001-200512.nc" and for PRECL respectively).

I need to perform multiple cdo (climate data operator) commands with the loop (add PRECC and PRECL files, and change time series form 1920-2005 to 1955-2005).

This is the code that I use:

  datadir="path_to_mydatat"
    workdir="path_to_folder_for_newfiles"
    members="{001 .. 020}"

    for model in $members
    do

    echo 'working with model' ${model}

    echo cdo -s add ${datadir}/modelmember${members}.PRECC.192001-200512.nc${datadir}/modelmember${members}.PRECL.192001-200512.nc ${workdir}/modelmember${members}PRECT.192001-200512.nc
    # echo cdo -s selyear,1955/2005 ${workdir}/modelmember${members}.PRECT.192001-200512.nc ${workdir}/modelmember${members}.PRECT.195501-200512.nc

Eventually I need 20 files with the name 

"modelmember001.PRECT.195501-200512.nc", "modelmember002.PRECT.195501-200512.nc", ... , "modelmember020.PRECT.195501-200512.nc"

This is what I get when I run my code (deliberately with an "echo" in front of the cdo line):

$./cdo_add.sh

{001 .. 020}

working with model {001

cdo -s add /path_to_mydatat/modelmember{001 .. 020}.PRECC.192001-200512.nc         /path_to_mydatat/modelmember{001 .. 020}.PRECL.192001-200512.nc  /path_to_folder_for_newfiles/modelmember{001 .. 020}.PRECT.192001-200512.nc

working with model ..

cdo -s add /path_to_mydatat/modelmember{001 .. 020}.PRECC.192001-200512.nc /path_to_mydatat/modelmember{001 .. 020}.PRECL.192001-200512.nc /path_to_folder_for_newfiles/modelmember{001 .. 020}.PRECT.192001-200512.nc

working with model 020}

cdo -s add /path_to_mydatat/modelmember{001 .. 020}.PRECC.192001-200512.nc /path_to_mydatat/modelmember{001 .. 020}.PRECL.192001-200512.nc /path_to_folder_for_newfiles/modelmember{001 .. 020}.PRECT.192001-200512.nc

My code doesn't seem to loop through the members. There is something wrong with the way I use the placeholder "members" but I can't figure out how to fix it. Does anyone have a suggestion? Cheers!

ClimateUnboxed
  • 7,106
  • 3
  • 41
  • 86
Angie
  • 21
  • 1
  • 4

1 Answers1

2

Your code does not seem to loop because you cannot assign a brace expansion to a variable and expect it to expand when substituted in a for loop. The following saves the literal string "{001 .. 020}" to the variable members, e.g.

members="{001 .. 020}"

When you use members in for model in $members, normal word-splitting occurs because it is just a string and you loop once with 001, then with .. and finally with 020 -- not the expected sequence from 001, 002, 003, ... 020. (there should be no spaces between the number and .. to begin with -- but that still doesn't allow you to use the expansion in a variable)

To properly use the expansion, get rid of the members variable altogether and use {001..020} in the loop, e.g.

for model in {001..020}   ## (notice NO space between 001 and ..)

example:

$ for m in {001..020}; do echo $m; done
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020

That will allow you to loop with your sequence in model.

From the conversation in the comments, I know understand that you have 40 files prefixed by modelmemberXXX (where XXX is 001-020) followed by .PRECC* or .PRECL* (20 files each) that you want to coordinate feeding matching pairs to a cdo command. While the preferred way would be to loop over one matching glob, e.g. for i in modelmember*.PRECC*; do, you can also use your brace expansion approach, e.g.

for i in {001..020}
do 
    a=$(echo modelmember${i}.PRECC*)
    b=$(echo modelmember${i}.PRECL*)
    if [ -e "$a" ] && [ -f "$b" ]
    then
        printf "%s\n%s\n\n" "$a" "$b"
    fi
done

(note the [ -e "$a" ] && [ -f "$b" ] test just makes sure both files in the pair exist before proceeding with the command (printf here))

Example Output

modelmember001.PRECC.192001-200512.nc
modelmember001.PRECL.192001-200512.nc

modelmember002.PRECC.192001-200512.nc
modelmember002.PRECL.192001-200512.nc

modelmember003.PRECC.192001-200512.nc
modelmember003.PRECL.192001-200512.nc
...
modelmember020.PRECC.192001-200512.nc
modelmember020.PRECL.192001-200512.nc

You simply need to make use of $a and $b with whatever cdo_cmd you need within the loop. (as noted in the comments, you need to change to the directory containing the files, or precede the filenames with path/to/the/files)

Preferred Way

Rather than using your brace expansion, it is probably preferred to loop over one set (either PRECC or PRECL), validate the other exists, then execute the command, e.g.

for i in modelmember*.PRECC* 
do
    b="${i/PRECC/PRECL}"
    if [ -e "$i" ] && [ -f "$b" ]
    then
        printf "%s\n%s\n\n" "$i" "$b"
    fi
done

(same output)

David C. Rankin
  • 81,885
  • 6
  • 58
  • 85
  • Thank you for your suggestion! This solves the problem of the looping. If I get rid of the variable "members" though, how do I tell the loop to perform the cdo functions pairwise (e.g. `modelmember001PRECC.192001-200512.nc` and `modelmember001PRECL.192001-200512.nc` ? – Angie Dec 06 '17 at 03:55
  • Does the variable `PRECL` really hold comma separated values? e.g. `PRECL("modelmember001.PRECC.192001-200512.nc", "modelmember002.PRECC.192001-200512.nc",...`?? If it were me, I would read one variable into an array (e.g. `IFS=$', \t\n'; arrayCL=(echo $PRECL)`, then loop over each file in that array `for i in ${arrayCL[@]}; do somecdocmd "$i" ${i/PRECC/PRECL}"; done` where the *parameter expansion with substring replacement* changes `PRECC` to `PRECL` to give you the corresponding pair file. (you can add a `test -e ${i/PRECC/PRECL}"` to make sure the paired file exists.) – David C. Rankin Dec 06 '17 at 04:08
  • If `PRECL(...)` is already an array, then just loop over the indexes in each array (assuming they have the same number of elements) `for ((i = 0; i < ${#PRECL[@]}; i++)); do somecdocmd "${PRECL[I]}" "${PRECC[I]}"; done`. You can test that the elements are pairs with the parameter expansion in the previous comment. – David C. Rankin Dec 06 '17 at 04:13
  • Edit -- in the first comment it should be `arrayCL=($(echo $PRECL))` - I forgot the command substitution. You can use a *herestring* as well in bash `$(<<<$PRECL)`. (you can use a similar expansion to solve your rename to `CT` as well, e.g. `${i/PRECC/PRECT}` to replace the `CC` to `CT` (or `CL` to `CT`)) – David C. Rankin Dec 06 '17 at 04:16
  • Grrr this is why you do not use ALLCAPS for variable names. It should be `do somecdocmd "${PRECL[i]}" "${PRECC[i]}"` in the 2nd comment -- caps lock was still on `:(` – David C. Rankin Dec 06 '17 at 04:22
  • I have 40 individual files in netCDF format. Each file consists of multiple variables e.g. the variable `PRECC` (or `PRECL` ) itself is 3 dimensional (time, lon, lat). So no comma separated files nor arrays. Just a folder with 40 files (20 have `PRECL` and `PRECC` in their name respectively). I know that there is a way to do it via the "member" option, I have done it before, just can't figure out how exactly. Is there no way to define the members="001..020" in a way so it uses one argument after the other in the file name? – Angie Dec 06 '17 at 05:14
  • @DavidCRankin : do you think your approach would still work in my case (no array, nor comma separated values)? – Angie Dec 06 '17 at 05:19
  • Or sure, in the directory (or appending a path to the variables). `for model in {001..020}; do [ -e "PRECL*$model*" ] && [ -e "PRECC*$model*" ] && somecdocmd "PRECL*$model*"; done` would work by verifying both `XXX` (e.g. `001`) files exist and then running `somecdocmd` with each as arguments. You must not have 3 files with the same `XXX` in the name. (you can narrow the globbing further, e.g. `"PRECC*$model*.nc"` or adding any other characteristics you like) – David C. Rankin Dec 06 '17 at 05:22
  • @Angie - look at the addition to the answer. I believe I understand your file pair needs. Also note the comment about the *preferred way* to loop over the set of files. – David C. Rankin Dec 06 '17 at 06:05
  • Glad to help. You will be a bash wizard soon. Note, the `-e` (exists) should probably be `-f` (file) in both `[ ... ]` tests for consistency sake. Either `-e` exists, `-f` file or `-r` readable, will work, but `-f` will distinguish a file from a directory in case you happened to have a stray directory with the expected filename. Good luck and always use [Bash Guide](http://mywiki.wooledge.org/BashGuide) and [Bash FAQ](http://mywiki.wooledge.org/BashFAQ) for good references and [shellcheck.net](https://www.shellcheck.net/) for good checking. – David C. Rankin Dec 07 '17 at 05:41