Unix for loop skipping one value in script, works properly in command line

Question

I am running the following commands in command line:

for DATAFILE in `find dir_name -type f -mtime +10 | egrep -v -e 'archive/'`
do
    echo 'Data file name- ' "$DATAFILE"
    echo 'Base name ' 
    BASENAME=`basename "${DATAFILE}"`
    DESTFILE="${BASENAME}"_`date +"%Y%m%d%H%M%S"`
    echo "Dest file - "$DESTFILE
done

I get the following result for this:

Data file name-  DIR_PATH_1/file_1.txt
Base name
Dest file - file_1.txt_20120719041239
Data file name-  DIR_PATH_2/file_2.txt
Base name
Dest file - file_2.txt_20120719041239

When I put the same commands in a shell script and execute, I get the following result:

Data file name-  DIR_PATH_1/file_1.txt
DIR_PATH_2/file_2.txt
Base name
Dest file - file_2.txt_20120719040956

I have checked the script for Control-M and other junk characters. Also, I don't have any extra steps in the shell script (no parameters and all).

Can someone point me in the right direction.

Thanks.

Update 1:

I made the following change to the loop:

Earlier:

for DATAFILE in `find ${ROOT_DIR} -type f -mtime +$DAYS_ARCH | 
                     egrep -v -e 'archive/'`

Now:

find ${ROOT_DIR} -type f -mtime +$DAYS_ARCH |
     egrep -v -e 'archive/' | while read DATAFILE

It seems to be working properly now. I am still testing to confirm this.

Update 2:

Changing from FOR to WHILE loop has fixed the issue. But still I am not able to understand why this is happening. Anyone?

To add, the users running the shell script and from the command line are different. But I have given '777' to all files and directories. — visakh, Jul 19 '12 at 08:18
Your for loop appears to be treating the whole output of `find` as a single field. What else do you have in your script? Has the `IFS` var been modified? — Shawn Chin, Jul 19 '12 at 08:25
What shell is being used for the script? It looks like some feature has been turned on in the script that isn't turned on at the CLI. I regularly get these issues as I use zsh as by day-to-day shell, but write most scripts in bash as it's more likely to be installed — Anya Shenanigans, Jul 19 '12 at 08:29
As @Petesh says: I'd start by putting `#!/bin/sh` on top of the script. (**must** be the first line!) — wildplasser, Jul 19 '12 at 08:45
Sorry for the delay in replying. I am using ksh. I am putting #!bin/ksh on top of the script. — visakh, Jul 19 '12 at 09:25
Please see my update. Can someone tell me why it works with WHILE and not with FOR? — visakh, Jul 19 '12 at 09:43
Likely, because you have file names with special characters or spaces. — fork0, Jul 19 '12 at 12:03
http://stackoverflow.com/questions/10748703/iterate-over-lines-instead-of-words-in-a-for-loop-of-shell-script/10752407#10752407 — Lesmana, Jul 24 '12 at 13:14

score 1 · Answer 1 · answered Aug 09 '12 at 07:40

Add to your script:

echo "My shell is: " $0
echo "My IFS is:" $IFS

and compare it with results from interactive shell.

Make sure your script is executed by script you want, by adding a hashbang line.

According to man sh, IFS is defined as:

Input Field Separators. This is normally set to ⟨space⟩, ⟨tab⟩, and ⟨newline⟩. See the White Space Splitting section for more details.

score 1 · Accepted Answer · answered Aug 10 '12 at 00:04

Capturing find output in backticks will effectively join all lines. This will break file names with embedded spaces.

for DATAFILE in find ${ROOT_DIR} -type f -mtime +$DAYS_ARCH | egrep -v -e 'archive/'

The while read ... loop will read exactly one pathname at a time, even if they contain white space or other special characters.

Unix for loop skipping one value in script, works properly in command line

2 Answers2