Bashshell - argument too long message shows after for loop-echo command

Question

I am working on CentOS with bash shell.

I do not see anything wrong in the code, but it keeps print arguments list too long error.

It is not only apply to aws cp, after for cat /dev/null > $FILE_DIR/dailey_member.csv also prints same message.

echo " ===== dailey vote by member"
cat /dev/null > $FILE_DIR/dailey_member.csv
QUERY_DAILEY_MEMBER_RS=`mysql -h $MYSQL_URL -u$MYSQL_USER -p$MYSQL_PW $MYSQL_SCHEMA -e "SELECT voteDate, usrId, count(usrId) FROM tbl_vote GROUP BY voteDate, usrId;"`
IFS=$'\n' LINES=($QUERY_DAILEY_MEMBER_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/dailey_member.csv
done
echo "aws s3 cp $FILE_DIR/dailey_member.csv " $S3_PATH # copy output works.
aws s3 cp "$FILE_DIR/dailey_member.csv" $S3_PATH
exit 0

How could I fix this?

Thanks.

FYI, What I am trying to do is...

do MYSQL query
parse result and write as csv file
send to s3 repository, so people can download file

============== edited #1 ================

I set my variable like on top of the file ...

export FILE_DIR="/home/jyoh/applications18/cron/file"
export S3_PATH="s3://juneyoung/applications18/voteLogs/"

The result puts on the console.

[jyoh@ip-XX-XX-X-XXX cron]$ sh ./daileyVoteByMember.sh 
 ===== dailey vote by member
aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
./daileyVoteByMember.sh: line 27: /usr/bin/aws: arguments list too long
[jyoh@ip-XX-XX-X-XXX cron]$ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
upload: file/dailey_member.csv to s3://juneyoung/applications18/voteLogs/dailey_member.csv

line27 is where aws s3 cp "$FILE_DIR/dailey_member.csv" $S3_PATH is.

============== edited #2 ================

result with option -x.

...SKIP...

+ LINE_IDX=265782
+ LINE='20180129    9>2387463256785462364   1'
+ LINE='20180129,9>2387463256785462364,1'
+ echo -e '20180129,9>2387463256785462364,1'
+ for i in '"${!LINES[@]}"'
+ LINE_IDX=265783
+ LINE='20180129    9>3658743656387567834   1'
+ LINE='20180129,9>3658743656387567834,1'
+ echo -e '20180129,9>3658743656387567834,1'
+ echo 'aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/'
+ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
./daileyVoteByMember.sh: line 27: /usr/bin/aws: arguments list too long
+ exit 0

============== edited #3 ================

Copy aws in new bash file and execute

the file contents

#! /bin/bash

aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/

exit 0

console output

[jyoh@ip-XX-XX-X-XXX cron]$ sh testSh.sh 
upload: file/dailey_member.csv to s3://juneyoung/applications18/voteLogs/

============== edited #4 ================

This works fine and clean. Maybe some problem with for and >> file redirection?

function sendS3 () {
   echo "S3 send command : aws s3 cp $1 $S3_PATH"     
   aws s3 cp $1 $S3_PATH
}

echo " ===== daily vote"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #1;" | sed $'s/\t/,/g' > "$FILE_DIR"/daily.csv
sendS3 "$FILE_DIR"/daily.csv


echo " ===== dailey vote by member"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #2;" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_member.csv
sendS3 "$FILE_DIR"/dailey_member.csv


echo " ===== daily vote by ip"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #3" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_ip.csv
sendS3 "$FILE_DIR"/daily_ip.csv


echo " ===== daily vote by member,item"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #4" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_item.csv
sendS3 "$FILE_DIR"/daily_item.csv

exit 0

============== edited #5 ================

This is the full script. I replaced url and some variables for security.

#! /bin/bash

export SLACK_URL="SLACK_API_URL"
export MYSQL_URL="MYSQL_CONNECTION_URL"
export MYSQL_USER="MYSQL_USER"
export MYSQL_PW="MYSQL_PW"
export MYSQL_SCHEMA="USER"

export FILE_DIR="FILE_PATH"

export QUERY_DAILY_RS=""
export QUERY_DAILY_MEMBER_RS=""
export QUERY_DAILY_IP_RS=""
export QUERY_DAILY_MEMBER_ITEM_RS=""
export S3_PATH="S3_URL_TO_STORE"

echo -e "Sending vote report to Slack"
echo $MYSQL_URL
echo $MYSQL_USER::$MYSQL_PW


function sendS3 () {
   echo "S3 send command : aws s3 cp $1 $S3_PATH"     
   aws s3 cp $1 $S3_PATH
}


echo " ===== daily vote"
cat /dev/null > $FILE_DIR/daily.csv
QUERY_DAILY_RS=`mysql SELECT QUERY 1;`
IFS=$'\n' LINES=($QUERY_DAILY_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> "$FILE_DIR"/daily.csv
done
sendS3 "$FILE_DIR/daily.csv"



echo " ===== daily vote by member"
cat /dev/null > $FILE_DIR/daily_member.csv
QUERY_DAILY_MEMBER_RS=`mysql SELECT QUERY 2`
IFS=$'\n' LINES=($QUERY_DAILY_MEMBER_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/daily_member.csv
done
sendS3 "$FILE_DIR/daily_member.csv"



echo " ===== daily vote by ip"
cat /dev/null > $FILE_DIR/daily_ip.csv
QUERY_DAILY_IP_RS=`mysql SELECT QUERY 3`
IFS=$'\n' LINES=($QUERY_DAILY_IP_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/daily_ip.csv
done
sendS3 "$FILE_DIR/daily_ip.csv"



echo " ===== daily vote by member,item"
cat /dev/null > $FILE_DIR/daily_member_item.csv
QUERY_DAILY_MEMBER_ITEM_RS=`mysql SELECT QUERY 4`
IFS=$'\n' LINES=($QUERY_DAILY_MEMBER_ITEM_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/daily_member_item.csv
done

sendS3 "$FILE_DIR/daily_member_item.csv"
curl -X POST -H 'Content-type: application/json' --data '{"text":"Vote data collecting done"}' $SLACK_URL

exit 0

is there any space characters in your $FILE_DIR? try : cat /dev/null > "$FILE_DIR"/dailey_member.csv — Shen Yudong, Jan 28 '18 at 12:16
@yudongshen Thanks for the answer, but I already tried that one. — Juneyoung Oh, Jan 28 '18 at 12:18
Are you sure the command `cat /dev/null > "$FILE_DIR"/dailey_member.csv` really prints `arguments list too long` and not other script lines?. Please run your script in "**debugging mode**" (`sh -x`). Please, show us also the value of the `FILE_DIR` variable. — Jdamian, Jan 28 '18 at 13:40
Show the values of the variables you have, $FILE_DIR and $S3_PATH at least, there might be something in there... echo it first... — Monty Montemayor, Jan 29 '18 at 04:42
@MontyMontemayor Add file declaration and echo and execution result. Copy echo result and execute is works, but it does not in bash — Juneyoung Oh, Jan 29 '18 at 05:02
I translated `arguments list too long` since os language is Korean. — Juneyoung Oh, Jan 29 '18 at 05:03
it seems it is coming from inside aws command and not from your main script. — Monty Montemayor, Jan 29 '18 at 05:10
@MontyMontemayor But I still do not get it, since the command works independently with copy and paste. — Juneyoung Oh, Jan 29 '18 at 05:21
which command? the line aws .. ? put it inside a new script with just that line. use shebang and let us now the output. — Monty Montemayor, Jan 29 '18 at 05:24
and call "getconf ARG_MAX" from inside that script, show us the output of this as well — Monty Montemayor, Jan 29 '18 at 05:27
Tangentially, capturing the result as a string so you can copy it to an array (unquoted!) so you can loop over it is potentially a huge waste of memory. Just pipe `mysql` output into a `while IFS=$'\n' read -r line` loop if you have no other reason to keep the results in memory. — tripleee, Jan 29 '18 at 05:27
Looking at this in more detail, this looks like you simply need `mysql | sed $'s/\t/,/g' > file.csv` though there is probably some option to have MySQL create output in the format you need directly. — tripleee, Jan 29 '18 at 05:29
But those are tangential remarks I agree with @MontyMontemayor that this seems to be a bug in `aws s3 cp` — tripleee, Jan 29 '18 at 05:31
`echo -e $LINE` without quotes looks like a bug waiting to blow up on you anyway. [You need quotes](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) unless you specifically *require* the shell to perform whitespace tokenization and wildcard expansion on the value. — tripleee, Jan 29 '18 at 05:33
@MontyMontemayor in new file result of `getconf ARG_MAX` is `2621440`, but In the original script it returns `arguments list too long` — Juneyoung Oh, Jan 29 '18 at 05:43
@tripleee It works beautiful you suggested in reply. using `mysql` and `sed`, a wonderful method to achieve the goal I mentioned. b — Juneyoung Oh, Jan 29 '18 at 06:13
As @MontyMontemayor pointed out, this issue has to do with `aws`, not with `bash`. And in the debug mode, `cat /dev/null > "$FILE_DIR"/dailey_member.csv` shows no error message. — Jdamian, Jan 29 '18 at 12:22
Did aws command ran without any issues in the new script? If it did, there's something more in the original script that is limiting the ARG_MAX causing this issue. — Monty Montemayor, Jan 29 '18 at 12:28
@Jdamian Indeed `cat` does not the problem - I just can not reproduce. In original script, I got same logic of this code four times and after `for` ~ `done`, this error happens. — Juneyoung Oh, Jan 29 '18 at 13:45
@MontyMontemayor I added extra information to the post. I am not sure what is different with `aws` command... — Juneyoung Oh, Jan 29 '18 at 13:50
Using `sh` to run a `bash` script is also at the very least misdirected. See https://stackoverflow.com/questions/5725296/difference-between-sh-and-bash (and read the question about quoting I linked to again; you are still making the same type of mistake with complete abandon). — tripleee, Jan 29 '18 at 14:15
btw, your shebang has space after !, there should be no space. `#! /bin/bash` so... aws ran without any issues in a separate script, but not in the original one. I highly suspect that it is somewhere in your original script that is causing it, based from what you are telling us now, this doesn't seem an aws issue, even a simple command i.e getconf would produce the same error. I suggest to carefully review your original script. But without your letting us look at your entire script, there's no way we can assist you further, we can only guess. — Monty Montemayor, Jan 29 '18 at 21:21
@MontyMontemayor Thanks. I thought using like `#! /bin/bash` is normal. I added the original script to the post. please review the script, tell me anything not good. — Juneyoung Oh, Jan 30 '18 at 02:10
BTW, environment variables count against your maximum command-line length (so if you `export` excessively large variables, then you can prevent even external programs with short command lines from running). If any scripts you've run go will-nilly `export`ing things that could just as easily be shell-local variables instead of exporting them to the environment, that's liable to be an exacerbating factor. Check the length of values you've `export`ed, and if you don't need to `export` something, then **don't**. — Charles Duffy, Jan 30 '18 at 02:15
If you haven't run your code through http://shellcheck.net/ and fixed what it identifies, by the way, I'd suggest doing that sooner rather than later. — Charles Duffy, Jan 30 '18 at 02:16
as what @CharlesDuffy suggested, ran it through shellcheck, and fix your shebang. Run it and see if the error still exist. — Monty Montemayor, Jan 30 '18 at 04:40

Charles Duffy · Accepted Answer · 2018-01-31T21:58:05.460

Your code is setting too many (or too-large) environment variables.

Environment variables live in the same per-process space as command line arguments, so exporting variables to the environment reduces the space available for command-line arguments. (If you aren't explicitly exporting content inside your loop, check whether echo $- inside your script contains a, a flag which tells the shell to automatically export every variable set; if it does, find the place where you're turning that option on, and disable it).

To demonstrate this (with GNU xargs), you can run:

xargs --show-limits </dev/null

Output should look something like:

Your environment variables take up 2615 bytes
POSIX upper limit on argument length (this system): 257481
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 254866
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647

Subtract the number on the second line from the number on the first line, and that's your maximum command-line length (roughly). Number on the first line too big? Look through your environment, and either unset large variables, or don't export them in the first place.

Thanks for all. using `export` is the one of my habits when I writing a script. (`#! /bin/bash` also... I removed the empty space) I should care of using `export` keyword. bb — Juneyoung Oh, Feb 05 '18 at 06:17

Bashshell - argument too long message shows after for loop-echo command

1 Answers1

Linked