Argument list too long for input

Question

I am executing a shell script, on a high level it reads the records from a csv and consequently do some DB operation. I have analyzed by manually running the script. Its running fine for less than 900 records in file but it gives the error for more than 900 records. Below is the screenshot of the error which i get after some time:

There is a part of the script which is picking record 1 by 1:

Could you please suggest why this is happening? I have read similar topics when user got that error, but unable to relate them with my scneario.

Cheers

I don't think you posted lines 109-111, where the errors are occurring. — Barmar, Jul 03 '14 at 18:10
_Could you please suggest why this is happening?_ ... because *the argument list is too long*. records are passed to shell, shell has a limit on argument list length. — , Jul 03 '14 at 18:12
Try to replace `find *XYZ*.csv > list_of_files.txt` with `find . -name "*XYZ*.csv" > list_of_files.txt` to avoid globbing. — Cyrus, Jul 03 '14 at 18:16
@Barmar I have updated my question with those lines. I was suspecting this its failing after 900 records, something is wrong with the above code pasted as there it is selecting the record. — FelixTheCode, Jul 03 '14 at 18:17
@vaxquis Can you please tell me how to check the limit on argument list length. — FelixTheCode, Jul 03 '14 at 18:18
I don't see how those lines can be causing `tail: argument list too long` errors. You only give one argument to `tail`. — Barmar, Jul 03 '14 at 18:19
@Barmar I am also trying to figure out the same thing. But not able to make out why it is happening. I have pasted the full script here : http://pastebin.com/vtRCezD8 — FelixTheCode, Jul 03 '14 at 18:27
The line numbers there don't seem to match up with the error messages. — Barmar, Jul 03 '14 at 18:29
btw, obviously one of the variables used in batch processing is set to some absurdly long value somewhere (>250k len), probably due to low quality of the script itself. I'd suggest looking at $v_DateTime as the culprit. — , Jul 03 '14 at 18:29
@vaxquis `$v_DateTime` is pretty simple: `v_DateTime=\`date +%d%m%y_%H%M%S\` ` — Barmar, Jul 03 '14 at 18:31
Put `set -x` at the beginning of the script. It will then show you what it's doing, with all the variables expanded. — Barmar, Jul 03 '14 at 18:32
@Barmar Sorry i have removed the comments, http://pastebin.com/suVB3ugz I will work on your and vaxquis suggestions, thanks you gave direction to debug it. — FelixTheCode, Jul 03 '14 at 18:34
it could have been $filename or $BATCH_PROCESSES too, with either a typo or a logic error somewhere in them. the question asked by OP was "why is this happening", and I consider this question answered already. I don't see a question asking "how to fix this" here. Also, I'd tag it as "too localized" if the tag was still there... — , Jul 03 '14 at 18:43
@vaxquis: It could also be cumulative environment spam, eg. `for i in $(seq 20000); do export JABBERWOCKY_$i="'Twas brillig and the slithy toves did gyre and gimble in the wabe: all mimsy were the borogoves, and the mome raths outgrabe."; done` — rici, Jul 03 '14 at 20:45
@user3802822: Count lines with just awk: `file_lines=$(awk 'END{print NR}' file)`. Loop over comma-separated file: `while IFS=, read _ _ Action Reason Date Account Orig Name Value _; do ... done < "$BATCH_PROCESSES/bulk/input/$v_DateTime/$filename"` — rici, Jul 03 '14 at 20:51

score 2 · Answer 1 · edited May 23 '17 at 10:26

I've hit this problem before and it's quite easy to replicate:

unset a; export a=$(perl -e 'print "a"x(1024*64)'); whoami
tiago

unset a; export a=$(perl -e 'print "a"x(1024*128)'); whoami
bash: /usr/bin/whoami: Argument list too long

perl -e 'print "a"x(1024*64)' | wc  -c
65536

perl -e 'print "a"x(1024*128)' | wc  -c
131072

So something between 65536 and 131072 bytes breaks, When I had This problem instead of export the value I was printing and used pipes to work with data. Another way around is to use files.

You can find nice experiments: What is the maximum size of an environment variable value?

Argument list too long for input

1 Answers1