1

Attempting to convert a twitter account of over 10K tweets into another format with a bash script on a maxed out MBP 16" running the latest macOS.

After running for several minutes outputting many periods it says, line 43: /bin/ls: Argument list too long. Assuming this issue relates to the number of tweets so while I could attempt to break into small pieces as a last resort, not knowing what the max number to avoid the error is, decided to first search for a solution.

Searched Google and SO and found, "bash: /bin/ls: Argument list too long". If my issue is the same it sounds like replacing "ls" with "find -name" may help. Tried and same error, but perhaps not the correct syntax.

The two lines that use "ls" currently are the following (the first is the one the error currently complains about):

for fileName in `ls ${thisDir}/dotwPosts/p*` ; do

and

printf "`ls ${thisDir}/dotwPosts/p* | wc -l` posts left to import.\n"

Tried changing the first line to (with the error saying /usr/bin/find: Argument list too long).

for fileName in `find -name ${thisDir}/dotwPosts/p*` ; do

May need to provide additional code, but didn't want to make the question too specific to my needs and more general hopefully for others seeing this common error where the other stackoverflow answer didn't seem to apply.

TechRemarker
  • 2,788
  • 11
  • 37
  • 60
  • 4
    Why do you need to use `ls`? Just `for fileName in ${thisDir}/dotwPosts/p*` – Barmar Feb 11 '20 at 01:22
  • 2
    Barmar is quite right; but for the record, if you *did* want to fix this by using `find`, the correct version is `find "$thisDir/dotwPosts" -maxdepth 1 -name 'p*'`. *Merely* replacing `ls` with `find -name` couldn't possibly work, because you'd be sending all of the same arguments to `find` that you'd sent to `ls`. If the kernel won't allow the latter, then it won't allow the former, either. – ruakh Feb 11 '20 at 01:24
  • 1
    `find` is also no better than `ls` with regard to iterating over its output. – chepner Feb 11 '20 at 01:34
  • @cchiera : Another way to shorten the argument list would be to do a `cd ${thisDir}/dotwPosts; for fileName in *;...; do`. Of course you have then only the basename of the file in `fileName` and your working directory is also different, but both are issues which are trivial to deal with. If you still have too many directory entries, consider to replace the loop by a `find .... | xargs ....`, if your logic permits it. Since I don't know what's going on inside the loop, I don't know how feasible this alternative is. – user1934428 Feb 11 '20 at 05:22

2 Answers2

1

To iterate over file in a directory in bash, print the filenames as a zero separated stream and read it. That way you don't need to store all filenames at once in any place:

find "${thisDir}/dotwPosts/" -maxdepth 1 -type f -name 'p*' -print0 |
while IFS= read -d '' -r file; do
   printf "%s\n" "$file"
done

To get the count, output a single character for each file and count the characters:

find "${thisDir}/dotwPosts/" -maxdepth 1 -type f -name 'p*' -printf . | wc -c

Don't use ` backticks, they use is discouraged. Bash hackers wiki discouraged and deprecated syntax. Use $(...) instead.

for fileName in $(...) is a common antipattern in bash. Most probably if you want to iterate over output of another command, you should use while IFS= read -r line loop. bashfaq How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?

KamilCuk
  • 120,984
  • 8
  • 59
  • 111
0

Try this:

for file in "${thisDir}/dotwPosts/p"*
do
    # exclude non plain files
    [[ -f $file ]] || continue 
    # do something with "$file"
    ...
done

I quoted "${thisDir}/dotwPosts/p", so var thisDir can't contain a relevant wildcards, but works with blanks. Otherwise remove the quotes.

Wiimm
  • 2,971
  • 1
  • 15
  • 25