3

I am trying to load and parse a really large text file. Although the loading is not a problem, but there are particular lines that have 2908778 characters on a single line.

This is causing an error in my script.

On the script below, I removed all logic and just got straight to read line. I also removed all valid lines and just left the really long line in one text file. When running I get the below error :

$ dowhiledebug.sh dump.txt
dowhiledebug.sh[6]: no space
Script Ended dump.txt

The actual script:

 #!/bin/sh
 filename=$1
 count=1
 if [ -f ${filename} ]; then
    echo "after then"
    while read line;
            do
            echo "$count"
            count=$((count+1))
            done < $filename
 else
    echo "Could not open file $filename"
 fi
 echo "Script Ended $filename"

Updated (2013-01-17)

Follow up question : Is it possible to increase the maximum number of characters that ksh variable accepts?

Community
  • 1
  • 1
javapadawan
  • 897
  • 6
  • 12
  • 29
  • 2
    Why not use `wc -l` to count the lines in the file? It won't have the limits of the shell. I guess the answer is "because I need to do other processing which I've removed for the reproduction". – Jonathan Leffler Jan 17 '13 at 02:40
  • On my Mac with `bash` 3.2, I created a file 4194304 characters and no newline at all, and then `bash` ignored the line altogether. I appended a single newline, and `bash` was quite happy to read the whole lot into memory. So, your size is not a hard limit. You'll need to look at how much memory there is on your system (more than 3 MiB, I'm sure), and whether the shell has many huge variables using up memory. – Jonathan Leffler Jan 17 '13 at 03:50
  • And, FWIW, the `sysconf` value for ARG_MAX on the machine is 256 KiB, just as in the answer. I don't think the limit is directly related to ARG_MAX (though, I confess, I'm mildly surprised that I was able to echo a 4 MiB string to `wc`). This is on the Mac, still. – Jonathan Leffler Jan 17 '13 at 03:57
  • what OS and version of ksh? Can you `echo ${.sh.version}` and get a value? If so, please include in your question above. Or could this be pdksh? Good luck to all. – shellter Jan 17 '13 at 04:09
  • @shelter AIX $ Version M-11/16/88f – javapadawan Jan 17 '13 at 05:25
  • 2
    Well don't do that then! Find some solution for your problem that doesn't require you to load the entire line into a ksh variable. – me_and Jan 17 '13 at 11:59
  • @user1985725 : `ver 88/f` That's part of your problem. See if you SysAdmin can point you to a newer version. If you're running on AIX, you're likely not in an environment where you can have someone build a one-off with larger ARG_MAX. It's likely time to consider another language, or analyse your processing to find another solution. Good luck. – shellter Jan 17 '13 at 12:56
  • I would use `python` or some other scripting language for that purpose. – Basile Starynkevitch Jan 17 '13 at 20:33
  • [What is the maximum size of an environment variable value?](http://stackoverflow.com/q/1078031/995714) – phuclv Nov 24 '16 at 10:32
  • 1
    @JonathanLeffler: The reason you were able to `echo` such a large string is that `echo` is a _builtin_ in Bash, in which case ARG_MAX doesn't apply. Compare `echo "$(awk 'BEGIN { while (c++ < '"$(( $(getconf ARG_MAX) + 1 ))"') printf "=" }')"` to `/bin/echo "$(awk 'BEGIN { while (c++ < '"$(( $(getconf ARG_MAX) + 1 ))"') printf "=" }')"`. That said, I too think that ARG_MAX is not relevant to the OP's problem. – mklement0 Jan 17 '17 at 16:57

2 Answers2

6

what OS and version of ksh? Can you echo ${.sh.version} and get a value? If so, please include in your question above. Or could this be pdksh?

Here's a test that will get you in the ballpark, assuming a modern ksh supporting (( i++ )) math evaluations:

#100 char var
var=1234578901234456789012345678901234567890123456789012345789012344567890123456789012345678901234567890

$ while (( i++ < 10000 )) ;do  var="$var$var" ; print "i=$i\t" ${#var} ; done
i=1      200
i=2      400
i=3      800
i=4      1600
i=5      3200
i=6      6400
i=7      12800
i=8      25600
i=9      51200
i=10     102400
i=11     204800
i=12     409600
i=13     819200
i=14     1638400
i=15     3276800
i=16     6553600
i=17     13107200
i=18     26214400
i=19     52428800
i=20     104857600
i=21     209715200
i=22     419430400
-ksh: out of memory

$ print -- ${.sh.version}
Version JM 93t+ 2010-05-24

AND that is just the overall size of the environment that can be supported. When dealing with the command-line environment and "words" after the program name, there is a limit to the number of words, regardless of overall size.

Some shells man page will have a section LIMITS that may show something like max-bytes 200MB, max-args 2048. This information may be in a different section, it will definitely have different labels and different values I have included, OR it may not be there at all, hence the above code loop, so look carefully around and if you find a source for this info, either add an answer to this Q, or update this one.

The bash 4.4 std man page doesn't seem to have this information and its harder to find a ksh doc all the time. Check your man ksh and hope that you can find a documented limit.

IHTH

shellter
  • 36,525
  • 7
  • 83
  • 90
  • Hi shelter, assuming that there is a limit. Is there a way to extend t this limit? Because I'm sure that the records I'm loading does not stop at 2M characters per line? Is there an alternative to "while read line"? – javapadawan Jan 17 '13 at 05:19
1

The limit for any shell is the limit of the C command line maximum. Here's a little program that pulls the information out of /usr/include/limits.h for you:

cpp <<HERE | tail -1
#include <limits.h>
ARG_MAX
HERE

Mine gives me (256 * 1024) or 262144 characters.

Doesn't work if the C compiler isn't installed, but it's probably a similar limit.

David W.
  • 105,218
  • 39
  • 216
  • 337
  • While it's good to know the max. length of a command line when invoking an external utility (which can more easily be obtained with `getconf ARG_MAX`), this limit does _not_ apply here, because the data is read from a file, not passed on the command line; also, `read` is a shell builtin, not an external utility. The following `ksh` command reads a line that is 1 byte larger than `getconf ARG_MAX`, which succeeds: `read line < <(awk 'BEGIN { while (c++ < '"$(( $(getconf ARG_MAX) + 1 ))"') printf "=" }'); echo "${#line}"`. – mklement0 Jan 17 '17 at 16:44