4

This is a follow up question to

What is the maximum number of characters that the ksh variable accepts?

I checked my environment and it's allowing only 

#include <sys/limits.h>
$ cpp <<  HERE | tail -1
> #include <limits.h>
> ARG_MAX
> HERE
1048576

Is there a way to increase this? Or any alternatives for

 while read line;
   do
      #parse logic
   done < $filename

To handle really long lines? Based from the records I'm parsing it will not stop at 2M character lines.

Environment Details :

 AIX $ KSH Version M-11/16/88f
Community
  • 1
  • 1
javapadawan
  • 897
  • 6
  • 12
  • 29

1 Answers1

3

You could compile a Linux 3.7.x kernel, and edit its include/uapi/linux/limits.h file to increase the ARG_MAX argument (to some bigger power of two, e.g. 2097152). But you should rather have a lot of RAM (e.g. 8GBytes) if you want to increase it more.

The actual limit is related to execve(2). That man page has a paragraph on it.

But you could probably avoid having huge shell variables (in the Unix environment). Did you consider using some other tool (awk, python, perl ....) to read your file? Their variable environment is not the shell environment transmitted to forked programs, so they can have variables with very long values. Maybe ksh has some builtin (unexport) to avoid exporting some variable into the Unix environment.

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • Hi Basile, thanks for your response. I don't have liberty to recompile the kernel. One question, does this apply to all scripting languages? If I use perl will I also encounter this when doing a read line? – javapadawan Jan 17 '13 at 22:33
  • I am parsing a large file and some lines have that many records. The script is working 99% of the time, but for 1% of the records it has 2M+ characters per line. – javapadawan Jan 17 '13 at 23:40
  • 1
    `ARG_MAX` doesn't come into play here, because it limits the length of the command line when invoking _external utilities_, whereas `read` is a shell _builtin_ (and the data is read from a file, not passed as a command-line argument). The following `ksh` command demonstrates that it is possible to read lines that are longer than `ARG_MAX`: `IFS= read line < <(printf "%$(( $(getconf ARG_MAX) + 1 ))s"); echo "${#line}"` – mklement0 Jan 17 '17 at 18:08