Is there a line length limit for text files created from Perl?

Question

While writing a Perl script, I got a requirement to write the user names with comma separation in only one line of the file.

That's why I would like to know is there any restriction on the maximum size of the line in the .txt file.

What file size are you expecting to create? – brian d foy Sep 20 '09 at 17:50 — brian d foy, Sep 20 '09 at 17:50

score 14 · Answer 1 · edited Sep 20 '09 at 17:45

14

Text files are just like any other files and newline character is like any othe character, so only the usual filesize restrictions apply (4Gb size limit on older file systems, file must fit on the disk etc.)

You won't encounter any problem reading and writing it, unless you're reading it line by line—you can run out of memory then or encounter a buffer overflow of sorts. This may happen in any text editor or text processing program (such as sed or awk), because, unlike OS kernel, in those line separation matters

I would suggest keeping one user per line, as it's more natural to read and less error-prone when you process the file with an external program.

edited Sep 20 '09 at 17:45

brian d foy

129,424
31
207
592

answered Sep 19 '09 at 12:59

P Shved

96,026
17
121
165

At least kwrite and vi doesnt get affected by the line size (tested on a 4Mb single line XML file) – Cem Kalyoncu Sep 19 '09 at 13:01
There's certainly a limit. It has to fit on the disk, and if you are reading it line-by-line, it has to fit in memory. In addition to that, you might need large file support to deal with files over 4 Gb. – brian d foy Sep 19 '09 at 23:32
@brian d foy: since you're more experienced here, over SO, I'm following your advice and playing Captain Obvious, so now my post says that a file on a disk should not exceed the size of that disk. Sigh. – P Shved Sep 20 '09 at 05:40
2

Nothing is obvious. You might be able to create a string in Perl that you can't save to your full disk but that you can fit in program memory, and you might not have enough program memory to read an entire file in one go. They are real problems you have to handle when you play with very large strings and file, but most people never think about them. – brian d foy Sep 20 '09 at 17:49
@brian d foy: okay, you say many right things everyone should remember about. But didn't you notice that the topic has changed from "newlines and OS" to "handling large files"? Are you sure it's the right way to go? – P Shved Sep 20 '09 at 19:05
Am I sure what is the right way? If you're talking about lines, see my answer. – brian d foy Sep 21 '09 at 00:13

score 5 · Answer 2 · answered Sep 19 '09 at 23:45

The only thing you need to worry about is the size of the file that you can create and the size of the file that you can read.

Computers don't know anything about lines, which is an interpretation of the bytes in a file. We decide that there is some sequence of characters that demarcate the end of a line, and then tell our programs to grab stuff out of the file until it hits that sequence. To us, that's a line.

For instance, you can define a line in your text file to end with a comma:

 $/ = ',';

 while( <DATA> )
    {
    chomp;
    print "Line is: $_\n";
    }

 __DATA__
 a,b,c,d,e,f,g

Even though it looks like I have a single line under __DATA__, it's only because we're used to books. Computers don't read books. Instead, this program thinks everything between commas is a line:

Line is: a
Line is: b
Line is: c
Line is: d
Line is: e
Line is: f
Line is: g

score 3 · Answer 3 · answered Sep 19 '09 at 12:59

3

There is no size limit except your filesystem's which is most probably 2TB or something.

answered Sep 19 '09 at 12:59

Cem Kalyoncu

14,120
4
40
62

score 3 · Answer 4 · answered Sep 19 '09 at 13:04

3

No, there is no such limit until you hit any file-size limits.

answered Sep 19 '09 at 13:04

Joe Casadonte

15,888
11
45
57

score 2 · Answer 5 · answered Oct 16 '11 at 07:29

I just came across such a problem with a line about 1M chrs long in Kwrite.

Though there is no theoretical limit, if you are to work on you file, you'll have to have the line wrapped to display width. At every edit, many calculations are performed, perchance involving swap memory. Thar makes editing clumsy. Long lines may be quite inconvenient.

score 1 · Answer 6 · answered Sep 19 '09 at 13:27

1

On some old Unix systems, some text utilities (e.g. join, sort and even some old awk) have a limit on the maximum line size. I think this is the limit of utilities but not the OS. GNU utilities do not have such a limit as far as I know and therefore Linux never has this problem.

answered Sep 19 '09 at 13:27

user172818

4,518
1
18
20

I used to have this problem on an old version of IRIX and AIX. Then I installed GNU textutils (not merged to coreutils) in my home directory, which solved the problem – user172818 Sep 19 '09 at 20:07

score -1 · Answer 7 · answered Sep 19 '09 at 19:09

-1

file size depends on your OS's file system. Tools has no limit for such (or at-least I have never seen so far..)

answered Sep 19 '09 at 19:09

Kartik Mistry

445
1
3
8

Some tools have limits because they use a four-byte int address space, which is why there is large file support in some tools. – brian d foy Sep 19 '09 at 23:33

Is there a line length limit for text files created from Perl?

7 Answers7

Linked