Splitting a big file into smaller ones basing on a line

Question

I have a pretty big file (more than 20GB) and I'd like to split it into smaller ones, like multiple files of 2GB.

One thing is I have to split before a specific line:

I'm using Python, but if there another solution in shell for example, I'm up for it.

This is how the big file looks like:

bigfile.txt (20GB)

Recno:: 0
some data...

Recno:: 1
some data...

Recno:: 2
some data...

Recno:: 3
some data...

Recno:: 4
some data...

Recno:: 5
some data...

Recno:: x
some more data...

This is what I want:

file1.txt (2 GB +/-)

Recno::0
some data...

Recno:: 1
some data...

file2.txt (2GB +/-)

Recno:: 2
some data...

Recno:: 4
some data...

Recno:: 5
some data...

And so on, and so on...

Thanks !

Possible duplicate of this? http://stackoverflow.com/questions/2016894/how-to-split-a-large-text-file-into-smaller-files-with-equal-number-of-lines — Chris_Rands, Jul 26 '16 at 13:10
It would be useful if you showed us some small examples with a few lines, showing where the files would be split (or not). — Tom Fenech, Jul 26 '16 at 13:10
@Chris_Rands Not really because I don't want to split with a given set of lines but with specific lines. Only when it exceeds 2Go and when a Recno:: *int* appears. — Difender, Jul 26 '16 at 13:12
@TomFenech I added some more in my example, please let me know if I'm not precise enough. — Difender, Jul 26 '16 at 13:14
As the other answer says you can use `split -b` in bash to split on bytes — Chris_Rands, Jul 26 '16 at 13:15
To be honest I don't understand what the conditions are for splitting the file at that point. — Tom Fenech, Jul 26 '16 at 13:16
@cricket_007 My bad it's GB, I'm French and I juste wrote Go.. — Difender, Jul 26 '16 at 13:17
@TomFenech I got a file with a lot of data but starting with `Recno:: int` each time. I want to split it to make little files of 2 GB but the problem is I cannot split somewhere random, I have to do it juste before the line `Recno:: int` so the datas inside are not splitted. — Difender, Jul 26 '16 at 13:19
Do you have Enought memory to store 2GB in your RAM? If yes, give me 10 minutes. — Thiago Baldim, Jul 26 '16 at 13:21
@TomFenech I added a better example, I think you will understand it better like that. — Difender, Jul 26 '16 at 13:22
So once you have read 2GB, do you want to go back to the previous section or continue to the next one? — Tom Fenech, Jul 26 '16 at 13:24
@TomFenech Continue to the next one. I can overlap a bit this is not a problem. — Difender, Jul 26 '16 at 13:27

score 1 · Accepted Answer · edited Jul 27 '16 at 11:31

You could do something like this:

import sys

try:
    _, size, file = sys.argv
    size = int(size)
except ValueError:
    sys.exit('Usage: splitter.py <size in bytes> <filename to split>')

with open(file) as infile:
    count = 0
    current_size = 0
    # you could do something more
    # fancy with the name like use
    # os.path.splitext
    outfile = open(file+'_0', 'w+')
    for line in infile:
        if current_size > size and line.startswith('Recno'):
            outfile.close()
            count += 1
            current_size = 0
            outfile = open(file+'_{}'.format(count), 'w+')
        current_size += len(line)
        outfile.write(line)
    outfile.close()

This is exactly what I was looking for, thank you so much! – Difender Jul 26 '16 at 13:55 — Difender, Jul 26 '16 at 13:55

score -1 · Answer 2 · answered Jul 26 '16 at 13:24

-1

As comment above mentions you can use split in the bash shell:

split -b 20000m <path-to-your-file>

answered Jul 26 '16 at 13:24

JoshuaBox

735
1
4
16

As I said I do not want to split ONLY on the size. I must split on the size but also on a given line. For example, each file has to start with a `Recno:: x` – Difender Jul 26 '16 at 13:25
you could monitor the file size in Python with `os.stat('/path/to/file/').st_size` in a while loop – JoshuaBox Jul 26 '16 at 13:38

Splitting a big file into smaller ones basing on a line

2 Answers2