python: outfile to another text file if exceed certain file size

Question

I using scapy with python in ubuntu. I would like to ask if anyone would know how to code the example:

let say I have two text files which are writing while the script is running then I would like to check the file is exceed example 500bytes, if does then it will store whatever in the file and create a new text file to write. (output1.txt, output2.txt,etc..)

Would be appreciated if any expertise will help. Thanks

part of my code is:

file = open("output.txt","w")

def example(p):
    if p.haslayer(Dot11Beacon):
        if p.addr2 not in uniqueAP:
            file.writelines(p.addr2 + "\n")

so while the script is running in the terminal, it will write it into the file called output.txt but i would want to improve the script to check file size of the text file and if exceed it would stop writing in the current and create a new output2.txt for example and continue.

I would keep track of the bytes written to each file within your program, within a special file writer class, for example. Anyway, what have you tried so far? — Joel Cornett, Jun 05 '12 at 09:28
Do you mean _the script_ is writing to the files, or something else and you just check the size? — phipsgabler, Jun 05 '12 at 09:29
I would look into [RotatingFileHandlers](http://docs.python.org/library/logging.handlers.html#rotatingfilehandler). — miku, Jun 05 '12 at 09:31

Mike Pennington · Accepted Answer · 2012-06-05T15:00:45.253

17

A simple example if you don't want to use RotatingFileHandler.

You should use os.stat('filename').st_size to check file sizes.

import os
import sys

class RotatingFile(object):
    def __init__(self, directory='', filename='foo', max_files=sys.maxint,
        max_file_size=50000):
        self.ii = 1
        self.directory, self.filename      = directory, filename
        self.max_file_size, self.max_files = max_file_size, max_files
        self.finished, self.fh             = False, None
        self.open()

    def rotate(self):
        """Rotate the file, if necessary"""
        if (os.stat(self.filename_template).st_size>self.max_file_size):
            self.close()
            self.ii += 1
            if (self.ii<=self.max_files):
                self.open()
            else:
                self.close()
                self.finished = True

    def open(self):
        self.fh = open(self.filename_template, 'w')

    def write(self, text=""):
        self.fh.write(text)
        self.fh.flush()
        self.rotate()

    def close(self):
        self.fh.close()

    @property
    def filename_template(self):
        return self.directory + self.filename + "_%0.2d" % self.ii

if __name__=='__main__':
    myfile = RotatingFile(max_files=9)
    while not myfile.finished:
        myfile.write('this is a test')

After running this...

[mpenning@Bucksnort ~]$ ls -la | grep foo_
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_01
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_02
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_03
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_04
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_05
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_06
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_07
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_08
-rw-r--r--  1 mpenning mpenning    50008 Jun  5 06:51 foo_09
[mpenning@Bucksnort ~]$

edited Jun 05 '12 at 15:00

answered Jun 05 '12 at 11:52

Mike Pennington

41,899
19
136
174

Thanks Mike Pennington you saved my life! But is it possible to add current date to the filename when it has done written with the file? example: foo_*current-date*_01, etc. – leong Jun 13 '12 at 01:41
@leong, if my answer was useful, please consider accepting it. You can add the date to the filename by changing the return value of `filename_template()` to be `return self.directory + self.filename + str(datetime.date.today()) + "_%0.2d" % self.ii`. If you do this, also `import datetime` at the top of the script – Mike Pennington Jun 13 '12 at 02:58
nice! sorry one last question: will it works with date plus current time? – leong Jun 13 '12 at 05:02
actually my the other concept is filename by current date-plus-time with millisecond, then if all are same it will then increase the `ii`, if not it will use the current date-plus-time for the name. i tried replace with `str(time.strftime("%m/%d/%y %H:%M:%S:%f"))` but error. – leong Jun 13 '12 at 06:32
1

leong, you should use `str(datetime.datetime.now()).replace(' ', '_')`; however, the [os keeps file creation time as metadata (at least to second precision, sometimes to millisecond precision)](http://stackoverflow.com/a/237082/667301), so I'm surprised that you'd want to write the file creation time into the filename – Mike Pennington Jun 13 '12 at 08:23
although I said creation time, remember that you can also get last modification time if you like – Mike Pennington Jun 13 '12 at 10:29
recently I tried to change the output dir but it doesn't seem to work. Now the problem is if i change `directory='/home/user/Output'` it did not output the text file to the dir of the folder of "Output" instead it name the text file to the same dir and the text filename goes "Output". – leong Jul 05 '12 at 01:38
1

Please use `directory='/home/user/Output/'` – Mike Pennington Jul 05 '12 at 01:47
Nice answer, I'm curious if there's any reason *not* to do a similar thing but using `self.fh.tell()` instead of `os.stat(path).st_size`? – JawguyChooser Feb 14 '18 at 00:41

score 1 · Answer 2 · answered Aug 27 '18 at 12:49

Another way (a bit shorter) would be to use a with clause having an if condition checking for the file size:-

def RotateFile(count, file_size, file_name):
    indX = 0
    while indX < count:  
        with open(file_name,"a") as fd:  
            ##
            fd.write("DUMMY TEXT\n")      
            ##  
            if int(os.path.getsize(file_name)) > file_size:
                indX += 1
                file_name = "new_file"+str(indX)+".txt"

with statement creates a context manager, so by the time you reach the end of if condition , the file object in fd will have been automatically closed.

Next is to specify the options you need and pass on to the method:

if __name__ == "__main__":
    count = 10                           ## 10 number of files
    file_size = 10000                    ## 10KB in size
    file_name = "new_file.txt"           ## with this name
    RotateFile(count, file_size, file_name)

After execution, it will give 10 files of size 10KB each (in the current dir)...

python: outfile to another text file if exceed certain file size

2 Answers2

Linked