Python use last x lines

Question

I've got script to read data from Arduino and save it as TXT file. I also have script to separate data and to add date/time, than save in new TXT file.

I've got also few Cron scripts like:

Read temeprature every 1 minute and save as TXT file (Script 1)
Read temperature every 10 minutes and save as TXT file (Script 2)
Read temperature every 60 minutes and save as TXT file (Script 3)

I would like to make a few charts:

Temperature in last hour
Temperature in last 3 hours
Temperature in last 12 hours
Temperature in last 24 hours
Temperature in last 7 days

So I need new Python script to:

Get last 60 lines from Script 1 - save it as new TXT file, to use it as chart "Temperature in last hour"
Get last 18 lines from Script 2 - save it as new TXT file, to use it as chart "Temperature in last 3 hours"
Get last 72 lines from Script 2 - save it as new TXT file, to use it as chart "Temperature in last 12 hours"
Get last 144 lines from Script 2 - save it as new TXT file, to use it as chart "Temperature in last 24 hours"
Get last 168 lines from Script 3 - save it as new TXT file to use it as chart "Temperature in last 7 days"

Can you help me to write simple script for example to get last 60 lines from FILE.txt and save them as FILE2.txt ? I will edit it for next scripts I need.

EDIT: I probbably know how to keep my chart files sized as I want (60 lines), but now I would like to ask is it possible to make any script, who will delete FIRST 30 lines?

I have only about 1 GB space, so I need to clean TXT files ;) If I will get temperature every minute after two weeks It will full my hard drive ;)

So I think CRON action to delete first X lines from txt file can help me a lot. Do you know any script?

If I will get it, I will finally finish my master project and ofcourse I will show you results :)

Here is the simple script to put the last 60 lines of `FILE.txt` in `FILE2.txt`, considering you're using Linux, Mac OS X or another unix based OS: `tail -n60 FILE.txt > FILE2.txt` — tamasgal, May 11 '13 at 07:54

score 1 · Answer 1 · answered May 11 '13 at 08:00

1

You can use the tail recipe from collections.deque here:

from collections import deque

def tail(filename, n=10):
    with open(filename) as f:
        return deque(f, n)

lines = tail("script",18)

answered May 11 '13 at 08:00

Ashwini Chaudhary

244,495
58
464
504

1

Yes but this is not good for huge files since you are still iterating through the whole file, although it is very elegant – jamylak May 11 '13 at 08:10

score 0 · Answer 2 · answered May 11 '13 at 08:12

Here's my version of glenbot's idea :

import os

def tail(f, n=1, _buffer=1):
    """Tail a file and get X lines from the end"""
    # place holder for the n found
    lines = []

    block_counter = -1

    while True:
        try:
            f.seek(block_counter * _buffer, os.SEEK_END)
        except IOError:  # either file is too small, or too many n requested
            f.seek(0)
            return f.readlines()

        lines = f.readlines()

        # we found enough, get out
        if len(lines) > n:
            return lines[-n:]

        # decrement the block counter to get the next X bytes
        block_counter -= 1

score 0 · Accepted Answer · answered May 11 '13 at 08:47

Here is your small program, just run it every one minutes in cron

#!/usr/bin/env python
from random import randrange
import datetime

now = datetime.datetime.now()

#generate random temperature
temperature = randrange(50, 100)

#database structure
structure = "{'last_hour': [], 'last_3_hours': [], 'last_12_hours': [], " \
            "'last_24_hours': [], 'last_7_days': []}"

#file for the database
database = 'temperature_database.txt' #use absolute path when running from cron

#check database file exists, if not write the database structure
try:
    with open(database): pass
except IOError:
    with open(database, 'w') as handler:
        handler.write(structure)

#read the contents of the database
with open(database, 'r') as handler:
    db_contents = eval(handler.read())

#first save every minute
db_contents['last_hour'].append(temperature)
if len(db_contents['last_hour']) > 60:
    db_contents['last_hour'] = db_contents['last_hour'][-60:] #get the last 60 elements

if now.minute in [10, 0]: #every ten minutes
    db_contents['last_3_hours'].append(temperature)
    if len(db_contents['last_3_hours']) > 18:
        db_contents['last_3_hours'] = db_contents['last_3_hours'][-18:]

    db_contents['last_12_hours'].append(temperature)
    if len(db_contents['last_12_hours']) > 72:
        db_contents['last_12_hours'] = db_contents['last_12_hours'][-72:]

    db_contents['last_24_hours'].append(temperature)
    if len(db_contents['last_24_hours']) > 144:
        db_contents['last_24_hours'] = db_contents['last_24_hours'][-144:]

if now.minute == 1: #every hour
    db_contents['last_7_days'].append(temperature)
    if len(db_contents['last_7_days']) > 168:
        db_contents['last_7_days'] = db_contents['last_7_days'][-168:]

#save the contents to the database
with open(database, 'w') as handler:
    handler.write(str(db_contents))

After four minutes the file contains

{'last_hour': [62, 99, 83, 71], 'last_12_hours': [], 'last_24_hours': [], 'last_3_hours': [], 'last_7_days': []}

Python use last x lines

3 Answers3