Combining three scripts into one

Question

I have three scripts I have made, in my mind they represent the three things the program will have to do.

The first script constantly reads the incoming serial port, and writes all the information to a file, I do this because I am unsure of the size of the serial buffer in the deployment device and it is vital that I collect every single piece of the information from the serial port. To combine this into a singe file I read briefly on pythons threading capabilities but I have not researched what threading in terms of python is, the deployment device however is only a single core device for right now, if that matters in the terms of python threading.

The second piece of the code is used to validate and parse the file created by the first script and place it into a SQLite database.

The third file is written in python with wxPython extensions. It (not entirely complete) reads the information from the file and updates the wxPython display widgets.

While I am sure that a lot of this is inefficient it is also not really the way I wanted to do it but I have been trying to get help from python forums and well the help has been a bit non existant.

Top priority is to make sure that all of the data is collected from the serial port, then it needs to be stored in the database, for datalogging, and I would have preferred to display the data on the display at the same time as storing it in the database but I think that will create too much of a delay, with the code this way I can place the display on a timer and every 250ms I can update the display from the next information from the database.

Now that you have the information here is the question, how do I combine the parts I have to make one script. As well as will it be possible to read from a file as I write to it, like the serial collection file and the SQLite database?

Pointers to methods I should research are just as welcome as a direct answer, and I really do appreciate any and all help in advance!

Serial Scrape Code:

import serial

ser = serial.Serial('/dev/pts/7', 19200, timeout=0)
print ser.name          # check which port was really used

ScratchFile = open('Data/Scratch.scr', 'a')

x = 1
while True:
 SData = ser.readline()
 Valid = bool(SData)
 if Valid == True:
  #SData = (SData + "\n")
  ScratchFile.write(SData)
 #print(b)

ser.close()

Validate and Store Code:

#GKPCM Database Test


#outline open database, read serial,write string, repeat

import sqlite3
db = sqlite3.connect('Data/telemetry.gkpcm')
cursor = db.cursor()

InsertQuery ="""INSERT INTO vehicletelemetry (date,time,cyclecount,rpm,speed,odometer,oiltemp,airtemp,fuellevel,enginetemp,ind1,ind2,ind3,ind4,ind5,ind6,ind7,ind8) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"""

tablename="vehicletelemetry"
cursor.execute(""" SELECT COUNT(*) FROM sqlite_master WHERE name = ?  """, (tablename, ))
QUERY = cursor.fetchone()
print bool(QUERY[0]) # True if exists
if bool(QUERY[0]) !=1:
 print('not in DB')
 cursor.execute('''CREATE TABLE vehicletelemetry(id INTEGER PRIMARY KEY, date INTEGER,time INTEGER, cyclecount INTEGER, rpm INTEGER, speed INTEGER, odometer INTEGER, oiltemp INTEGER, airtemp INTEGER, fuellevel INTEGER, enginetemp INTEGER, ind1 BOOL, ind2 BOOL, ind3 BOOL, ind4 BOOL, ind5 BOOL, ind6 BOOL, ind7 BOOL, ind8 BOOL)''')
 cursor.execute('INSERT INTO vehicletelemetry (date,time,cyclecount,rpm,speed,odometer,oiltemp,airtemp,fuellevel,enginetemp,ind1,ind2,ind3,ind4,ind5,ind6,ind7,ind8) VALUES (031514,013030,18960,3000,22,192768,210,72,98,210,0,0,0,0,0,0,0,0)')
 db.commit()
else:
 print('DB Table Exists')

#openfile to read
with open('Data/Scratch.scr') as ScratchFile:
 for DataLine in ScratchFile:
  #Read Database for last record date
  LastEntry = cursor.execute('SELECT * FROM vehicletelemetry ORDER BY id DESC LIMIT 1')
  for Records in LastEntry:
    LastLogDate = Records[1]
    LastLogTime = Records[2]
    LastLogCycles = Records[3]
  DataLine1 = DataLine.strip()
  DataLine2 = DataLine1.split(',')
#Check Packet for Correct Length
  if len(DataLine2) - 2 != 18:
   print ("Invalid Data Length")
  else:
#Check Packet DataQualifiers to ensure proper package introduction and termination
   if DataLine2[0] != "7887" and DataLine2[18] != "0420":
    print ("Invalid Data Qulifier")
   else:
#Remove Qualifiers So data can be stored
    PiP = 1
    DataLine3 = []
    for Packet in DataLine2:
     if PiP >= 1 and PiP <= 18:
      DataLine3.append(DataLine2[PiP])
      PiP = PiP + 1
#Compare Date Time and Cycle Count to Current Record
    #print(DataLine3)
    if int(DataLine2[1]) >= int(LastLogDate): 
     if int(DataLine2[2]) >= int(LastLogTime): 
      if int(DataLine2[3]) > int(LastLogCycles):
       cursor.execute(InsertQuery,DataLine3)
       db.commit()
       print(Records,DataLine2[3],LastLogCycles,"Data Valid")

db.close()
#todo

Display Code:

import wx
import os
import wx.lib.agw.speedmeter as SM
import sys
import math
import wx.gizmos as gizmos
from wx.lib.colourdb import getColourList
import wx.lib.buttons
from math import pi, sqrt
import time


#GUI--------------------------------------------------------------------------------------------------------------------
#------------------------------------------------------------------------------------------------------------------------
#------------------------------------------------------------------------------------------------------------------------


class MyFrame(wx.Frame):

    def __init__(self,parent):
        wx.Frame.__init__(self,parent,-1,"test",pos=(3, 3),size=(480, 320))

        SPEEDOpanel = wx.Panel(self, -1,style=wx.NO_BORDER, pos=(240, 0), size=(240, 240))
        TACHpanel = wx.Panel(self, -1, pos=(0, 0), size=(240, 240))
        ODOpanel = wx.Panel(self, -1, pos=(125, 200), size=(125, 30), style=wx.NO_BORDER)
        AIRpanel = wx.Panel(self, -1, pos=(0, 240), size=(80, 80), style=wx.NO_BORDER)
        OILpanel = wx.Panel(self, -1, pos=(120, 240), size=(80, 80), style=wx.NO_BORDER)
        FUELpanel = wx.Panel(self, -1, pos=(240, 240), size=(80, 80), style=wx.NO_BORDER)
        ENGpanel = wx.Panel(self, -1, pos=(360, 240), size=(80, 80), style=wx.NO_BORDER)

        INDpanel1 = wx.Panel(self, -1, pos=(80, 240), size=(40, 40), style=wx.NO_BORDER)
        INDpanel2 = wx.Panel(self, -1, pos=(80, 280), size=(40, 40), style=wx.NO_BORDER)
        INDpanel3 = wx.Panel(self, -1, pos=(200, 240), size=(40, 40), style=wx.NO_BORDER)
        INDpanel4 = wx.Panel(self, -1, pos=(200, 280), size=(40, 40), style=wx.NO_BORDER)
        INDpanel5 = wx.Panel(self, -1, pos=(320, 240), size=(40, 40), style=wx.NO_BORDER)
        INDpanel6 = wx.Panel(self, -1, pos=(320, 280), size=(40, 40), style=wx.NO_BORDER)
        INDpanel7 = wx.Panel(self, -1, pos=(440, 240), size=(40, 40), style=wx.NO_BORDER)
        INDpanel8 = wx.Panel(self, -1, pos=(440, 280), size=(40, 40), style=wx.NO_BORDER)


        SPEEDOpanel.SetBackgroundColour(wx.BLACK)
        TACHpanel.SetBackgroundColour(wx.BLUE)
        ODOpanel.SetBackgroundColour(wx.RED)
        AIRpanel.SetBackgroundColour(wx.BLUE)
        OILpanel.SetBackgroundColour(wx.CYAN)
        FUELpanel.SetBackgroundColour(wx.BLACK)
        ENGpanel.SetBackgroundColour(wx.BLUE)

        INDpanel1.SetBackgroundColour(wx.BLUE)
        INDpanel2.SetBackgroundColour(wx.RED)
        INDpanel3.SetBackgroundColour(wx.BLUE)
        INDpanel4.SetBackgroundColour(wx.RED)
        INDpanel5.SetBackgroundColour(wx.BLUE)
        INDpanel6.SetBackgroundColour(wx.RED)
        INDpanel7.SetBackgroundColour(wx.BLUE)
        INDpanel8.SetBackgroundColour(wx.RED)


#SPEEDOMETER----------------------------------------------------------------------------------------------------------

        SPEEDO = SM.SpeedMeter(SPEEDOpanel, agwStyle=SM.SM_DRAW_HAND|SM.SM_DRAW_SECTORS|SM.SM_DRAW_MIDDLE_TEXT|SM.SM_DRAW_SECONDARY_TICKS, pos=(0, 0), size=(240, 240))

        # Set The Region Of Existence Of SpeedMeter 
        SPEEDO.SetAngleRange(-7.25, -2)

        # SpeedMeter In Sectors
        intervals = range(0, 260, 20)
        SPEEDO.SetIntervals(intervals)

        # Assign The Same Colours To All Sectors 
        # Usually This Is Black
        colours = [wx.BLACK]*12
        SPEEDO.SetIntervalColours(colours)

        # Assign The Ticks
        ticks = [str(interval) for interval in intervals]
        SPEEDO.SetTicks(ticks)
        # Set The Ticks/Tick Markers Colour
        SPEEDO.SetTicksColour(wx.RED)
        # We Want To Draw 5 Secondary Ticks Between The Principal Ticks
        SPEEDO.SetNumberOfSecondaryTicks(10)

        # Set The Font For The Ticks Markers
        SPEEDO.SetTicksFont(wx.Font(7, wx.SWISS, wx.NORMAL, wx.NORMAL))

        # Set The Text In The Center Of SpeedMeter
        SPEEDO.SetMiddleText("MPH")
        # Assign The Colour To The Center Text
        SPEEDO.SetMiddleTextColour(wx.WHITE)
        # Assign A Font To The Center Text
        SPEEDO.SetMiddleTextFont(wx.Font(8, wx.SWISS, wx.NORMAL, wx.BOLD))

        # Set The Colour For The Hand Indicator
        SPEEDO.SetHandColour(wx.Colour(255, 255, 0))

        # Set The Colour For The Gauge Background
        SPEEDO.SetSpeedBackground(wx.BLACK)

        # Do Not Draw The External (CONTAINER) Arc
        SPEEDO.DrawExternalArc(False)
        SPEEDO.SetSpeedValue(0)

#TACHOMETER----------------------------------------------------------------------------------------------------------

        TACH = SM.SpeedMeter(TACHpanel, agwStyle=SM.SM_DRAW_HAND|SM.SM_DRAW_PARTIAL_SECTORS|SM.SM_DRAW_MIDDLE_TEXT|SM.SM_DRAW_SECONDARY_TICKS, pos=(0, 0), size=(240, 240))

        # Set The Region Of Existence Of SpeedMeter 
        TACH.SetAngleRange(-6, -2)

        # SpeedMeter In Sectors
        intervals = range(0, 15, 1)
        TACH.SetIntervals(intervals)

        # Assign The Same Colours To All Sectors 
        # Usually This Is Black
        colours = [wx.BLACK]*10
        colours.append(wx.Colour(255, 255, 0))
        colours.append(wx.Colour(255, 255, 0))
        colours.append(wx.Colour(255, 255, 0))
        colours.append(wx.RED)
        TACH.SetIntervalColours(colours)

        # Assign The Ticks
        ticks = [str(interval) for interval in intervals]
        TACH.SetTicks(ticks)
        # Set The Ticks/Tick Markers Colour
        TACH.SetTicksColour(wx.RED)
        # We Want To Draw 5 Secondary Ticks Between The Principal Ticks
        TACH.SetNumberOfSecondaryTicks(1)

        # Set The Font For The Ticks Markers
        TACH.SetTicksFont(wx.Font(7, wx.SWISS, wx.NORMAL, wx.NORMAL))

        # Set The Text In The Center Of SpeedMeter
        TACH.SetMiddleText("RPM")
        # Assign The Colour To The Center Text
        TACH.SetMiddleTextColour(wx.WHITE)
        # Assign A Font To The Center Text
        TACH.SetMiddleTextFont(wx.Font(8, wx.SWISS, wx.NORMAL, wx.BOLD))

        # Set The Colour For The Hand Indicator
        TACH.SetHandColour(wx.Colour(255, 255, 0))

        # Set The Colour For The Gauge Background
        TACH.SetSpeedBackground(wx.BLACK)

        # Do Not Draw The External (CONTAINER) Arc
        TACH.DrawExternalArc(False)
        TACH.SetSpeedValue(0)
#ODOMETER--------------------------------------------------------------------------------------------------------------
        DECIMAL1=wx.gizmos.LEDNumberCtrl(ODOpanel,-1, pos=wx.Point(0, 0), size=wx.Size(125, 30), style=wx.gizmos.LED_ALIGN_LEFT)
        DECIMAL1.SetBackgroundColour("BLACK")
        DECIMAL1.SetForegroundColour("RED")
        DECIMAL1.SetValue("192768")
#AIRTEMP---------------------------------------------------------------------------------------------------------------
        AIR = SM.SpeedMeter(AIRpanel, agwStyle=SM.SM_DRAW_HAND|SM.SM_DRAW_PARTIAL_SECTORS,bufferedstyle=SM.SM_BUFFERED_DC, pos=(0, 0), size=(80, 80))

        # Air Temp Control
        AIR.SetAngleRange(5,7.25)

        intervals = range(0, 5)
        AIR.SetIntervals(intervals)

        colours = [wx.BLACK]*3
        colours.append(wx.RED)
        AIR.SetIntervalColours(colours)

        ticks = ["140", "", "210", "", "280"]
        AIR.SetTicks(ticks)
        AIR.SetTicksColour(wx.RED)
        AIR.SetHandColour(wx.Colour(255, 255, 0))
        AIR.SetSpeedBackground(wx.BLACK)        
        #AIR.SetArcColour(wx.RED)
        AIR.SetTicksFont(wx.Font(7, wx.SWISS, wx.NORMAL, wx.NORMAL))
        AIR.SetSpeedValue(0)
        AIR.SetDirection("Reverse")


#OILTEMP---------------------------------------------------------------------------------------------------------------
        OIL = SM.SpeedMeter(OILpanel, agwStyle=SM.SM_DRAW_HAND|SM.SM_DRAW_PARTIAL_SECTORS,bufferedstyle=SM.SM_BUFFERED_DC, pos=(0, 0), size=(80, 80))

        # Air Temp Control
        OIL.SetAngleRange(5,7.25)

        intervals = range(0, 5)
        OIL.SetIntervals(intervals)

        colours = [wx.BLACK]*3
        colours.append(wx.RED)
        OIL.SetIntervalColours(colours)

        ticks = ["140", "", "210", "", "280"]
        OIL.SetTicks(ticks)
        OIL.SetTicksColour(wx.RED)

        OIL.SetHandColour(wx.Colour(255, 255, 0))
        OIL.SetSpeedBackground(wx.BLACK)        
        #OIL.SetArcColour(wx.RED)
        OIL.SetTicksFont(wx.Font(7, wx.SWISS, wx.NORMAL, wx.NORMAL))        
        OIL.SetSpeedValue(0)
        OIL.SetDirection("Reverse")
#FUELLEVEL---------------------------------------------------------------------------------------------------------------
        FUEL = SM.SpeedMeter(FUELpanel, agwStyle=SM.SM_DRAW_HAND|SM.SM_DRAW_SECTORS,bufferedstyle=SM.SM_BUFFERED_DC, pos=(0, 0), size=(80, 80))

        # Air Temp Control
        FUEL.SetAngleRange(5,7.25)

        intervals = range(0, 5)
        FUEL.SetIntervals(intervals)

        colours = [wx.BLACK]*4
        FUEL.SetIntervalColours(colours)

        ticks = ["E", "", "1/2", "", "F"]
        FUEL.SetTicks(ticks)
        FUEL.SetTicksColour(wx.RED)

        FUEL.SetHandColour(wx.Colour(255, 255, 0))
        FUEL.SetSpeedBackground(wx.BLACK)        
        #FUEL.SetArcColour(wx.RED)
        FUEL.SetTicksFont(wx.Font(7, wx.SWISS, wx.NORMAL, wx.NORMAL))        
        FUEL.SetSpeedValue(0)
        FUEL.SetDirection("Reverse")

#ENGTEMP---------------------------------------------------------------------------------------------------------------
        ENG = SM.SpeedMeter(ENGpanel, agwStyle=SM.SM_DRAW_HAND|SM.SM_DRAW_PARTIAL_SECTORS,bufferedstyle=SM.SM_BUFFERED_DC, pos=(0, 0), size=(80, 80))

        # Air Temp Control
        ENG.SetAngleRange(5,7.25)

        intervals = range(0, 5)
        ENG.SetIntervals(intervals)

        colours = [wx.BLACK]*3
        colours.append(wx.RED)
        ENG.SetIntervalColours(colours)

        ticks = ["140", "", "210", "", "280"]
        ENG.SetTicks(ticks)
        ENG.SetTicksColour(wx.RED)

        ENG.SetHandColour(wx.Colour(255, 255, 0))
        ENG.SetSpeedBackground(wx.BLACK)        
        #ENG.SetArcColour(wx.RED)
        ENG.SetTicksFont(wx.Font(7, wx.SWISS, wx.NORMAL, wx.NORMAL))        
        ENG.SetSpeedValue(0)
        ENG.SetDirection("Reverse")
#INDICATORS-------------------------------------------------------------------------------------------------------------

        CEL_ON = wx.Image('Images/cel_on.gif', wx.BITMAP_TYPE_ANY)
        CEL_OFF = wx.Image('Images/cel_off.ico', wx.BITMAP_TYPE_ANY)
        LOWFUEL_ON = wx.Image('Images/fuel_on.gif', wx.BITMAP_TYPE_ANY)
        LOWFUEL_OFF = wx.Image('Images/fuel_off.ico', wx.BITMAP_TYPE_ANY)
        BATTERY_ON = wx.Image('Images/battery_on.gif', wx.BITMAP_TYPE_ANY)
        BATTERY_OFF = wx.Image('Images/battery_off.ico', wx.BITMAP_TYPE_ANY)
        SEATBELT_ON = wx.Image('Images/seatbelt_on.gif', wx.BITMAP_TYPE_ANY)
        SEATBELT_OFF = wx.Image('Images/seatbelt_off.ico', wx.BITMAP_TYPE_ANY)
        UPSHIFT_ON = wx.Image('Images/upshift_on.gif', wx.BITMAP_TYPE_ANY)
        UPSHIFT_OFF = wx.Image('Images/upshift_off.ico', wx.BITMAP_TYPE_ANY)
        TEMP_ON = wx.Image('Images/temp_on.gif', wx.BITMAP_TYPE_ANY)
        TEMP_OFF = wx.Image('Images/temp_off.ico', wx.BITMAP_TYPE_ANY)
        PBRAKE_ON = wx.Image('Images/brake_on.gif', wx.BITMAP_TYPE_ANY)
        PBRAKE_OFF = wx.Image('Images/brake_off.ico', wx.BITMAP_TYPE_ANY)
        LIGHTS_ON = wx.Image('Images/light_on.gif', wx.BITMAP_TYPE_ANY)
        LIGHTS_OFF = wx.Image('Images/light_off.ico', wx.BITMAP_TYPE_ANY)

        imageBitmap = wx.StaticBitmap(INDpanel1, wx.ID_ANY, wx.BitmapFromImage(CEL_ON))
        imageBitmap = wx.StaticBitmap(INDpanel2, wx.ID_ANY, wx.BitmapFromImage(LOWFUEL_ON))
        imageBitmap = wx.StaticBitmap(INDpanel3, wx.ID_ANY, wx.BitmapFromImage(BATTERY_ON))
        imageBitmap = wx.StaticBitmap(INDpanel4, wx.ID_ANY, wx.BitmapFromImage(SEATBELT_ON))
        imageBitmap = wx.StaticBitmap(INDpanel5, wx.ID_ANY, wx.BitmapFromImage(UPSHIFT_ON))
        imageBitmap = wx.StaticBitmap(INDpanel6, wx.ID_ANY, wx.BitmapFromImage(TEMP_ON))
        imageBitmap = wx.StaticBitmap(INDpanel7, wx.ID_ANY, wx.BitmapFromImage(PBRAKE_ON))
        imageBitmap = wx.StaticBitmap(INDpanel8, wx.ID_ANY, wx.BitmapFromImage(LIGHTS_ON))


        imageBitmap = wx.StaticBitmap(INDpanel1, wx.ID_ANY, wx.BitmapFromImage(CEL_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel2, wx.ID_ANY, wx.BitmapFromImage(LOWFUEL_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel3, wx.ID_ANY, wx.BitmapFromImage(BATTERY_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel4, wx.ID_ANY, wx.BitmapFromImage(SEATBELT_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel5, wx.ID_ANY, wx.BitmapFromImage(UPSHIFT_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel6, wx.ID_ANY, wx.BitmapFromImage(TEMP_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel7, wx.ID_ANY, wx.BitmapFromImage(PBRAKE_OFF))
        imageBitmap = wx.StaticBitmap(INDpanel8, wx.ID_ANY, wx.BitmapFromImage(LIGHTS_OFF))







APP=wx.App(0)

frame=MyFrame(None)
APP.SetTopWindow(frame)
frame.Show()
APP.MainLoop()

why would you down vote this question? is its legitimacy questioned? — t gillespie, Mar 08 '15 at 23:01

score 1 · Answer 1 · edited May 23 '17 at 12:12

You can run your serial code in a thread that is started by your GUI. I suspect the serial code will block the UI's mainloop otherwise. If you go that route, then you'll need to use a thread-safe method to communicate your results back to wxPython such as wx.CallAfter or wx.PostEvent. Here are a couple of links that show how to use threads in wxPython:

Whatever method you call to update your UI, you can use to also update your SQLite database. I would use this method to insert whatever data you receive from the serial code. You can add a different method to your UI's __init__ that checks if the database is created and if not, creates it for you.

The following article should help you in getting SQLite integration into your wx app:

http://www.blog.pythonlibrary.org/2011/11/10/wxpython-and-sqlalchemy-an-intro-to-mvc-and-crud/

You can query from the database and write to it within the limits of the SQLite spec. I'm not sure why you'd want to read and write to an open file, but I found this link that might help:

Beginner Python: Reading and writing to the same file

I appreciate the response, the information on threads what more than likely exactly what i was looking for, i will try out some of that code and see if it works, — t gillespie, Mar 09 '15 at 21:09

Combining three scripts into one

1 Answers1