Loading a directory full of decoder modules

Question

[TL;DR] Thanks to Alexi - I have a working solution below

I'm trying to figure out Python Classes and inheritence so that I can create a protocol decoder package.

The package needs to work as follows:

• I get a directory full of data files.
• My python code inspects each file, and determines a protocol id number.
• Based on the protocol_id I can calculate (yes, calculate) the file name of the specific decoder.
• If the decoder filename does not exist, I choose some default name
• Need help here: I have the filename in a variable how do I load it?

I've done this before in C code, with Shared Objects, I'm trying create something similar using python.

Part #1 Pseudo code is like this:

def GetDecoderViaId( idnumber ):
    filename = "all_decoders/protocol_%04x.py" % idnumber

    # Pseudo code use default if special is not found
    if (! os.access(filename,os.R_OK)):
        filename = "all_decoders/protocol_DEFAULT.py"

    # Question #1 How do I do this
    # Where each "filename" is a python class.
    theclass = import_by_filename( filename )
    return theclass

Part #2 - The decoders need to come from a common base class. I have done simple classes, but what does not work is when I try to load them using various techniques I have found via google. I can load something, I just can't seem to call the decoder.

Question#2 How do I do that?

My Decoder Base Class looks like this:

 # Simple base class
 def DecoderBase(object):
    def __init__(id,name,compatlist):
       self.id = id
       self.name = name
       self.compatiblelist = compatlist

    def Decode( bytes, byte0, byteN ):
       """ Default is just a hex dump """
       HexDump( bytes, byte0, byteN )

    def GetName(self):
      return self.name

    def GetId(self):
       return self.id

    def IsCompatible(id):
       return id in self.compatiblelist

Here are two example derived classes, they would be in separate files.

# example: "all_decoders/protocol_1234.py"
class DogDecoder( DecoderBase ):
    def __init__():
        # pass important things to base constructor
        super( DecoderBase, self ).__init__("dog","Snoopy", ["beagle", "peanuts" ])

    def Decode( self, byte0, byteN ):
        Snoopy specific code goes here

# Example: "all_decoders/protocol_4321.py"
class CatDecoder( DecoderBase ):
    def __init__():
        # pass important things to base constructor
        super( DecoderBase, self ).__init__("cat","Garfield", ["fat-cat", "cartoon-cat" ])

    def Decode( self, byte0, byteN ):
        Garfield specific code goes here

Can somebody point me to some examples of how to (1) load modules as above and (2) they need to be derived classes and (3) how do I call these things?

[Note: Some edits, typos noticed after I clicked SUBMIT]

Answer 1

1. To import a module given its path, see https://docs.python.org/2/library/importlib.html .

However:

1a. a module's path is not ``foo/bar.py'`: it's `foo.bar`.  So:
      1aX first, drop the trailing `.py`
      1aY then, change slashes into dots
    both before passing the resulting string to 
   `importlib.import_module`

1b. when you import this way, you get a **module**, **not** a class!
    I recommend you just name each class `Decoder` within its module
    so you can just access it that way

1c. to make directory `foo` into a Python package, the directory
    **must** contain a file `__init__.py` (which can be empty).

There's no need for each Decoder class to be derived from the same base class; if you want it to be, for your convenience, put the base class in e.g foo/base.py, have each of the specific decoder modules start with import .base, and derive each Decoder class from base.DecoderBase.

So your code would be something like:

import importlib

def GetDecoderViaId( idnumber ):
    filename = "all_decoders/protocol_%04x.py" % idnumber

    # Pseudo code use default if special is not found
    if (!os.access(filename, os.R_OK)):
        filename = "all_decoders/protocol_DEFAULT.py"

    module_path = filename[:-3].replace('/', '.')
    themodule = importlib.import_module(module_path)
    theclass = themodule.Decoder
    return theclass

Remember to have an empty all_decoders/__init__.py, an all_decoders.base.py defining a class DecoderBase, and e.g

# in file: "all_decoders/protocol_1234.py"
import .base
class Decoder( base.DecoderBase ):
    def __init__():
        # pass important things to base constructor
        super( DecoderBase, self ).__init__("dog","Snoopy", ["beagle", "peanuts" ]

    def Decode( self, byte0, byteN ):
        Snoopy specific code goes here

# in file: "all_decoders/protocol_4321.py"
import .base
class Decoder( DecoderBase ):
    def __init__():
        # pass important things to base constructor
        super( DecoderBase, self ).__init__("cat","Garfield", ["fat-cat", "cartoon-cat" ]

    def Decode( self, byte0, byteN ):
        Garfield specific code goes here

The "how to call" is now trivial, e.g in the same Python module as GetDecoderViaId...:

theclass = GetDecoderViaId(theid)
aninstance = theclass(whatever, init, args, are, needed)
decoded = aninstance.Decode(byte0, byteN)

Answer 2

working solution: Grazie very much Alexi

File 1: example/test.py

import importlib

def GetDecoder(name,xtra):
    theclass = importlib.import_module( 'Decoders.' + name )
    return theclass.Decoder(xtra)

sh = GetDecoder('snoopy', 'hockey');
sp = GetDecoder('snoopy', 'pilot');
g = GetDecoder('garfield', None );

print( "Snoopy Hocky says: %s" % sh.Decode( 3, 4 ) )
# This has an extra method
sh.Bark()

print( "SnoopyPilot says: %s" % sp.Decode( 5, 9 ) )
# And so does this one
sh.Bark()

print("Garfield says: %s " % g.Decode( 1,3 ))
# A different method
g.Purr()

File 2: example/Decoders/init.py

#
# This file is empty
#

File 3: example/Decoders/base.py

class DecoderBase(object):
    def __init__( self, n, idval ):
         self.name = n
         self.idcode = idval
         print( "Base created: %s %d " % (self.name, self.idcode));

    def GetName(self):
        return self.name

    def GetId(self):
        return self.idcode

    def Decode(self,b0,bn):
        msg = ("Decode from byte %d to %d" % (b0, bn ))
        return msg

File 4: example/Decoders/snoopy.py

from base import DecoderBase

class Decoder( DecoderBase ):
    def __init__( self, varient ):
        self.costume = varient
        super( Decoder, self ).__init__( "Snoopy-" + varient, 0x444f47 );

    # Override the decoder
    def Decode( self, b0, b1 ):
        # but only if snoopy is playing hockey
        if self.costume == 'hockey':
            return ("Hockey decode: %d to %d" % (b0, b1))
        else:
            return super(Decoder,self).Decode( b0, b1 )

    def Bark(self):
        print "Barking(%s) ..." % self.costume 

File 5: example/Decoders/garfield.py

from  base import DecoderBase

class Decoder( DecoderBase ):
    def __init__(self, varient ):
        # Unlike snoopy decoder, garfield does not have any varients
        super( Decoder, self ).__init__( "Garfield", 0x434154 );

    # Override the decoder
    def Decode( self, b0, b1 ):
        return ("Garfield lasagna %d to %d" % (b0, b1))

    def Purr( self ):
        print "Purring like a cat.."

END