Best way to add dynamic code to a perl application

Question

I know specific instances of this question have been answered before:

• How can I dynamically include Perl modules without using eval?
• How do I use a Perl package known only in runtime?

There are also good answers at Perl Monks:

• Writing a Perl module that dynamically loads other modules.
• Creating subroutines on the fly

But I would like a robust way to add functionallity to a Perl application that will be:

1. Efficient: if the code is not needed it should not be compiled.
2. Easy to debug: error reporting if something goes wrong at the dynamic code, should point at the right place at the dynamic code.
3. Easy to extend: adding new code should be as easy as adding a new file or directory+file.
4. Easy to invoke: the main application should be able to use an "add on" without much trouble. An efficient mechanism to check if the "add on" has already been loaded and if not load it, would be a plus.

To illustrate the point, here are some examples that would benefit from a good solution:

• A set of scripts that move data from different applications. For instance, moving data from OpenCart to Prestashop, where each entity in the data model has a specific "add on" that deals with the input or output; then an intermediate data model takes care of the transformation of the data. This could be used to move data in any direction or even between different versions of the same ecommerce.

• A web application that needs to render different types of HTML in different places. Each "module" knows how to handle a certain information and accepts parameters to do it. A module outputs HTML, another a list of documents, another a document, another a banner, and so on.

Here are some examples that I have used and that work.

Load a function at run time and output the possible compile errors:

eval `cat $file_with_function`;
if( $@ ) {
  print STDERR $@, "\n";
  die "Errors at file $file_with_function\n";
}

Or more robust using File::Slurp:

eval read_file("$file_with_function", binmode => ':utf8');

Check that a certain function has been defined:

if( !defined &myfunction ) {
  die "myfunction is not defined\n";
}

The function may be called from there on. This is fine with one function, but not for many.

If the function is put in a module:

require $file_with_function; # needs the ".pm" extension, i.e. addon/func.pm
$name_of_module->import();   # need to know the module name, i.e. Addon::Func

$name_of_module->myfunction(...);

Where the require may be protected inside an eval and then use $@ as before.

With Module::Load:

load $name_of_module;

Followed by the import and used in the same way. Security should not be a concern as it may be assumed that the dynamic code comes from a trusted place. Are there better ways? Which way would be considered good practice?

In case it helps, I will be using the solution (among other places, but not exclusively) within the Dancer framework.

EDIT: Given the comments, I add some more info. All cases that I have in mind have in common:

1. There is more than one dynamic piece of code. Probably many to start with.
2. Each bit of code has the same interface.

Answer 1

Given the comments and the lack of responses, I have done some research to answer my own question. Comments or other answers are welcome!

Dynamic code

By dynamic code I mean code that is evaluated at run-time. In general, I consider better to compile an application so that you have all the error checking the Perl compiler can offer before starting to execute. Added to use strict and use warnings, you can catch many common mistakes that way. So why using dynamic code at all? These are the reasons I consider:

1. An application performs many different actions that are chosen depending on the context of execution. For instance, an application extracts certain properties from a file. The way to extract them depends on the file type and we want to deal with many file types, but we do not want to change the application for each new file type we add. We also want the application to start quickly.
2. An application needs to be expanded on the fly in a way that does not require the application to restart.
3. We have a large application that contains a number of features. When we deploy the application, we do not want to provide all the possible features all the time, maybe because we licence them separately, maybe because not all of them are able to run under all platforms. By throwing in only the files with the features we want, we have a distribution that does not require changing any code or config files.

How do we do it?

Given the possibilities that Perl offers, solutions to adding dynamic code come in two flavors: using eval and using require. Then there are modules that may help do things in an easier or more maintainable way.

The quick and dirty way

The eval way uses the form eval EXPR to compile a piece of Perl code at run-time. The expression could be a string but I suggest putting the code in a file and grouping other similar files in a convenient place. Then, if possible using File::Slurp:

eval read_file("$file_with_code", binmode => ':utf8');
if( $@ ) {
  die "$file_with_code: error $@\n";
}
if( !defined &myfunction ) {
  die "myfunction is not defined at $file_with_code\n";
}

Specifying the character set to read_file makes sure that the file will be interpreted correctly. It is also good to check that the compilation was correct and that the function we expect was defined. So in $file_with_code, we will have:

sub myfunction(...) {
  # Do whatever; maybe return something
}

Then you may invoke the function normally. The function will be a different one depending on which file was loaded. Simple and dynamic.

The modular way (recommended)

The way I would do it with maintainability in mind would be using require. Unlike use, that is evaluated at compile-time, require may be used to load a module at run-time. Out of the various ways to invoke require, I would go for:

my $mymodule = 'MyCompany::MyModule'; # The module name ends up in $mymodule
require $mymodule;

Also unlike use, require will load the module but will not execute import. So we may use any functions inside the module and those function names will not polute the calling namespace. To access the function we will need to use:

$mymodule->myfunction($a, $b);

See below as to how the arguments get passed. This way of invoking a function will add an argument before $a and $b that is usually named $self. You may ignore it if you don´t know anything about object orientation.

As require will try to load a module and the module may not exist or it may not compile, to catch the error it will be better to use:

eval "require $mymodule";

Then $@ may be used to check for an error in the loading+compiling process. We may also check that the function has been defined with:

if( $mymodule->can('myfunction') ) {
  die "myfunction is not defined at module $mymodule\n";
}

In this case we will need to create a directory for the modules and a file with the .pm extension for each one:

MyCompany
  MyModule.pm

Inside MyModule.pm we will have:

package MyCompany::MyModule;

sub myfunction {
  my ($self, $a, $b);

  # Do whatever; maybe return something
  # $self will be 'MyCompany::MyModule'
}

1;

The package bit is essential and will make sure that whatever definitions we put inside will be at the MyCompany::MyModule namespace. The 1; at the end will tell require that the module initialization was correct.

In case we wanted to implement the module by using other libraries that could polute the caller namespace, we could use the namespace::clean module. This module will make sure the caller does not get any additions to the namespace coming from the module we are defining. It is used in this way:

package MyCompany::MyModule;

# Definitions by these modules will not be available to the code doing the require
use Library1 qw(def1 def2);
use Library2 qw(def3 def4);
...

# Private functions go here and will not be visible from the code doing the require
sub private_function1 {
  ...
}
...

use namespace::clean;

# myfunction will be available
sub myfunction {
  # Do whatever; maybe return something
}
...

1;

What happens if we include a module more than once?

The short answer is nothing. Perl keeps track of which modules have been loaded and from where using the %INC variable. Both use and require will not load a library twice. use will add any exported names to the callers namespace. require will not do that either. In case you want to check that a module has been loaded already, you could use %INC or better yet, you could use module::loaded which is part of the core in modern Perl versions:

use Module::Loaded;

if( !is_loaded( $mymodule ) {
  eval "require $mymodule" );
  ...
}

How do I make sure Perl finds my module files?

For use and require Perl uses the @INC variable to define the list of directories that will be used to look for libraries. Adding a new directory to it may be achieved (among other ways) by adding it to the PERL5LIB environment variable or by using:

use lib '/the/path/to/my/libs';

Helper libraries

I have found some libraries that may be used to make the code that uses the dynamic mechanism more maintainable. They are:

• The if module: will load a module or not depending on a condition: use if CONDITION, MODULE => ARGUMENTS;. May also be used to unload a module.
• Module::Load::Conditional: will not die on you while trying to load a module and may also be used to check the module version or its dependencies. It is also able to load a list of modules all at once even checking their versions before doing so.

Taken from the Module::Load::Conditional documentation:

use Module::Load::Conditional qw(can_load);

my $use_list = {
        CPANPLUS        => 0.05,
        LWP             => 5.60,
        'Test::More'    => undef,
};

print can_load( modules => $use_list )
        ? 'all modules loaded successfully'
        : 'failed to load required modules';