-2

I've read this answer: How to import a module given the full path? and I am still a little unsure about what the best way to go about solving my problem is and I'd like to see what the community has to say. I learn best by example, sorry if I'm being thick here!

I have a directory tree full of arbitrary directories and files, I am crawling it to find directories as categories and then the .py files inside are each a module. I have no problem crawling the directory tree and finding the files, the problem is importing them properly and using them later. I'm using Python 3.4.

I should mention that I have control over each of the files I want to import and can specify a format or name be available for parsing at import time. edit An example use case would be something like this: > use /parsers/foo then the module is loaded, and i issue parsers/foo > set filename bar.xml and then issue the parsers/foo > run command and the data is parsed and saved into a database.

I guess my questions are:

  • How would you set up this import loop,
  • How would you use them at a later time

This is a CLI interface that I'm building with prompt_toolkit. The idea is to issue a command like "use category/module" kinda like Metasploit or other similar interfaces.

Here is the directory/module crawling code: def _load_modules(self, mod_path): self.modules = {} # walk through the module tree for dirpath, subdirs, filenames in os.walk(mod_path): if len(filenames) > 0: for filename in [f for f in filenames if f.endswith('.py')]: # extract the category from the module directory cat = (re.search('/modules/([^/]*)', dirpath).group(1)) self.modules.setdefault(cat,list()).append( (dirpath, filename) ) Thanks for your help.

Community
  • 1
  • 1
Blark
  • 307
  • 1
  • 3
  • 15
  • Please show the code you're using to crawl the directory tree and find files. – Kevin Dec 28 '15 at 14:15
  • @Kevin done. Thanks! – Blark Dec 28 '15 at 14:24
  • I'm guessing you're effectively asking "I can use importlib/imp to get the module object from the file, but how do I make it accessible to my CLI interface?". But this is tricky to answer without knowing how your CLI interface is implemented. If you're using some kind of CLI-builder-helper-API, there may be a parameter you can use to supply modules. If you're just running `eval` directly on user input, maybe you need to inject the modules into your current scope. It's hard to say. – Kevin Dec 28 '15 at 14:37
  • @Kevin I don't need a full walkthrough, I can figure it out. I'd just like to see how someone would manage importing a bunch of modules and then maybe a code example of how to call one later. – Blark Dec 28 '15 at 14:40

1 Answers1

0

You didn't mention how your CLI interface works, so I'm assuming you did the simplest thing and are simply eval-ing whatever the user gives you. I don't endorse this approach, but Occam's Razor and all that.

I'd just like to see how someone would manage importing a bunch of modules...

module_names = ["math", "string", "collections"]
modules = {}
for name in module_names:
    modules[name] = __import__(name)

...and then maybe a code example of how to call one later.

command = "collections.Counter(string.uppercase)"
result = eval(command, modules)
print "The result of your command is", repr(result)
Kevin
  • 74,910
  • 12
  • 133
  • 166
  • Hi Kevin, thanks again. I am wondering if you could help me understand why I would just use `__import__` instead of one of the suggestions listed in the answer I posted. Also, FYI I am using prompt_toolkit for the CLI I did mention that in the original post (https://github.com/jonathanslenders/python-prompt-toolkit) – Blark Dec 28 '15 at 14:57
  • If I understand correctly, `__import__` is more limited than the suggestions in your linked answer. This is because `__import__` can only import modules on the module search path, while `importlib` and `imp` can import any file from anywhere on your hard drive. If this is important to you, you'd need to change my code to do `modules[name] = imp.load_source(...)`. (or `SourceFileLoader(...)` or whatever solution you're using) – Kevin Dec 28 '15 at 15:01
  • The modules are custom and may not be in the python search path so it seems I'll have to use another method. I envision that one would be an XML parser for example, and I would issue the command `use parser\foo` and then I would set options such as `set filename bar.xml` and then `run` and the `run` command would execute the Module passing it the necessary variables. I hope that helps a bit more to describe my use case. Thanks again for the help. – Blark Dec 28 '15 at 15:07