11

Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other protocol? If so, how? If not, why?

I guess it's about making Python use more the one protocol(reading the filesystem) and enabling it to use others as well. Yes I agree it would be many folds slower, but some optimization and larger future bandwidths would certainly balance it out.

E.g.:

import site

site.addsitedir("https://bitbucket.org/zzzeek/sqlalchemy/src/e8167548429b9d4937caaa09740ffe9bdab1ef61/lib")

import sqlalchemy
import sqlalchemy.engine
Marcin
  • 48,559
  • 18
  • 128
  • 201
Bleeding Fingers
  • 6,993
  • 7
  • 46
  • 74
  • I suppose you could add the URI to `sys.path`, but I think you'll need to mount it to a local mount point. I haven't heard of imports working over HTTP, though I have little experience with this notion – inspectorG4dget Sep 11 '13 at 16:59
  • You can download the .py file to any directory, then import it. – 7stud Sep 11 '13 at 17:03
  • 2
    @7stud duh. The point I'm trying to emphasis here is that whether the file is in a directory or over the internet, it finally ends up in the memory so why the extra logistics of downloading and then importing it, when the only difference, on the surface, is "protocol". – Bleeding Fingers Sep 11 '13 at 17:30
  • @inspectorG4dget what would be the absolute URI of the above stated URL because, as is the code above doesn't work. Mounting is an obvious solution but wouldn't be so portable and practical given multiple sources. Imagine not having to locally install 3rd party packages and simply importing them over the internet! – Bleeding Fingers Sep 11 '13 at 17:38
  • It's worth pointing out that JavaScript has done this in the browser for decades, and Deno, the Javascript runtime made by the original creator of Node.js, transparently replicates this behaviour using the ES6+ `import`syntax: `import { someExportedProperty } from 'https://any.website'`. Deno does as you describe: it just downloads the remote dependencies into a predetermined permanent cache (which you can set to a local directory), then executes the main script. – iono Oct 06 '19 at 10:58

4 Answers4

5

Another version,

I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.

This is the result:

import os
import imp
import requests

def import_cdn(uri, name=None):
    if not name:
        name = os.path.basename(uri).lower().rstrip('.py')

    r = requests.get(uri)
    r.raise_for_status()

    codeobj = compile(r.content, uri, 'exec')
    module = imp.new_module(name)
    exec (codeobj, module.__dict__)
    return module

Usage:

redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")

# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')
  • Tip - place the import_cdn function in a common library, this way you could re-use this small function
  • Bear in mind It will fail when no connectivity to that file over http
Community
  • 1
  • 1
Jossef Harush Kadouri
  • 32,361
  • 10
  • 130
  • 129
  • I don't see this as a simplification - you just wove retrieval in with the module loading code. – Marcin Dec 28 '15 at 14:05
  • 1
    The library `imp` is deprecated since version 3.4. For newer versions `import imp` with `import types` and `imp.new_module(name)` with `types.ModuleType(name)` – Giancarlo Sportelli May 29 '20 at 09:29
4

In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.

To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.

I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)

In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.

import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
    #if loading from the internet, you'd probably want to parse the uri, 
    # and use the last part as the modulename. It'll come up in tracebacks
    # and the like.
    if not modname: modname = 'newmodulename'
    #must be exec mode
    # every module needs a source to be identified, can be any value
    # but if loading from the internet, you'd use the URI
    codeobj = compile(modulesource, sourcestr, 'exec')
    newmodule = imp.new_module(modname)
    exec(codeobj,newmodule.__dict__)
    return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)

At this point newmodule is already a module object in scope, so you don't need to import it or anything. 

modulesource = '''
a = 'foo'
def myfun(astr):
    return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))

Ideone here: http://ideone.com/dXGziO

Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).

Marcin
  • 48,559
  • 18
  • 128
  • 201
1

This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.

Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.

glglgl
  • 89,107
  • 13
  • 149
  • 217
  • It is fairly straightforward. I think the trickiest bit is that which my answer deals with; and they have an example there which covers similar ground (although it is more complex). – Marcin Sep 13 '13 at 18:21
0

As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport. It uses a custom finder/loader object to locate resources using HTTP/S.

Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.

@Marcin's answer contains a good portion of the code of the httpimport's loader class.

operatorequals
  • 381
  • 2
  • 5