Mounting Virtual Directory, or Running Code in Memory for python 3

Question

I'm making a program that generates code randomly in batches, and given an input, tests to see if that code generates the desired output. I'm wanting to make the batches of generated code pretty large, say 10,000 randomly generated files per batch, so I want to work within memory instead of messing with writing them to the disk. I also want the compiler to do all the work for me, where the file, once created, is imported as a module, and then generated function in that module is run, and the result is tested against the desired output.

I found that Python has tempfile.SpooledTemporaryFile(), which will allow me to create and manipulate a file-like object, but once created, I want to access it as if it were a python script. However, since the file is in RAM, and is deleted once closed, I'm struggling to find a way to do this. What I'm wanting is something like this:

import os, tempfile, importlib, desiredInAndOut, codeGenerator
generatedCodeAsAString = codeGenerator.generateCode()
cwd = os.getcwd()
successfulCode = []
batchSize = int(input(Enter desired batch size:))
runs = 0
while runs < batchSize:
    with tempfile.SpooledTemporaryFile(10240, 'w+b', None, None, '\n', '.py', 'tmp', cwd) as stf:
        stf.write(generatedCodeAsAsAString.encode())
        import tmp #import the newly generated file (that maybe has a 'shortcut' in the cwd? Somehow?)
        importlib.invalidate_caches() #make sure the compiler knows that there is a new module for the first time this runs
        importlib.reload(tmp) #if the compiler gets smart and re-uses the module from the first import/first iteration of this loop, make sure it reloads/updates it to match the new file that was just created
        out = tmp.generatedFunction(desiredInAndOut.GIVEN_INPUT)
        if out == desiredInAndOut.DESIRED_OUTPUT:
            successfulCode.append(generatedCodeAsAString+'\n'+'#BATCHFILE NO '+str(runs))
        runs += 1
print(successfulCode)

Despite my efforts, the tmp file is in no way linked to the current working directory, so the import above won't work. I need to find a way to feed the memory address of that temporary file to the import statement where it would expect a standard '/foo/bar/tmp.py', and make sure it interprets it as a file.

Mounting a virtual drive directly on the computer is not an option because of permission difficulties. If you know of an in-python way to do something like that without admin access to the computer, I'm all ears. I'm currently exploring the functionality of the built-in function exec - not sure how it will respond outside of the shell. I'm hoping it will work, but we'll see.

Is there a correct/better way to do what I'm trying? More specifically, is there a way to point the compiler in the direction of the file existing in the memory so it can be manipulated as if it were an existing file and/or run? Or a python formatted string, it doesn't necessarily need to be a file if I can run it in string form.

score 0 · Accepted Answer · answered Dec 07 '17 at 01:36

https://stackoverflow.com/a/2849077/6047611 (If manipulating as SpooledTemporaryFile)

AND

https://stackoverflow.com/a/3906309/6047611

Provided me with the information necessary to make this solution using the built-in exec() function:

import sys
from io import StringIO

def main():
    GIVEN_INPUT = input('Enter input as an integer:')
    code = "def main():\n\tinput = "+GIVEN_INPUT+"\n\tprint(input + 3)\nmain()"
    #this code would be randomly generated elsewhere. As is, simply adds 3 to GIVEN_INPUT

    old_stdout = sys.stdout
    redirected_output = StringIO()
    sys.stdout = redirected_output

    exec(code)#, sys._getframe(1).f_globals, sys._getframe(1).f_locals)

    sys.stdout = old_stdout 

    print('function completed')
    OUTPUT = redirected_output.getvalue().strip()
    print(OUTPUT+' is the result of:\n'+ code)

main()

In summary, exec() can't return a value that I could see, but it can print to sys.stdout, so instead of 'returning' an outcome, it prints an outcome, and I'm able to get what it prints by switching out sys.stdout for a new StringIO while the generated code runs, switching it back to normal once it's finished, and then getting the contents of that StringIO object. If it needed to be in file form, that could easily be done with tempfile.SpooledTemporaryFile, see my first link above, but I'm already generating the code as a string so I have no need to convert it into file form first to run it. I'm hoping exec() doesn't have a max characters limit, that could cause significant problems...

No import statements needed to open/run the generated code, and can be run many times without messing with a mounted virtual directory, or even file objects in general.

Mounting Virtual Directory, or Running Code in Memory for python 3

1 Answers1