19

In the interest of not rewriting an open source library, I want to treat a string of text as a file in python 3.

Suppose I have the file contents as a string:

not_a_file = 'there is a lot of blah blah in this so-called file'

I want to treat this variable, i.e. the contents of a file, as a path-like object that way I can use it in python's open() function.

Here is a simple example which shows my dilemma:

not_a_file = 'there is a lot of blah blah in this so-called file'
file_ptr = open(not_a_file, 'r')

Clearly the example doesn't work because not_a_file is not a path-like object. I don't want to write a file nor create any temporary directories for portability purposes.

With that said, I need is to solve this mystery:

not_a_file = 'there is a lot of blah blah in this so-called file'
... Something goes here ... 
file_ptr = open(also_not_a_file, 'r') 

What I've Tried So Far

  1. I've looked into StringIO and tried using that as a path-like object and no dice: import StringIO output = StringIO.StringIO() output.write('First line.\n') file_ptr = open(output,'r') Well this doesn't work because StringIO isn't a path-like object.

  2. I've tried tempfile in a similar fashion with no success. import tempfile tp = tempfile.TemporaryFile() tp.write(b'there is a lot of blah blah in this so-called file') open(tp,'r')

  3. Finally I tried mmap to see if I can write the string into memory and then open the memory pointer with open with no success.

Any help is appreciated! :-)

Edit 1: What I'm thinking of to possibly solve problem

So pathlib.PurePath can work with open() if PurePath is initialized to a file. Perhaps I can create an instance of a class that inherits PurePath and when read by open(), it reads my string. Let me give an example:

from pathlib import PurePath

not_a_file = 'there is a lot of blah blah in this so-called file'
class Magic(PurePath):
    def __init__(self, string_input):
        self.file_content = string_input
        PurePath.__init__(self, 'Something magical goes here')
    #some more magic happens in this class

also_not_a_file = Magic(not_a_file)
fp = open(also_not_a_file,'r')
print(fp.readlines()) # 'there is a lot of blah blah in this so-called file'
torrho
  • 1,823
  • 4
  • 16
  • 21
  • Why do you want to use the open command? output.write("line 1\n"); output.seek(0); file_ptr=output – clockwatcher Jun 06 '17 at 04:07
  • @clockwatcher so in a different library that I'm going to use, there is this `open` command expecting the usual file input. For my purposes and portability I need to somehow pass the string into the `open` somehow. – torrho Jun 06 '17 at 04:27
  • Is there a way to use pathlib.PurePath with a string to create an object that will read the string? `open(pathlib.PurePath('path/to/filename.ext'),'r')` is valid python. Can I incorporate `PurePath` and my string into a class which when passed into `open()` reads my string? – torrho Jun 06 '17 at 04:54
  • Does this answer your question? [How do I wrap a string in a file in Python?](https://stackoverflow.com/questions/141449/how-do-i-wrap-a-string-in-a-file-in-python) – Kermit Mar 15 '22 at 15:48

5 Answers5

28

StringIO returns an StringIO object, it's almost equivalent to the file object returned by the open statement. So basically, you can use the StringIO in place of the open statement.

# from StringIO import StringIO  # for Python 2, old
from io import StringIO
with StringIO('there is a lot of blah blah in this so-called file') as f:
    print(f.read())

Output:

there is a lot of blah blah in this so-called file
Mayou36
  • 4,613
  • 2
  • 17
  • 20
Taku
  • 31,927
  • 11
  • 74
  • 85
  • 4
    This won't suffice. I would need a path like object `also_not_a_string` to be a valid input into `open()`. This object, when read via `open()` should have the 'there is a lot of blah blah in this so-called file' string. – torrho Jun 06 '17 at 04:24
  • So you're basically wanting to create a real existing file onto storage? Since `open()` must take an openable file name/path as an argument. You *can* use tempfile, but that's just like creating a file, use it, and delete it later. Is that what you want instead? – Taku Jun 06 '17 at 04:26
  • See my revised edit of #2. That is an example of tempfile which does not work. Again, I need a path-like object when read via `open()`, yields the string 'there is a lot of blah blah in this so-called file' – torrho Jun 06 '17 at 04:36
  • Sometimes (rarely) this simply is not possible. TensorFlow is one such rare problem. You must specify a filename and cannot send in a file object/pointer. – Robert Lugg Aug 03 '21 at 06:11
10

You can create a temporary file and pass its name to open:

On Unix:

tp = tempfile.NamedTemporaryFile()
tp.write(b'there is a lot of blah blah blah in this so-called file')
tp.flush()
open(tp.name, 'r')

On Windows, you need to close the temporary file before it can be opened:

tp = tempfile.NamedTemporaryFile(delete=False)
tp.write(b'there is a lot of blah blah blah in this so-called file')
tp.close()
open(tp.name, 'r')

You then become responsible for deleting the file once you're done using it.

Community
  • 1
  • 1
user2313067
  • 593
  • 1
  • 3
  • 8
  • Did you pass `tp.name` and not `tp` to open? – user2313067 Jun 06 '17 at 05:13
  • The unix example isn't working for me. Is NamedTemporaryFile() a path-like object? I get: Traceback (most recent call last): File "", line 1, in TypeError: expected str, bytes or os.PathLike object, not _TemporaryFileWrapper – torrho Jun 06 '17 at 05:13
  • I'm still not able to read the contents of the object properly. `open(tp.name, 'r').readlines()` just gives me `''` – torrho Jun 06 '17 at 05:18
3

With what I can tell from your comments and recent edits, you want a file that can be opened using the open statement. (I'll leave my other answer be since it's the more correct approach to this type of question)

You can use tempfile to solve your problem, it basically is doing this: create your file, do stuff to your file, then delete your file upon closure.

import os
from tempfile import NamedTemporaryFile

f = NamedTemporaryFile(mode='w+', delete=False)
f.write("there is a lot of blah blah in this so-called file")
f.close()
with open(f.name, "r") as new_f:
    print(new_f.read())

os.unlink(f.name) # delete the file after
Taku
  • 31,927
  • 11
  • 74
  • 85
0

the other answers didn't work for me, but I managed to figure it out.

When using Python 3, you'll want to use io package.

    import io
    with io.StringIO("some initial text data") as f:
        # now you can do things with f as if it was an opened file.
        function_that_requires_a_Fileobject_as_argument(f)
Strandtasche
  • 108
  • 2
  • 9
0

You can overwrite what open means for a particular function - similarly to what unittest.mock.patch does. However to do so, you have to keep track of the module where your target function is located. To make sure that later on open is restored, you can take advantage of decorators or context managers:

class FileToStringIOPatch:
    def __init__(self, module, new_reader=io.StringIO, **kwargs):
        self.new_reader = new_reader
        self.globals_dict = vars(module)
        self.old_open = self.globals_dict.get('open', open)
        self.kwargs = kwargs

    def __enter__(self):  # initialize context manager - overwrite `open`
        def new_open(file, mode='r', buffering=None, encoding=None, errors=None, newline=None, closefd=True):
            return self.new_reader(file, **self.kwargs)
        self.globals_dict['open'] = new_open

    def __exit__(self, exc_type, exc_val, exc_tb): # close ctx manager - restore `open`
        self.globals_dict['open'] = self.old_open

Example usage when "mocked" function declaration is in the same module as where it's called:

def read(path):
    with open(path, 'r') as f:
        print(f.read())

# ...
import sys
module_reference = sys.modules[__name__]

string = "siala_baba_mak"
with FileToStringIOPatch(module_reference)
    read(string)  # prints siala_baba_mak
read('path/to/file.txt')  # outside of context manager it still works as usual

Example usage when "mocked" function declaration is in another module

def read(path):
    with open(path, 'r') as f:
        print(f.read())

# ...

from xyz import read
import xyz
module_reference = xyz

string = "siala_baba_mak"
with FileToStringIOPatch(module_reference)
    read(string)  # prints siala_baba_mak
read('path/to/file.txt')  # outside of context manager it still works as usual

You can even provide custom readers to add additional functionalities e.g. reading from AWS S3 bucket.

from s3path import S3Path
path = S3Path("s3://path/to/file.txt")
with FileToStringIOPatch(module_reference, new_reader=path.open):
    read_(path)
YuseqYaseq
  • 244
  • 5
  • 16