5

I'm looking for a library that allows to create a "fake" directory in RAM and get path to it that would work like a path to normal directory on disk.

My scenario is that I have a python script that executes another program (third party that I cannot modify) but it produces files and writes them to a specified location. Then, in my script, I read these files and do something with them. It's slow and I know that the bottleneck here is reading/writing files to/from disk (even if it's SSD).

Is it possible that I don't change the core of my script and only replace the temporary folder path that I use to store intermediate files? I don't need them and I remove them after they are processed.

The perfect solution would be something like this:

import fakeRAM
tmp_dir_path = fakeRAM.get_path()

...

os.system("program.exe " + tmp_dir_path)
martineau
  • 119,623
  • 25
  • 170
  • 301
kostek
  • 801
  • 2
  • 15
  • 32
  • 2
    You mean like a RAMdisk? – Ignacio Vazquez-Abrams Jul 03 '18 at 22:16
  • Why don't you create a RAMdisk and install this program on it? – Pitto Jul 03 '18 at 22:19
  • Note that OP wants a ramdisk which is available to a blackbox external program, so it needs to be available in the regular OS, outside of python. – Hack Saw Jul 03 '18 at 22:23
  • Added an example how I want to use it. – kostek Jul 03 '18 at 22:34
  • You could use a `dict()` with mapping of `(str, list)` or `(str, dict)` for pathname => children paths – OneCricketeer Jul 03 '18 at 22:37
  • @cricket_007 I don't see how it solves my problem. – kostek Jul 03 '18 at 22:38
  • Because you can traverse and add "files or directories" as just named dictionaries – OneCricketeer Jul 03 '18 at 22:39
  • My point is that in memory variables are "in RAM" since you are not writing to disk. Then you are only missing the construct of "navigation" or "modification" within that structure, however, `program.exe` will not have access to that value. – OneCricketeer Jul 03 '18 at 22:40
  • 1
    Create a virtualfs such as tmpfs (UX-like) or RAMdisk (windows) and have your program run in there. Not sure you could reasonably wedge your Python between another program and its calls to your OS (even even you did, it would probably not be good for its performance). – Ondrej K. Jul 03 '18 at 23:09
  • This is not a Python problem, as you really want to create a RAMdisk / tmpfs volume that's available to the operating system (outside of Python), since you want to run a .exe against it. There's plenty ways of doing that, depending on your OS, and you could then run that solution from Python like any other command. – Grismar Sep 04 '19 at 06:08
  • To improve the speed, I suggest to use somthing like multiprocessing or celery to make it faster to read files or do some tasks async. – Waket Zheng Sep 04 '19 at 06:11

1 Answers1

0
import tempfile
import pandas as pd
import io
data=pd.read_csv("data.csv")
temp = tempfile.NamedTemporaryFile(prefix="", suffix=".csv", mode='w+b', delete=False)
data.to_csv(temp.name)
print("The created file is", temp.name)
user3743703
  • 1
  • 1
  • 4
    `NamedTemporaryFile` still creates a file on the file system. Using `SpooledTemporaryFile` might solve the problem, as the file is spooled in memory until a certain size is reached, but that still doesn't really answer OP's question directly. – Grismar Sep 04 '19 at 06:04
  • Also, it won't help them running an external .exe against it, if they do manage to keep it in memory. – Grismar Sep 04 '19 at 06:11