0

I'm not sure how to word my question exactly, and I have seen some similar questions asked but not exactly what I'm trying to do. If there already is a solution please direct me to it.

Here is what I'm trying to do:

At my work, we have a few pkgs we've built to handle various data types. One I am working with is reading in a csv file into a std_io object (std_io is our all-purpose object class that reads in any type of data file).

I am trying to connect this to another pkg I am writing, so I can make an object in the new pkg, and covert it to a std_io object.

The problem is, the std_io object is meant to read an actual file, not take in an object. To get around this, I can basically write my data to temp.csv file then read it into a std_io object.

I am wondering if there is a way to eliminate this step of writing the temp.csv file.

Here is my code:

x #my object
df = x.to_df() #object class method to convert to a pandas dataframe
df.to_csv('temp.csv') #write data to a csv file
std_io_obj = std_read('temp.csv') #read csv file into a std_io object

Is there a way to basically pass what the output of writing the csv file would be directly into std_read? Does this make sense?

The only reason I want to do this is to avoid having to code additional functionality into either of the pkgs to directly accept an object as input.

Hope this was clear, and thanks to anyone who contributes.

Derek Eden
  • 4,403
  • 3
  • 18
  • 31
  • Do you really have to give a filename as parameter to `std_read`, or could you pass it a file object? – Thierry Lathuille Jan 08 '20 at 15:18
  • https://stackoverflow.com/questions/18550127/how-to-do-virtual-file-processing – gstukelj Jan 08 '20 at 15:20
  • I have to pass a filename..which makes me think this might not be possible and I might've worded this weirdly – Derek Eden Jan 08 '20 at 15:22
  • `df.to_csv()` without a filename? – Quang Hoang Jan 08 '20 at 15:22
  • maybe I'd be best to create an object class method that initializes an empty std_io object then populates it? – Derek Eden Jan 08 '20 at 15:23
  • @QuangHoang thanks.. I didn't know I could do that, using this I might be able to just make a very slight modification to the std_read function that will accept this instead of a filename – Derek Eden Jan 08 '20 at 15:24
  • show the code of `std_read`, so we know what data you need at the end and what changes you need to do in the `std_read`. Probably @QuangHoang already answered the question. And @gst has a solution with memory files in worst case. – ya_dimon Jan 08 '20 at 15:32
  • I can't post the std_read code..but I can say that it just opens the csv file and makes a csv.reader object to parse the data out – Derek Eden Jan 08 '20 at 15:40
  • 1
    I might be able to just pass the df.to_csv() string into csv.reader like this?? https://stackoverflow.com/questions/41977519/how-to-use-string-as-input-for-csv-reader-without-storing-it-to-file – Derek Eden Jan 08 '20 at 15:41
  • for any interested..creating a virtual file is likely the way I will be going https://stackoverflow.com/questions/49072771/how-to-create-a-virtual-file-for-output which requires the least changes in my std_read function – Derek Eden Jan 10 '20 at 00:41

1 Answers1

0

For those interested, or who may have this same kind of issue/objective, here's what I did to solve this problem.

I basically just created a temporary named file, linked a .csv filename to this temp file, then passed it into my std_read function which requires a csv filename as an input.

This basically tricks the function into thinking it's taking the name of a real file as an input, and it just opens it as usual and uses csvreader to parse it up.

This is the code:

import tempfile
import os

x #my object I want to convert to a  std_io object
text = x.to_df().to_csv() #object class method to convert to a pandas dataframe then generate the 'text' of a csv file

filename = 'temp.csv'

with tempfile.NamedTemporaryFile(dir = os.path.dirname('.')) as f:
    f.write(text.encode())
    os.link(f.name, filename)

stdio_obj = std_read(filename)
os.unlink(filename)
del f

FYI - the std_read function essentially just opens the file the usual way, and passes it into csvreader:

with open(filename, 'r') as f:
    rdr = csv.reader(f)
Derek Eden
  • 4,403
  • 3
  • 18
  • 31