0

Hello I have one csv file stored in a bucket, I would like to use that file in a cloud function, so I need to download it and then use this file in a process as follows:

def plot(event, context):
    client = storage.Client()
    df = pd.read_csv('call_conversations.csv', index_col=0)
    objects = df['filepart']
    y_pos = np.arange(len(objects))
    performance = df['confidence']
    plt.bar(y_pos, performance, align='center', alpha=0.99,color='blue')
    plt.xticks(y_pos, objects,rotation=90)
    plt.ylabel('Confianza') 
    plt.title('')
    plt.savefig('cloud.png')
    print('successfull')

I tried with: 

def plot(event, context):
    client = storage.Client()

Here I get the csv file successfully as a string, 

    csv = client.bucket(event['bucket']).blob(event['name']).download_as_string()
    df = pd.read_csv(csv, index_col=0)
    objects = df['filepart']
    y_pos = np.arange(len(objects))
    performance = df['confidence']
    plt.bar(y_pos, performance, align='center', alpha=0.99,color='blue')
    plt.xticks(y_pos, objects,rotation=90)
    plt.ylabel('Confianza') 
    plt.title('Nivel de Confianza Transcripciones')
    plt.savefig('cloud.png')
    print('successfull')

however I got:

  File "local.py", line 67, in <module>
    trigger()
  File "local.py", line 64, in trigger
    plot(event,None)
  File "local.py", line 49, in plot
    df = pd.read_csv(csv, index_col=0)
  File "/home/adolfo/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 702, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/adolfo/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 429, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/adolfo/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 895, in __init__
    self._make_engine(self.engine)
  File "/home/adolfo/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 1122, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/home/adolfo/.local/lib/python3.6/site-packages/pandas/io/parsers.py", line 1853, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 387, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 725, in pandas._libs.parsers.TextReader._setup_parser_source
OSError: Expected file path name or file-like object, got <class 'bytes'> type

Since I need to translate this code to cloud functions I would like to find a way to download the csv from the bucket and keep it in memory, to be used then with pandas,

I also tried with: StringIO 

def plot(event, context):

    client = storage.Client()
    csv = client.bucket(event['bucket']).blob(event['name']).download_as_string()

    df = pd.read_csv(StringIO(csv), index_col=0)
    objects = df['filepart']
    y_pos = np.arange(len(objects))
    performance = df['confidence']
    plt.bar(y_pos, performance, align='center', alpha=0.99,color='blue')
    plt.xticks(y_pos, objects,rotation=90)
    plt.ylabel('Confianza') 
    plt.title('Nivel de Confianza Transcripciones')
    plt.savefig('cloud.png')
    print('successfull')

However I got:

Traceback (most recent call last):
  File "local.py", line 67, in <module>
    trigger()
  File "local.py", line 64, in trigger
    plot(event,None)
  File "local.py", line 49, in plot
    df = pd.read_csv(StringIO(csv), index_col=0)
TypeError: initial_value must be str or None, not bytes
neo33
  • 1,809
  • 5
  • 18
  • 41

1 Answers1

2

The problem is that the Pandas read_csv() API is expecting a file name or file like object to read from. In your call, you are passing in a string read from the object found at the bucket. This means that you have already read the content and want to parse that content into your data frame. I did a search for how to achieve that and found the following recipe:

Create Pandas DataFrame from a string

which seems to have an excellent solution using StringIO. Have a read at that link and I am hoping the integration into your own solution will be straight forward.

If the data is bytes, we can use io.BytesIO as the source of data into read_csv(). See for example:

StringIO replacement that works with bytes instead of strings?

Kolban
  • 13,794
  • 3
  • 38
  • 60