0

I am trying to open a csv file which is present in gcs bucket to read the content and process the data accordingly in BigQuery. I am able to list out the csv's present in bucket , but not able to open it . My code is as below:

with cloudstorage.open(bucket_name/gs_file,"r+") as csvFile:
     reader = csv.reader(iter(csvFile.readline,''), delimiter = '|')
     csvfilearray = next(reader)
     print (csvfilearray)

Getting below error :

NameError: name 'cloudstorage' is not defined

Imported cloudstorage as below:

import cloudstorage as gcs

and tried with gcs.open again , But still error :

File "./GCS_Connection_Test.py", line 10, in <module>
    import cloudstorage as gcs
ImportError: No module named cloudstorage

Can anyone please help on how to open a file present on GCS for processing .

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
Shikha
  • 321
  • 1
  • 4
  • 19
  • Did you install the [`google-cloud-storage`](https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python) lib? – Dan Cornilescu Aug 19 '17 at 03:12
  • Or, if you're on GAE, the [`appengine-gcs-client`](https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/setting-up-cloud-storage#downloading_the_client_library) lib which matches your import according to [Reading and Writing to Google Cloud Storage](https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/read-write-to-cloud-storage) – Dan Cornilescu Aug 19 '17 at 03:21
  • executing the python code from google vm instance and I am able to fetch the object list under a particular bucket.So, seems google-cloud-storage lib is installed there.Not able to open the file for processing. I have written below imports in code "from google.cloud import storage" storage_client = storage.Client() – Shikha Aug 19 '17 at 05:46
  • Tried with installing using "pip install GoogleAppEngineCloudStorageClient " .still getting error as below : import cloudstorage as gcs File "/usr/local/lib/python2.7/dist-packages/cloudstorage/__init__.py", line 20, in from .api_utils import RetryParams File "/usr/local/lib/python2.7/dist-packages/cloudstorage/api_utils.py", line 45, in from google.appengine.api import app_identity ImportError: No module named appengine.api Please help, what needs to be done to resolve it – Shikha Aug 21 '17 at 12:57
  • Is this a GAE app or not? If it is, you need the 2nd lib, installed in your app (not on the system), otherwise you need the 1st lib, installed on the system. Not the other way around. – Dan Cornilescu Aug 21 '17 at 13:03
  • It is GAE app. I have tried with installing Library as below: – Shikha Aug 23 '17 at 12:24
  • Then re-check the instructions, you need to install it into your app, i.e. note the `-t` arg in the `pip install ... -t lib` cmd. Your traceback clearly shows it's running the lib installed into your system's dist packages, which won't work on GAE. – Dan Cornilescu Aug 23 '17 at 12:31
  • It is GAE app. I have tried with installing Library as below: pip install GoogleAppEngineCloudStorageClient I could not run "pip install GoogleAppEngineCloudStorageClient -t " As i am not sure what is meant by my Cloudstorage folder contains below files: test_utils.py,storage_api.py,rest_api.py,__init__.py,errors.py,common.py,cloudstorage_api.py,api_utils.py,__init__.pyc,api_utils.pyc cloudstorage_api.pyc,errors.pyc,common.pyc,rest_api.pyc,test_utils.pyc,storage_api.pyc Could you please suggest further – Shikha Aug 23 '17 at 12:38
  • Your [GAE](https://cloud.google.com/appengine/docs/python/) app code directory. Hm, what you showed doesn't quite look like a GAE app. what's your `app.yaml` file content? – Dan Cornilescu Aug 23 '17 at 12:46
  • cat app.yaml runtime: python env: flex entrypoint: gunicorn -b :$PORT main:app runtime_config: python_version: 3 #[START env] env_variables: CLOUD_STORAGE_BUCKET: your-bucket-name #[END env] – Shikha Aug 23 '17 at 13:00

1 Answers1

1

You're on the flexible environment according to your app.yaml file.

You used GoogleAppEngineCloudStorageClient, which is a standard environment library (just as appengine-gcs-client), you need to use the google-cloud-storage library, see Edit project configuration and install dependencies section in the Using Cloud Storage guide, which is the one you need to follow.

You also followed the standard environment instructions for installing the library, you need to use the flexible environment ones, see Using Python Libraries.

General note: look for the docs/flexible vs docs/standard strings in the doc page URLs to distinguish them (or the environment type displayed close to the top of the documentation left-side navigation bar, you may need to scroll up to see it)

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • related: https://stackoverflow.com/questions/45842772/how-to-tell-if-a-google-app-engine-documentation-page-applies-to-the-standard-or/45842773#45842773 – Dan Cornilescu Aug 23 '17 at 14:47
  • First of all, Thanks a lot for responding on my queries. I am new to Google Cloud and kind of lost in documentations and also tired now to find solution to this issue.I followed the link you suggested. Created virtual env too but still at same position. Can you please suggest whether the the cloudstorage import what I am using and Cloudstorage.open() function is Valid(will work )or not in a flexible environment ? is there any other way to open a csv from GCS bucket and process the file according to the requirement. This method looked simple to me but not getting any soultion for import issue. – Shikha Aug 23 '17 at 19:11
  • No, the library you're using now won't work. Get some rest and then start fresh with the right documentation. – Dan Cornilescu Aug 23 '17 at 22:06