1

Our company is migrating from s3 to GCS. While the command-line utility gsutil works fine, I am facing difficulty in configuring Hadoop (core-site.xml) to enable access to GCS. This google page https://storage.googleapis.com/hadoop-conf/gcs-core-default.xml lists the name-value pairs that need to be added, but I don't find any of these in the ~/.boto file. The .boto file only has the following set:

  1. gs_oauth2_refresh_token under [Credentials]
  2. default_project_id under [GSUtil]
  3. Few others like api_version etc.. The [OAuth2] section is empty.

Can I somehow generate the necessary keys using gs_oauth2_refresh_token and add them to Hadoop config? Or can I get these from any other gsutil config files?

Manjunath Ballur
  • 6,287
  • 3
  • 37
  • 48
Aditya
  • 13
  • 2

1 Answers1

0

For hadoop configuration you'll likely want to use a service-account rather than gsutil credentials that are associated with an actual email address; see these instructions for manual installation of the GCS connector for more details about setting up a p12 keyfile along with the other necessary configuration parameters.

Community
  • 1
  • 1
Dennis Huo
  • 10,517
  • 27
  • 43