Bigquery DATABRICKS CONNECTIVITY

Question

How to access data from Big Query to dataframe in databricks using credentials in secrets.

df = spark
    .read
    .format("bigquery")
    .option("credentialsFile",credentialfilepath)
    .option("parentProject",projectName)
    .option("table",table).load()

Used above but since i am passing the account details of GCP from secrets not sure on how to get it done.

Answer 1

The docs on this say:

Credentials can also be provided explicitly, either as a parameter or from Spark runtime configuration. They should be passed in as a base64-encoded string directly.

// Globally
spark.conf.set("credentials", "<SERVICE_ACCOUNT_JSON_IN_BASE64>")
// Per read/Write
spark.read.format("bigquery").option("credentials", "<SERVICE_ACCOUNT_JSON_IN_BASE64>")

Assuming you have your access details in Databricks secret scope, you just read it from there using dbutils:

df = spark
    .read
    .format("bigquery")
    .option("credentials",dbutils.secrets.get("<scope>", "<secret>"))
    .option("parentProject",projectName)
    .option("table",table).load()