Reading Row values in DataBricks Notebook

Question

I have a Databricks python notebook reading ADL configurations from a JSON file on DBFS. I am able to read the file fine and the DF generated shows the values with the below schema.

testJsonData:pyspark.sql.dataframe.DataFrame

clientId:string
oauth2Credential:string
oauth2RefreshUrl:string
providerType:string

I am having challenges in reading a specific row value. I am getting a row object but doesn't give me the value of the row.

testJsonData.select('clientId').collect()
Out[65]: [Row(clientId='6xxxx08vvvvvvvvv7f')]

Appreciate any pointers helping with this.

I found the solution in another article - https://stackoverflow.com/questions/38610559/convert-spark-dataframe-column-to-python-list — Satya Azure, Mar 03 '19 at 22:18

Satya Azure · Accepted Answer · 2019-03-03T22:47:36.950

0

Hope this helps someone having a similar issue, I read the config values like this

   configlist = testJsonData.select("*").collect()
   dataProviderType = configlist[0].providerType
   dataClientId = configlist[0].clientId
   dataOAuthCredential = configlist[0].oauth2Credential
   dataOAuthUrl = configlist[0].outh2RefreshUrl

edited Mar 03 '19 at 22:47

answered Mar 03 '19 at 22:22

Satya Azure

459
7
22

Reading Row values in DataBricks Notebook

1 Answers1