8

I'm running some large jobs in Databricks, which for now, include inventorying the data lake. I'm trying to print all blob names within a prefix (sub-folder). There are a lot of files in these sub-folders, and I'm getting about 280 rows of file names printed, but then I see this: *** WARNING: skipped 494256 bytes of output *** Then, I get another 280 rows printed.

I'm guessing there is a control to change this, right. I certainly hope so. This is designed to work with BIG data, not ~280 records. I understand that huge data sets can easily crash a browser, but common, this is basically nothing.

ASH
  • 20,759
  • 19
  • 87
  • 200

2 Answers2

3

you can check all printed values in your driver’s logs

enter image description here

enter image description here enter image description here

Karthikeyan Rasipalay Durairaj
  • 1,920
  • 13
  • 35
1

After a little more research, I stumbled on something that worked for me.

enter image description here

Also, this will display the contents of a dataframe, 

display(df)

So, that will generate the view you see directly above.

ASH
  • 20,759
  • 19
  • 87
  • 200