Databricks display() function equivalent or alternative to Jupyter

Question

I'm in the process of migrating current DataBricks Spark notebooks to Jupyter notebooks, DataBricks provides convenient and beautiful display(data_frame) function to be able to visualize Spark dataframes and RDDs ,but there's no direct equivalent for Jupyter(im not sure but i think its a DataBricks specific function), i tried :

dataframe.show()

But it's a text version of it ,when you have many columns it breaks , so i'm trying to find an alternative to display() that can render Spark dataframes better than show() functions. Is there any equivalent or alternative to this?

https://stackoverflow.com/questions/26873127/show-dataframe-as-table-in-ipython-notebook — OneCricketeer, Sep 09 '17 at 03:26
Thanks, so `.show()` is pretty much a worse version of `display()` in databricks (where available) — information_interchange, Jul 23 '19 at 19:22
If you're using Scala Almond kernel in Jupyter, you can use this solution https://github.com/almond-sh/almond/issues/180 (add `import org.apache.spark.sql.DataFrame` to the top) — ELinda, Aug 20 '21 at 14:59

score 11 · Accepted Answer · edited May 31 '22 at 11:43

11

When you use Jupyter, instead of using df.show() use myDF.limit(10).toPandas().head(). And, as sometimes, we are working multiple columns it truncates the view. So just set your Pandas view column config to the max.

# Alternative to Databricks display function.
import pandas as pd
pd.set_option('max_columns', None)

myDF.limit(10).toPandas().head()

edited May 31 '22 at 11:43

Duncan Lukkenaer

12,050
13
64
97

answered Mar 18 '20 at 15:19

AP-Big Data

182
1
9

As of Version 1.4 of pandas, you have to use the key "display.max_columns". Reference https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.set_option.html#pandas.set_option – Giuseppe Mar 30 '22 at 09:16

Erkan Şirin · Answer 2 · 2019-05-01T10:55:46.167

First Recommendation: When you use Jupyter, don't use df.show() instead use df.limit(10).toPandas().head() which results perfect display even better Databricks display()

Second Recommendation: Zeppelin Notebook. Just use z.show(df.limit(10))

Additionally in Zeppelin;

You register your dataframe as SQL Table df.createOrReplaceTempView('tableName')
Insert new paragraph beginning %sql then query your table with amazing display.

score 2 · Answer 3 · answered Sep 10 '17 at 19:06

In recent IPython, you can just use display(df) if df is a panda dataframe, it will just work. On older version you might need to do a from IPython.display import display. It will also automatically display if the result of the last expression of a cell is a data_frame. For example this notebook. Of course the representation will depends on the library you use to make your dataframe. If you are using PySpark and it does not defined a nice representation by default, then you'll need to teach IPython how to display the Spark DataFrame. For example here is a project that teach IPython how to display Spark Contexts, and Spark Sessions.

score 1 · Answer 4 · answered Apr 27 '22 at 11:49

1

Without converting to pandas dataframe. Use this... This will render dataframe in proper grids.

from IPython.core.display import HTML
display(HTML("<style>pre { white-space: pre !important; }</style>"))

df.show()

answered Apr 27 '22 at 11:49

Itachi

2,817
27
35

score 1 · Answer 5 · answered Dec 08 '22 at 11:19

You can set config spark.conf.set('spark.sql.repl.eagerEval.enabled', True). This will allow to display native pyspark DataFrame without explicitly using df.show() and there is also no need to transfer DataFrame to Pandas either, all you need to is just df.

score -3 · Answer 6 · answered Sep 09 '17 at 03:23

-3

Try Apache Zeppelin (https://zeppelin.apache.org/). There's some nice standard visualizations of dataframes, specifically if you use the sql interpreter. There's also support for other useful interpreters as well.

answered Sep 09 '17 at 03:23

josephpconley

1,703
12
12

Databricks display() function equivalent or alternative to Jupyter

6 Answers6