30

I am struggling to massage a dataframe in pandas into the correct format for seaborn's heatmap (or matplotlib really) to make a heatmap.

My current dataframe (called data_yule) is:

     Unnamed: 0  SymmetricDivision         test  MutProb      value
3             3                1.0  sackin_yule    0.100  -4.180864
8             8                1.0  sackin_yule    0.050  -9.175349
13           13                1.0  sackin_yule    0.010 -11.408114
18           18                1.0  sackin_yule    0.005 -10.502450
23           23                1.0  sackin_yule    0.001  -8.027475
28           28                0.8  sackin_yule    0.100  -0.722602
33           33                0.8  sackin_yule    0.050  -6.996394
38           38                0.8  sackin_yule    0.010 -10.536340
43           43                0.8  sackin_yule    0.005  -9.544065
48           48                0.8  sackin_yule    0.001  -7.196407
53           53                0.6  sackin_yule    0.100  -0.392256
58           58                0.6  sackin_yule    0.050  -6.621639
63           63                0.6  sackin_yule    0.010  -9.551801
68           68                0.6  sackin_yule    0.005  -9.292469
73           73                0.6  sackin_yule    0.001  -6.760559
78           78                0.4  sackin_yule    0.100  -0.652147
83           83                0.4  sackin_yule    0.050  -6.885229
88           88                0.4  sackin_yule    0.010  -9.455776
93           93                0.4  sackin_yule    0.005  -8.936463
98           98                0.4  sackin_yule    0.001  -6.473629
103         103                0.2  sackin_yule    0.100  -0.964818
108         108                0.2  sackin_yule    0.050  -6.051482
113         113                0.2  sackin_yule    0.010  -9.784686
118         118                0.2  sackin_yule    0.005  -8.571063
123         123                0.2  sackin_yule    0.001  -6.146121

and my attempts using matplotlib was:

plt.pcolor(data_yule.SymmetricDivision, data_yule.MutProb, data_yule.value)

which threw the error: 

ValueError: not enough values to unpack (expected 2, got 1)

and the seaborn attempt was:

sns.heatmap(data_yule.SymmetricDivision, data_yule.MutProb, data_yule.value)

which threw:

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

It seems trivial as both functions want rectangular dataset, but I'm missing something, clearly.

cancerconnector
  • 1,225
  • 2
  • 14
  • 21

4 Answers4

68

Use pandas.DataFrame.pivot (no aggregation of values=) or pandas.DataFrame.pivot_table (with aggregation of values=) to reshape the dataframe from a long to wide form. The index will be on the y-axis, and the columns will be on the x-axis. See Reshaping and pivot tables for an overview.

In [96]: result
Out[96]: 
MutProb               0.001      0.005      0.010     0.050     0.100
SymmetricDivision                                                    
0.2               -6.146121  -8.571063  -9.784686 -6.051482 -0.964818
0.4               -6.473629  -8.936463  -9.455776 -6.885229 -0.652147
0.6               -6.760559  -9.292469  -9.551801 -6.621639 -0.392256
0.8               -7.196407  -9.544065 -10.536340 -6.996394 -0.722602
1.0               -8.027475 -10.502450 -11.408114 -9.175349 -4.180864

Then you can pass the 2D array (or DataFrame) to seaborn.heatmap or plt.pcolor:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# load the sample data
df = pd.DataFrame({'MutProb': [0.1,
  0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001], 'SymmetricDivision': [1.0, 1.0, 1.0, 1.0, 1.0, 0.8, 0.8, 0.8, 0.8, 0.8, 0.6, 0.6, 0.6, 0.6, 0.6, 0.4, 0.4, 0.4, 0.4, 0.4, 0.2, 0.2, 0.2, 0.2, 0.2], 'test': ['sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule'], 'value': [-4.1808639999999997, -9.1753490000000006, -11.408113999999999, -10.50245, -8.0274750000000008, -0.72260200000000008, -6.9963940000000004, -10.536339999999999, -9.5440649999999998, -7.1964070000000007, -0.39225599999999999, -6.6216390000000001, -9.5518009999999993, -9.2924690000000005, -6.7605589999999998, -0.65214700000000003, -6.8852289999999989, -9.4557760000000002, -8.9364629999999998, -6.4736289999999999, -0.96481800000000006, -6.051482, -9.7846860000000007, -8.5710630000000005, -6.1461209999999999]})

# pivot the dataframe from long to wide form
result = df.pivot(index='SymmetricDivision', columns='MutProb', values='value')

sns.heatmap(result, annot=True, fmt="g", cmap='viridis')
plt.show()

yields

enter image description here

Trenton McKinney
  • 56,955
  • 33
  • 144
  • 158
unutbu
  • 842,883
  • 184
  • 1,785
  • 1,677
4

Another option using .grouby() and .unstack()

df_m = df.groupby(["SymmetricDivision", "MutProb"])
          .size()
          .unstack(level=0)

sns.heatmap(df_m)
Antiez
  • 679
  • 7
  • 11
0

I have to create a heatmap to represent results of hyperparameter tuning for my DecisionTreeClassifier model, the hyperparameters being ['max_depth', 'min_samples_split']. The metric to represent the accuracy of model is auc_roc_score.

we will use seaborn heatmap to create a dataset for sns.heatmap().

we use the pandas.pivot_table() to pivot a DataFrame in pandas, One of the manipulation do before making heatmap is it use Pandas pivot functionality to reshape the data for making heatmaps. For further understanding, pandas pivot_table()

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt[![enter image description here][2]][2]

max_depth = [1, 5, 10, 50]
min_samples_split = [5, 10, 100, 500]

train_set_auc = [0.5478688747761202, 0.5478688747761202, 0.5478688747761202, 
                 0.5478688747761202, 0.6384311073066989, 0.6383831998001249, 
                 0.6382879025310534, 0.6381476488966438, 0.7099806625823346, 
                 0.7090830674744466, 0.7035862433327912, 0.6987068807767094, 
                 0.9203751110844185, 0.9140516357169505, 0.8909876822536001, 
                 0.845991656101182]

# Now, let me explain the approach
# 1. we will use pandas.pivot_table()[seaborn heatmap tutorial][1] [pandas pivot_table()][1] to create a 
# dataset for sns.heatmap()

# convert train_set_auc into 4X4 np array

def return_dataframe(lst, max_depth, min_samples):
    """ this function will take a list and convert it into a 
        4X4 numpy array. Further, it will create an empty dataframe 
        with columns= min_samples_split and indices as max_depth. Then 
        it will fill the entries with values from the numpy array
   """
    arr = []
    idx = 0

    for idx in range(0, len(lst), 4):
        arr.append(np.array(lst[idx:idx+4]))

    arr = np.array(arr)

    df = pd.DataFrame(data=arr, index=min_samples, columns=max_depth)

   return df

train_set1 = return_dataframe(lst=train_set1_auc, max_depth=max_depth, min_samples=min_samples_split)
fig, ax = plt.subplots()

ax = sns.heatmap(train_set1, annot=True, fmt='g', cmap='viridis')
plt.xlabel("Your x-label")
plt.ylabel("Your y-label")
plt.show()

i have added plt.xlabel() & plt.ylabel() in the edit. Sorry for this now shown in the image below. By the way, this works. Try it once, if you haven't

sns.heatmap here

hansrajswapnil
  • 549
  • 1
  • 6
  • 14
0

Once you pivot the data using pd.DataFrame.pivot(), you can also use pd.DataFrame.style to make a heatmap.

Load Data

import pandas as pd

# load the sample data
df = pd.DataFrame({'MutProb': [0.1,
  0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001, 0.1, 0.05, 0.01, 0.005, 0.001], 'SymmetricDivision': [1.0, 1.0, 1.0, 1.0, 1.0, 0.8, 0.8, 0.8, 0.8, 0.8, 0.6, 0.6, 0.6, 0.6, 0.6, 0.4, 0.4, 0.4, 0.4, 0.4, 0.2, 0.2, 0.2, 0.2, 0.2], 'test': ['sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule', 'sackin_yule'], 'value': [-4.1808639999999997, -9.1753490000000006, -11.408113999999999, -10.50245, -8.0274750000000008, -0.72260200000000008, -6.9963940000000004, -10.536339999999999, -9.5440649999999998, -7.1964070000000007, -0.39225599999999999, -6.6216390000000001, -9.5518009999999993, -9.2924690000000005, -6.7605589999999998, -0.65214700000000003, -6.8852289999999989, -9.4557760000000002, -8.9364629999999998, -6.4736289999999999, -0.96481800000000006, -6.051482, -9.7846860000000007, -8.5710630000000005, -6.1461209999999999]})

Pivot data

pivot_table_df = df.pivot(index='SymmetricDivision', columns='MutProb', values='value')

Create heatmap

axis=None means use all of the values in the table

pivot_table_df.style.background_gradient(axis=None)

enter image description here

Create inline bars

pivot_table_df.style.bar(axis=None)

enter image description here

PaulMest
  • 12,925
  • 7
  • 53
  • 50