I am trying to conduct hierarchical clustering through Japanese words/terms and using scipy.cluster.hierarchy.dendrogram to plot the results. However, the plot cannot show the Japanese words/terms but instead use small rectangles. At first, I was thinking this may be because when I create the dictionary, the keys are unicode not Japanese (as the question I asked here). Then I was suggested to use Python3 to solve such issue and I finally make the dictionary key in Japanese words instead of unicode (as the question I ask here). However, it turns out that even if I feed the label parameter of scipy.cluster.hierarchy.dendrogram with Japanese words/terms, the plot still cannot show those words. I have checked several similar posts but it seems like there is still no clear solution. My codes are as follows:
import pandas as pd
import numpy as np
from sklearn import decomposition
from sklearn.cluster import AgglomerativeClustering as hicluster
from scipy.spatial.distance import cdist, pdist
from scipy import sparse as sp ## Sparse Matrix
from scipy.cluster.hierarchy import dendrogram
import matplotlib as mpl
import matplotlib.pyplot as plt
plt.style.use('ggplot')
## Import Data
allWrdMat10 = pd.read_csv("../../data/allWrdMat10.csv.gz",
encoding='CP932')
## Set X as CSR Sparse Matrix
X = np.array(allWrdMat10)
X = sp.csr_matrix(X)
def plot_dendrogram(model, **kwargs):
# Children of hierarchical clustering
children = model.children_
# Distances between each pair of children
# Since we don't have this information, we can use a uniform one
for plotting
distance = np.arange(children.shape[0])
# The number of observations contained in each cluster level
no_of_observations = np.arange(2, children.shape[0]+2)
# Create linkage matrix and then plot the dendrogram
linkage_matrix = np.column_stack([children, distance,
no_of_observations]).astype(float)
# Plot the corresponding dendrogram
dendrogram(linkage_matrix, **kwargs)
dict_index = {t:i for i,t in enumerate(allWrdMat10.columns)}
dictlist = []
temp = []
akey = []
avalue = []
for key, value in dict_index.items():
akey.append(key)
avalue.append(value)
temp = [key,value]
dictlist.append(temp)
avalue = np.array(avalue)
X_transform = X[:, avalue < 1000].transpose().toarray()
freq1000terms = akey
freq1000terms = np.array(freq1000terms)[avalue < 1000]
hicl_ward = hicluster(n_clusters=40,linkage='ward', compute_full_tree =
False)
hiclwres = hicl_ward.fit(X_transform)
plt.rcParams["figure.figsize"] = (15,6)
model1 = hiclwres
plt.title('Hierarchical Clustering Dendrogram (Ward Linkage)')
plot_dendrogram(model1, p = 40, truncate_mode = 'lastp', orientation =
'top', labels=freq1000terms[model1.labels_], color_threshold = 991)
plt.ylim(959,1000)
plt.show()
