ENH: Hierarchical clustering of the correlation matrix

[celprov]

Implement a new feature to perform hierarchical clustering on the correlation matrix before plotting it.

The hierarchical clustering can be activated by the flag sort in plot_corrmat(), which is by default set to False, such that the default behavior of this function remains unchanged.

fix : correct the alignment of the labels on the horizontal axis

[oesteban]

looking good. main caveat is whether pandas is not an overkill

[oesteban]

Numpy should be sufficient to reorder, something like

Suggested change

	# Build a new dataframe with the sorted columns
	for idx, i in enumerate(data.columns[labels_order]):
	if idx == 0:
	clustered = pd.DataFrame(data[i])
	else:
	df_to_append = pd.DataFrame(data[i])
	clustered = pd.concat([clustered, df_to_append], axis=1)
	data = clustered
	data = np.take(data, labels_order, axis=0)

Q2 - don't you want to also sort the rows?

[celprov]

Wow very fast, thanks.
The panda implementation reorder both the rows and the columns.

[oesteban]

Okay, I think np.take will then work for you with something like (labels_order, labels_order) or zip((labels_order, labels_order)) for the indexes and no axis argument.

[celprov]

Unfortunately, none of the suggestions work and with a quick search on internet, I couldn't figure out how to reorder both rows and columns in a np.array. I thus suggest we keep the panda implementation.

[oesteban]

I think this is easier than you think:

reordered_idx = (0, 1, 2, 4, 5, 3, 6, 7, 8, 9)
data.take(indices=reordered_idx, axis=0).take(indices=reordered_idx, axis=1)

The only caveat is that you need to do the reordering on the full correlation matrix, and only after the reordering drop the upper triangle (if you want to do so).

[celprov]

Thanks a lot, it works with this suggestion. I really could not figure out how to do the reordering on np.array.
It indeed greatly simplifies the code.
Can I merge the PR now?

[celprov]

@oesteban time to revive this. I think it is ready to just merge it, as I could run the code on the IQMs from the IXI dataset and since we already worked on reviews long time back.

Here is a resulting correlation matrix plot