A Python package for extracting information from large and high-dimensional mixed-type data through explainable cluster analysis.
clust-learn enables users to run end-to-end explainable cluster analysis to extract information from large and high-dimensional
mixed-type data, and it does so by providing a framework that guides the user through data preprocessing, dimensionality reduction,
clustering, and classification of the obtained clusters. It is designed to require very few lines of code, and with a strong
focus on explainability.
clust-learn is organized into four modules, one for each component of the methodological framework presented here:
Figue 1 shows the package layout with the functionalities covered by each module along with the techniques used, the explainability strategies available, and the main functions and class methods encapsulating these techniques and explainability strategies.
The package is implemented with Python 3.9 using open source libraries. It relies heavily on pandas and scikit-learn. Read the complete list of requirements here.
It can be installed manually or from pip/PyPI (see Section 4. Installation).
The package is on PyPI. Simply run:
pip install clust-learn
Please report bugs and feature requests through creating a new issue here.
clust-learn is organized into four modules:
Figue 1 shows the package layout with the functionalities covered by each module along with the techniques used, the explainability strategies available, and the main functions and class methods encapsulating these techniques and explainability strategies.
The four modules are designed to be used sequentially to ensure robust and explainable results. However, each of them is independent and can be used separately to suit different use cases.
Data preprocessing consists of a set of manipulation and transformation tasks performed on the raw data before it is used for its analysis. Although data quality is essential for obtaining robust and reliable results, real-world data is often incomplete, noisy, or inconsistent. Therefore, data preprocessing is a crucial step in any analytical study.
compute_missing(df, normalize=True)
Calculates the pct/count of missing values per column.
Parameters
df : pandas.DataFramenormalize : boolean, default=TrueReturns
missing_df : pandas.DataFrame
missing_values_heatmap(df, output_path=None, savefig_kws=None)
Plots a heatmap to visualize missing values (light color).
Parameters
df : pandas.DataFrame
output_path : str, default=None
savefig_kws : dict, default=None
impute_missing_values(df, num_vars, cat_vars, num_pair_kws=None, mixed_pair_kws=None, cat_pair_kws=None, graph_thres=0.05, k=8, max_missing_thres=0.33)
This function imputes missing values following this steps:
Parameters
df : pandas.DataFrame
num_vars : str, list, pandas.Series, or numpy.array
cat_vars : str, list, pandas.Series, or numpy.array
{num,mixed,cat}_pair_kws : dict, default=None
corr_thres and method for setting the correlation coefficient threshold and method. By default, corr_thres=0.7 and method='pearson'.np2_thres for setting the a threshold on partial eta square with 0.14 as default value.mi_thres for setting a threshold on mutual information score. By default, mi_thres=0.6.graph_thres : float, default=0.05
k : int, default=8
max_missing_thres: float, default=0.33
Returns
final_pairs : pandas.DataFrame
var1: variable with values to impute; var2: variable to be used as independent variable for model-based imputation), together proportion of missing values of variables var1 and var2.plot_imputation_distribution_assessment(df_prior, df_posterior, imputed_vars, sample_frac=1.0, prior_kws=None, posterior_kws=None, output_path=None, savefig_kws=None)
Plots a distribution comparison of each variable with imputed variables, before and after imputation.
Parameters
df_prior : pandas.DataFrame
df_posterior : pandas.DataFrame
imputed_vars : list
sample_frac : float, default=1.0
{prior,posterior}_kws : dict, default=None
output_path : str, default=None
savefig_kws : dict, default=None
remove_outliers(df, variables, iforest_kws=None)
Removes outliers using the Isolation Forest algorithm.
Parameters
df : pandas.DataFrame
variables : list
iforest_kws : dict, default=None
Returns
pandas.DataFrame
pandas.DataFrame
All the functionality of this module is encapsulated in the DimensionalityReduction class so that the original data, the instances of the models used, and any other relevant information is self-maintained and always accessible.
dr = DimensionalityReduction(df, num_vars=None, cat_vars=None, num_algorithm='pca', cat_algorithm='mca', num_kwargs=None, cat_kwargs=None)
| Parameter | Type | Description |
|---|---|---|
df | pandas.DataFrame | Data table containing the data with the original variables |
num_vars | string, list, pandas.Series, or numpy.array | Numerical variable name(s) |
cat_vars | string, list, pandas.Series, or numpy.array | Categorical variable name(s) |
num_algorithm | string | Algorithm to be used for dimensionality reduction of numerical variables. By default, PCA is used. The current version also supports SPCA |
cat_algorithm | string | Algorithm to be used for dimensionality reduction of categorical variables. By default, MCA is used. The current version doesn’t support other algorithms |
num_kwargs | dictionary | Additional keyword arguments to pass to the model used for numerical variables |
cat_kwargs | dictionary | Additional keyword arguments to pass to the model used for categorical variables |
| Attribute | Type | Description |
n_components_ | int | Final number of extracted components |
min_explained_variance_ratio_ | float | Minimum explained variance ratio. By default, 0.5 |
num_trans_ | pandas.DataFrame | Extracted components from numerical variables |
cat_trans_ | pandas.DataFrame | Extracted components from categorical variables |
num_components_ | list | List of names assigned to the extracted components from numerical variables |
cat_components_ | list | List of names assigned to the extracted components from categorical variables |
pca_ | sklearn.decomposition.PCA | PCA instance used to speed up some computations and for comparison purposes |
transform(self, n_components=None, min_explained_variance_ratio=0.5)
Transforms a DataFrame df to a lower dimensional space.
num_main_contributors(self, thres=0.5, n_contributors=None, dim_idx=None, component_description=None, col_description=None, output_path=None)
Computes the original numerical variables with the strongest relation to the derived variable(s) (measured as Pearson correlation coefficient).
cat_main_contributors(self, thres=0.14, n_contributors=None, dim_idx=None, component_description=None, col_description=None, output_path=None)
Computes the original categorical variables with the strongest relation to the derived variable(s)(measured as correlation ratio).
cat_main_contributors_stats(self, thres=0.14, n_contributors=None, dim_idx=None, output_path=None)
Computes for every categorical variable's value, the mean and std of the derived variables that are strongly related to the categorical variable (based on the correlation ratio)).
plot_num_explained_variance(self, thres=0.5, plots='all', output_path=None, savefig_kws=None)
Plot the explained variance (ratio, cumulative, and/or normalized) for numerical variables.
plot_cat_explained_variance(self, thres=0.5, plots='all', output_path=None, savefig_kws=None)
Plot the explained variance (ratio, cumulative, and/or normalized) for categorical variables.
plot_num_main_contributors(self, thres=0.5, n_contributors=5, dim_idx=None, output_path=None, savefig_kws=None)
Plot main contributors (original variables with the strongest relation with derived variables) for every derived variable.
plot_cat_main_contributor_distribution(self, thres=0.14, n_contributors=None, dim_idx=None, output_path=None, savefig_kws=None)
Plot main contributors (original variables with the strongest relation with derived variables) for every derived variable.
The Clustering class encapsulates all the functionality of this module and stores the data, the instances of the algorithms used, and other relevant information so it is always accessible.
cl = Clustering(df, algorithms='kmeans', normalize=False)
| Parameter | Type | Description |
|---|---|---|
df | pandas.DataFrame | Data frame containing the data to be clustered |
algorithms | instance or list of instances | Algorithm instances to be used for clustering. They must implement the fit and set_params methods |
normalize | bool | Whether to apply data normalization for fair comparisons between variables. In case dimensionality reduction is applied beforehand, normalization should not be applied |
| Attribute | Type | Description |
dimensions_ | list | List of columns of they input data frame |
instances_ | dict | Pairs of algorithm name and its instance |
metric_ | string | The cluster validation metric used. Four metrics available: ['inertia', 'davies_bouldin_score', 'silhouette_score', 'calinski_harabasz_score'] |
optimal_config_ | tuple | Tuple with the optimal configuration for clustering containing the algorithm name, number of clusters, and value of the chosen validation metric |
scores_ | dict | Pairs of algorithm name and a list of values of the chosen validation metric for a cluster range |
compute_clusters(self, n_clusters=None, metric='inertia', max_clusters=10, prefix=None, weights=None)
Calculates clusters. If more than one algorithm is passed in the class constructor, first, the optimal number of clusters is computed for each algorithm based on the metric passed to the method. Secondly, the algorithm that provides the best performance for the corresponding optimal number of clusters is selected. Therefore, the result shows the clusters calculated with the best performing algorithm based on the criteria explained above.
describe_clusters(self, df_ext=None, variables=None, cluster_filter=None, statistics=['mean', 'median', 'std'], output_path=None)
Describes clusters based on internal or external continuous variables.
For categorical variables use describe_clusters_cat().
describe_clusters_cat(self, cat_array, cat_name, order=None, normalize=False, use_weights=False, output_path=None)
Describes clusters based on external categorical variables. The result is a contingency table.
For continuous variables use describe_clusters().
compare_cluster_means_to_global_means(self, df_original=None, output_path=None)
For every cluster and every internal variable, the relative difference between the intra-cluster mean and the global mean.
anova_tests(self, df_test=None, vars_test=None, cluster_filter=None, output_path=None)
Runs ANOVA tests for a given set of continuous variables (internal or external) to test dependency with clusters.
chi2_test(self, cat_array)
Runs Chi-squared tests for a given categorical variable to test dependency with clusters.
plot_score_comparison(self, output_path=None, savefig_kws=None)
Plots the comparison in performance between the different clustering algorithms.
plot_optimal_components_normalized(self, output_path=None, savefig_kws=None)
Plots the normalized curve used for computing the optimal number of clusters.
plot_clustercount(self, use_weights=False, output_path=None, savefig_kws=None)
Plots a bar plot with cluster counts.
plot_cluster_means_to_global_means_comparison(self, use_weights= False, df_original=None, xlabel=None, ylabel=None,
levels=[-0.50, -0.32, -0.17, -0.05, 0.05, 0.17, 0.32, 0.50],
output_path=None, savefig_kws=None)
Plots the normalized curve used for computing the optimal number of clusters.
plot_distribution_comparison_by_cluster(self, df_ext=None, xlabel=None, ylabel=None, output_path=None, savefig_kws=None)
Plots the violin plots per cluster and continuous variables of interest to understand differences in their distributions by cluster.
plot_clusters_2D(self, coor1, coor2, use_weights=False, style_kwargs=dict(), output_path=None, savefig_kws=None)
Plots two 2D plots:
- A scatter plot styled by the categorical variable hue.
- A 2D plot comparing cluster centroids and optionally the density area.
plot_cat_distribution_by_cluster(self, cat_array, cat_label, order=None, cluster_label=None, use_weights=False, output_path=None, savefig_kws=None)
Plots the relative contingency table of the clusters with a categorical variable as a stacked bar plot.
The functionality of this module is encapsulated in the Classifier class, which is also responsible for storing the original data, the instances of the models used, and any other relevant information.
classifier = Classifier(df, predictor_cols, target, num_cols=None, cat_cols=None)
| Parameter | Type | Description |
|---|---|---|
df | pandas.DataFrame | Data frame containing the data |
predictor_cols | list of string | List of columns to use as predictors |
target | numpy.array or list | Values of the target variable |
num_cols | list | List of numerical columns from predictor_cols |
cat_cols | list | List of categorical columns from predictor_cols |
| Attribute | Type | Description |
filtered_features_ | list | List of columns of the input data frame |
labels_ | list | List of class labels |
model_ | Instance of TransformerMixin and BaseEstimator from sklearn.base | Trained classifier |
X_train_ | numpy.array | Train split of predictors |
X_test_ | numpy.array | Test split of predictors |
y_train_ | numpy.array | Train split of target |
y_test_ | numpy.array | Test split of target |
grid_result_ | sklearn.model_selection.GridSearchCV | Instance of fitted estimator for hyperparameter tuning |
train_model(self, model=None, feature_selection=True, features_to_keep=[],
feature_selection_model=None, hyperparameter_tuning=False, param_grid=None,
train_size=0.8, balance_classes=False)
This method trains a classification model.
By default, it uses XGBoost, but any other estimator (instance of scikit-learn.Estimator) can be used.
The building process consists of three main steps:
Feature removing highly correlated variables using a classification model and SHAP values to determine which to keep, and Recursive Feature Elimination with Cross-Validation (RFECV) on the remaining features.
Runs grid search with cross-validation for hyperparameter tuning. Note the parameter grid must be passed.
Trains a classification model with the selected features and hyperparameters. By default, an XGBoost classifier will be trained.
Note both hyperparameter tuning and model training are run on a train set. Train-test split is performed
using sklearn.model_selection.train_test_split.
hyperparameter_tuning_metrics(self, output_path=None)
This method returns the average and standard deviation of the cross-validation runs for every hyperparameter combination in hyperparameter tuning.
confusion_matrix(self, test=True, sum_stats=True, output_path=None)
This method returns the confusion matrix of the classification model.
classification_report(self, test=True, output_path=None)
This method returns the sklearn.metrics.classification_report in pandas.DataFrame format.
This report contains the intra-class metrics precision, recall and F1-score, together with the global accuracy, and macro average and weighted average of the three intra-class metrics.
plot_shap_importances(self, n_top=7, output_path=None, savefig_kws=None)
Plots shap importance values, calculated as the combined average of the absolute values of the shap values for all classes.
plot_shap_importances_beeswarm(self, class_id, class_name=None, n_top=10, output_path=None, savefig_kws=None)
Plots a summary of shap values for a specific class of the target variable. This uses shap beeswarm plot.
plot_confusion_matrix(self, test=True, sum_stats=True, output_path=None, savefig_kws=None)
This function makes a pretty plot of an sklearn Confusion Matrix cf using a Seaborn heatmap visualization.
plot_roc_curves(self, test=True, labels=None, output_path=None, savefig_kws=None)
Plots ROC curve for every class.
Alvarez-Garcia, M., Ibar-Alonso, R., Arenas-Parra, M. (2024). A comprehensive framework for explainable cluster analysis. Information Sciences, 663 , 120282, https://doi.org/10.1016/j.ins.2024.120282
FAQs
A Python package for explainable cluster analysis
We found that clust-learn demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.