PyCM is a multi-class confusion matrix library written in Python that supports both input data vectors and direct matrix, and a proper tool for post-classification model evaluation that supports most classes and overall statistics parameters. PyCM is the swiss-army knife of confusion matrices, targeted mainly at data scientists that need a broad array of metrics for predictive models and accurate evaluation of a large variety of classifiers.
Fig1. ConfusionMatrix Block Diagram
| Open Hub |  |
| PyPI Counter | |
| Github Stars |  |
| Branch | master | dev |
| CI |  |  |
| Code Quality |
⚠️ PyCM 3.9 is the last version to support Python 3.5
⚠️ PyCM 2.4 is the last version to support Python 2.7 & Python 3.4
⚠️ Plotting capability requires Matplotlib (>= 3.0.0) or Seaborn (>= 0.9.1)
pip install pycm==4.1pip install .conda update condaconda install -c sepandhaghighi pycmAdd to PATH optionInstall pip optionpip install pycm>> pyversion PYTHON_EXECUTABLE_FULL_PATH
>>> from pycm import *
>>> y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
>>> y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]
>>> cm = ConfusionMatrix(actual_vector=y_actu, predict_vector=y_pred)
>>> cm.classes
[0, 1, 2]
>>> cm.table
{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}
>>> cm.print_matrix()
Predict 0 1 2
Actual
0 3 0 0
1 0 1 2
2 2 1 3
>>> cm.print_normalized_matrix()
Predict 0 1 2
Actual
0 1.0 0.0 0.0
1 0.0 0.33333 0.66667
2 0.33333 0.16667 0.5
>>> cm.stat(summary=True)
Overall Statistics :
ACC Macro 0.72222
F1 Macro 0.56515
FPR Macro 0.22222
Kappa 0.35484
Overall ACC 0.58333
PPV Macro 0.56667
SOA1(Landis & Koch) Fair
TPR Macro 0.61111
Zero-one Loss 5
Class Statistics :
Classes 0 1 2
ACC(Accuracy) 0.83333 0.75 0.58333
AUC(Area under the ROC curve) 0.88889 0.61111 0.58333
AUCI(AUC value interpretation) Very Good Fair Poor
F1(F1 score - harmonic mean of precision and sensitivity) 0.75 0.4 0.54545
FN(False negative/miss/type 2 error) 0 2 3
FP(False positive/type 1 error/false alarm) 2 1 2
FPR(Fall-out or false positive rate) 0.22222 0.11111 0.33333
N(Condition negative) 9 9 6
P(Condition positive or support) 3 3 6
POP(Population) 12 12 12
PPV(Precision or positive predictive value) 0.6 0.5 0.6
TN(True negative/correct rejection) 7 8 4
TON(Test outcome negative) 7 10 7
TOP(Test outcome positive) 5 2 5
TP(True positive/hit) 3 1 3
TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.5
>>> from pycm import *
>>> cm2 = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2": 2}, "Class2": {"Class1": 0, "Class2": 5}})
>>> cm2
pycm.ConfusionMatrix(classes: ['Class1', 'Class2'])
>>> cm2.classes
['Class1', 'Class2']
>>> cm2.print_matrix()
Predict Class1 Class2
Actual
Class1 1 2
Class2 0 5
>>> cm2.print_normalized_matrix()
Predict Class1 Class2
Actual
Class1 0.33333 0.66667
Class2 0.0 1.0
>>> cm2.stat(summary=True)
Overall Statistics :
ACC Macro 0.75
F1 Macro 0.66667
FPR Macro 0.33333
Kappa 0.38462
Overall ACC 0.75
PPV Macro 0.85714
SOA1(Landis & Koch) Fair
TPR Macro 0.66667
Zero-one Loss 2
Class Statistics :
Classes Class1 Class2
ACC(Accuracy) 0.75 0.75
AUC(Area under the ROC curve) 0.66667 0.66667
AUCI(AUC value interpretation) Fair Fair
F1(F1 score - harmonic mean of precision and sensitivity) 0.5 0.83333
FN(False negative/miss/type 2 error) 2 0
FP(False positive/type 1 error/false alarm) 0 2
FPR(Fall-out or false positive rate) 0.0 0.66667
N(Condition negative) 5 3
P(Condition positive or support) 3 5
POP(Population) 8 8
PPV(Precision or positive predictive value) 1.0 0.71429
TN(True negative/correct rejection) 5 1
TON(Test outcome negative) 7 1
TOP(Test outcome positive) 1 7
TP(True positive/hit) 1 5
TPR(Sensitivity, recall, hit rate, or true positive rate) 0.33333 1.0
matrix() and normalized_matrix() renamed to print_matrix() and print_normalized_matrix() in version 1.5threshold is added in version 0.9 for real value prediction.
For more information visit Example3
file is added in version 0.9.5 in order to load saved confusion matrix with .obj format generated by save_obj method.
For more information visit Example4
sample_weight is added in version 1.2
For more information visit Example5
transpose is added in version 1.2 in order to transpose input matrix (only in Direct CM mode)
relabel method is added in version 1.5 in order to change ConfusionMatrix classnames.
>>> cm.relabel(mapping={0: "L1", 1: "L2", 2: "L3"})
>>> cm
pycm.ConfusionMatrix(classes: ['L1', 'L2', 'L3'])
position method is added in version 2.8 in order to find the indexes of observations in predict_vector which made TP, TN, FP, FN.
>>> cm.position()
{0: {'FN': [], 'FP': [0, 7], 'TP': [1, 4, 9], 'TN': [2, 3, 5, 6, 8, 10, 11]}, 1: {'FN': [5, 10], 'FP': [3], 'TP': [6], 'TN': [0, 1, 2, 4, 7, 8, 9, 11]}, 2: {'FN': [0, 3, 7], 'FP': [5, 10], 'TP': [2, 8, 11], 'TN': [1, 4, 6, 9]}}
to_array method is added in version 2.9 in order to returns the confusion matrix in the form of a NumPy array. This can be helpful to apply different operations over the confusion matrix for different purposes such as aggregation, normalization, and combination.
>>> cm.to_array()
array([[3, 0, 0],
[0, 1, 2],
[2, 1, 3]])
>>> cm.to_array(normalized=True)
array([[1. , 0. , 0. ],
[0. , 0.33333, 0.66667],
[0.33333, 0.16667, 0.5 ]])
>>> cm.to_array(normalized=True, one_vs_all=True, class_name="L1")
array([[1. , 0. ],
[0.22222, 0.77778]])
combine method is added in version 3.0 in order to merge two confusion matrices. This option will be useful in mini-batch learning.
>>> cm_combined = cm2.combine(cm3)
>>> cm_combined.print_matrix()
Predict Class1 Class2
Actual
Class1 2 4
Class2 0 10
plot method is added in version 3.0 in order to plot a confusion matrix using Matplotlib or Seaborn.
>>> cm.plot()
>>> from matplotlib import pyplot as plt
>>> cm.plot(cmap=plt.cm.Greens, number_label=True, plot_lib="matplotlib")
>>> cm.plot(cmap=plt.cm.Reds, normalized=True, number_label=True, plot_lib="seaborn")
ROCCurve, added in version 3.7, is devised to compute the Receiver Operating Characteristic (ROC) or simply ROC curve. In ROC curves, the Y axis represents the True Positive Rate, and the X axis represents the False Positive Rate. Thus, the ideal point is located at the top left of the curve, and a larger area under the curve represents better performance. ROC curve is a graphical representation of binary classifiers' performance. In PyCM, ROCCurve binarizes the output based on the "One vs. Rest" strategy to provide an extension of ROC for multi-class classifiers. Getting the actual labels vector, the target probability estimates of the positive classes, and the list of ordered labels of classes, this method is able to compute and plot TPR-FPR pairs for different discrimination thresholds and compute the area under the ROC curve.
>>> crv = ROCCurve(actual_vector=np.array([1, 1, 2, 2]), probs=np.array([[0.1, 0.9], [0.4, 0.6], [0.35, 0.65], [0.8, 0.2]]), classes=[2, 1])
>>> crv.thresholds
[0.1, 0.2, 0.35, 0.4, 0.6, 0.65, 0.8, 0.9]
>>> auc_trp = crv.area()
>>> auc_trp[1]
0.75
>>> auc_trp[2]
0.75
PRCurve, added in version 3.7, is devised to compute the Precision-Recall curve in which the Y axis represents the Precision, and the X axis represents the Recall of a classifier. Thus, the ideal point is located at the top right of the curve, and a larger area under the curve represents better performance. Precision-Recall curve is a graphical representation of binary classifiers' performance. In PyCM, PRCurve binarizes the output based on the "One vs. Rest" strategy to provide an extension of this curve for multi-class classifiers. Getting the actual labels vector, the target probability estimates of the positive classes, and the list of ordered labels of classes, this method is able to compute and plot Precision-Recall pairs for different discrimination thresholds and compute the area under the curve.
>>> crv = PRCurve(actual_vector=np.array([1, 1, 2, 2]), probs=np.array([[0.1, 0.9], [0.4, 0.6], [0.35, 0.65], [0.8, 0.2]]), classes=[2, 1])
>>> crv.thresholds
[0.1, 0.2, 0.35, 0.4, 0.6, 0.65, 0.8, 0.9]
>>> auc_trp = crv.area()
>>> auc_trp[1]
0.29166666666666663
>>> auc_trp[2]
0.29166666666666663
This option has been added in version 1.9 to recommend the most related parameters considering the characteristics of the input dataset.
The suggested parameters are selected according to some characteristics of the input such as being balance/imbalance and binary/multi-class.
All suggestions can be categorized into three main groups: imbalanced dataset, binary classification for a balanced dataset, and multi-class classification for a balanced dataset.
The recommendation lists have been gathered according to the respective paper of each parameter and the capabilities which had been claimed by the paper.
>>> cm.imbalance
False
>>> cm.binary
False
>>> cm.recommended_list
['MCC', 'TPR Micro', 'ACC', 'PPV Macro', 'BCD', 'Overall MCC', 'Hamming Loss', 'TPR Macro', 'Zero-one Loss', 'ERR', 'PPV Micro', 'Overall ACC']
is_imbalanced parameter has been added in version 3.3, so the user can indicate whether the concerned dataset is imbalanced or not. As long as the user does not provide any information in this regard, the automatic detection algorithm will be used.
>>> cm = ConfusionMatrix(y_actu, y_pred, is_imbalanced=True)
>>> cm.imbalance
True
>>> cm = ConfusionMatrix(y_actu, y_pred, is_imbalanced=False)
>>> cm.imbalance
False
In version 2.0, a method for comparing several confusion matrices is introduced. This option is a combination of several overall and class-based benchmarks. Each of the benchmarks evaluates the performance of the classification algorithm from good to poor and give them a numeric score. The score of good and poor performances are 1 and 0, respectively.
After that, two scores are calculated for each confusion matrices, overall and class-based. The overall score is the average of the score of seven overall benchmarks which are Landis & Koch, Cramer, Matthews, Goodman-Kruskal's Lambda A, Goodman-Kruskal's Lambda B, Krippendorff's Alpha, and Pearson's C. In the same manner, the class-based score is the average of the score of six class-based benchmarks which are Positive Likelihood Ratio Interpretation, Negative Likelihood Ratio Interpretation, Discriminant Power Interpretation, AUC value Interpretation, Matthews Correlation Coefficient Interpretation and Yule's Q Interpretation. It should be noticed that if one of the benchmarks returns none for one of the classes, that benchmarks will be eliminated in total averaging. If the user sets weights for the classes, the averaging over the value of class-based benchmark scores will transform to a weighted average.
If the user sets the value of by_class boolean input True, the best confusion matrix is the one with the maximum class-based score. Otherwise, if a confusion matrix obtains the maximum of both overall and class-based scores, that will be reported as the best confusion matrix, but in any other case, the compared object doesn’t select the best confusion matrix.
>>> cm2 = ConfusionMatrix(matrix={0: {0: 2, 1: 50, 2: 6}, 1: {0: 5, 1: 50, 2: 3}, 2: {0: 1, 1: 7, 2: 50}})
>>> cm3 = ConfusionMatrix(matrix={0: {0: 50, 1: 2, 2: 6}, 1: {0: 50, 1: 5, 2: 3}, 2: {0: 1, 1: 55, 2: 2}})
>>> cp = Compare({"cm2": cm2, "cm3": cm3})
>>> print(cp)
Best : cm2
Rank Name Class-Score Overall-Score
1 cm2 0.50278 0.58095
2 cm3 0.33611 0.52857
>>> cp.best
pycm.ConfusionMatrix(classes: [0, 1, 2])
>>> cp.sorted
['cm2', 'cm3']
>>> cp.best_name
'cm2'
From version 4.0, MultiLabelCM has been added to calculate class-wise or sample-wise multilabel confusion matrices. In class-wise mode, confusion matrices are calculated for each class, and in sample-wise mode, they are generated per sample. All generated confusion matrices are binarized with a one-vs-rest transformation.
>>> mlcm = MultiLabelCM(actual_vector=[{"cat", "bird"}, {"dog"}], predict_vector=[{"cat"}, {"dog", "bird"}], classes=["cat", "dog", "bird"])
>>> mlcm.actual_vector_multihot
[[1, 0, 1], [0, 1, 0]]
>>> mlcm.predict_vector_multihot
[[1, 0, 0], [0, 1, 1]]
>>> mlcm.get_cm_by_class("cat").print_matrix()
Predict 0 1
Actual
0 1 0
1 0 1
>>> mlcm.get_cm_by_sample(0).print_matrix()
Predict 0 1
Actual
0 1 0
1 1 1
online_help function is added in version 1.1 in order to open each statistics definition in web browser
>>> from pycm import online_help
>>> online_help("J")
>>> online_help("SOA1(Landis & Koch)")
>>> online_help(2)
online_help() (without argument)alt_link = True (new in version 2.4)
PyCM can be used online in interactive Jupyter Notebooks via the Binder or Colab services! Try it out now! :
Examples in Document folderNLnet foundation has supported the PyCM project from version 3.6 to 4.0 through the NGI Assure Fund. This fund is set up by NLnet foundation with funding from the European Commission's Next Generation Internet program, administered by DG Communications Networks, Content, and Technology under grant agreement No 957073.
Python Software Foundation (PSF) grants PyCM library partially for version 3.7. PSF is the organization behind Python. Their mission is to promote, protect, and advance the Python programming language and to support and facilitate the growth of a diverse and international community of Python programmers.
Some parts of the infrastructure for this project are supported by:
If you use PyCM in your research, we would appreciate citations to the following paper :
Haghighi, S., Jasemi, M., Hessabi, S. and Zolanvari, A. (2018). PyCM: Multiclass confusion matrix library in Python. Journal of Open Source Software, 3(25), p.729.
@article{Haghighi2018,
doi = {10.21105/joss.00729},
url = {https://doi.org/10.21105/joss.00729},
year = {2018},
month = {may},
publisher = {The Open Journal},
volume = {3},
number = {25},
pages = {729},
author = {Sepand Haghighi and Masoomeh Jasemi and Shaahin Hessabi and Alireza Zolanvari},
title = {{PyCM}: Multiclass confusion matrix library in Python},
journal = {Journal of Open Source Software}
}
Download PyCM.bib
| JOSS |  |
| Zenodo |  |
Give a ⭐️ if this project helped you!
If you do like our project and we hope that you do, can you please support us? Our project is not and is never going to be working for profit. We need the money just so we can continue doing what we do ;-) .
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
feature_request.yml templateconfig.yml for issue templateSECURITY.mdthresholds_calc function updated__midpoint_numeric_integral__ function updated__trapezoidal_numeric_integral__ function updatedAUTHORS.md updatedREADME.md modifiedPython 3.12 added to test.ymlPython 3.13 added to test.ymlpycm_util.py renamed to utils.pypycm_test.py renamed to basic_test.pypycm_profile.py renamed to profile.pypycm_param.py renamed to params.pypycm_overall_func.py renamed to overall_funcs.pypycm_output.py renamed to output.pypycm_obj.py renamed to cm.pypycm_multilabel_cm.py renamed to multilabel_cm.pypycm_interpret.py renamed to interpret.pypycm_handler.py renamed to handlers.pypycm_error.py renamed to errors.pypycm_distance.py renamed to distance.pypycm_curve.py renamed to curve.pypycm_compare.py renamed to compare.pypycm_class_func.py renamed to class_funcs.pypycm_ci.py renamed to ci.pypycmMultiLabelError classMultiLabelCM classget_cm_by_class methodget_cm_by_sample method__mlcm_vector_handler__ function__mlcm_assign_classes__ function__mlcm_vectors_filter__ function__set_to_multihot__ functiondeprecated functionREADME.md modifiedOVERALL_PARAMS dictionary__imbalancement_handler__ functionvector_serializer functionlog_loss methodmetrics_off parameter added to ConfusionMatrix __init__ methodCLASS_PARAMS changed to a dictionarysort parameter added to relabel methodCONTRIBUTING.md updatedcodecov removed from dev-requirements.txtdistance method__contains__ method__getitem__ methodrelabel method sort bug fixedREADME.md modifiedCompare overall benchmarks default weights updatedCurve classROCCurve classPRCurve classpycmCurveError classCONTRIBUTING.md updatedmatrix_params_calc function optimizedREADME.md modifiedPython 3.11 added to test.ymlclasses parameter added to matrix_params_from_table functionnumpy.integer elements are now acceptedmatrix parameter accepting formatsREADME.md modifiedplot method updatedclass_statistics function modifiedoverall_statistics function modifiedBCD_calc function modifiedCONTRIBUTING.md updatedCODE_OF_CONDUCT.md updatedbrier_score methodJ (Jaccard index) section in Document.ipynb updatedsave_obj method updatedPython 3.10 added to test.ymlCONTRIBUTING.md updated__compare_weight_handler__ functionis_imbalanced parameter added to ConfusionMatrix __init__ methodclass_benchmark_weight and overall_benchmark_weight parameters added to Compare __init__ methodstatistic_recommend function modifiedweight parameter renamed to class_weightAUTHORS.md updatedREADME.md modifiedclasses_filter functionclasses parameter added to matrix_params_calc functionclasses parameter added to __obj_vector_handler__ functionclasses parameter added to ConfusionMatrix __init__ methodname parameter removed from html_init functionshortener parameter added to html_table functionshortener parameter added to save_html methodrequirements-splitter.pysensitivity_index methodoverall_statistics function modifiedCONTRIBUTING.md updatedplot_test.pyaxes_gen functionadd_number_label functionplot methodcombine methodmatrix_combine functionREADME.md modifiednotebook_check.pyto_array method__copy__ methodcopy methodaverage method refactoredlabel_map attributepositions attributeposition methodweighted_alpha methodCLASS_NUMBER_ERROR error type changed to pycmMatrixErrorrelabel method bug fixedREADME.md modifiedaverage methodweighted_average methodweighted_kappa methodpycmAverageError classREADME.md modifiedrelabel method bug fixedsparse_table_print function bug fixedmatrix_check function bug fixedCompare class fixedcustom_rounder functioncomplement functionsparse_matrix attributesparse_normalized_matrix attributesparse parameter added to print_matrix,print_normalized_matrix and save_stat methodsheader parameter added to save_csv methodpycm_handler.pypycm_error.pyverified_test.pyCONTRIBUTING.md updatedREADME.md modifiedprint_normalized_matrix method modifiednormalized_table_calc function modifiedsetup.py modifiedPython 3.8 added to .travis.yaml and appveyor.ymlPC_PI_calc function__version__ variableinstall.shautopep8.shCI method (supported statistics : ACC,AUC,Overall ACC,Kappa,TPR,TNR,PPV,NPV,PLR,NLR,PRE)test.sh moved to .travis folderAUTHORS.md updatedsave_stat,save_csv and save_html methods Non-ASCII character bug fixedCONTRIBUTING.md updatedREADME.md modifiedCI attribute renamed to CI95kappa_se_calc function renamed to kappa_SE_calcse_calc function modified and renamed to SE_calcpycm_ci.pysave_html method fixedFUNDING.ymlAUC_calc function modifiedsummary parameter added to save_html,save_stat,save_csv and stat methodssample_weight bug in numpy array format fixedalt_link parameter added to save_html method and online_help functionCompare class tests moved to compare_test.pywarning_test.pysave_stat and save_vector parameters added to save_obj methodREADME.md modifiedCompare class fixedpycm_help function modifiedCompare class score calculation modifiedREADME.md modifiedCompare class and parameters recommendation system block diagramsCompare classConfusionMatrix equal methodstat_print function bug fixedtable_print function bug fixedBeta parameter renamed to beta (F_calc function & F_beta method)normalize parameter added to save_html methodpycm_func.py splitted into pycm_class_func.py and pycm_overall_func.pyvector_filter,vector_check,class_check and matrix_check functions moved to pycm_util.pyRACC_calc and RACCU_calc functions exception handler modifiedCODE_OF_CONDUCT.mdISSUE_TEMPLATE.mdPULL_REQUEST_TEMPLATE.mdCONTRIBUTING.mdsave_html methodsave_matrix and normalize parameters added to save_csv methodREADME.md modifiedConfusionMatrix.__init__ optimizedrelabel method bug fixedversion_check.pycolor parameter added to save_html methodpycm_interpret.pypycm_util.pyelse and elif removed== changed to ispycm_profile.pyclass_name parameter added to stat,save_stat,save_csv and save_html methodsoverall_param and class_param parameters empty list bug fixedmatrix_params_calc, matrix_params_from_table and vector_filter functions optimizedoverall_MCC_calc, CEN_misclassification_calc and convex_combination functions optimizedoverall_param and class_param parameters added to stat,save_stat and save_html methodsclass_param parameter added to save_csv method_ removed from overall statistics namesREADME.md modified__len__ methodrelabel method__class_stat_init__ function__overall_stat_init__ functionmatrix attribute as dictnormalized_matrix attribute as dictnormalized_table attribute as dictREADME.md modifiedLR+ renamed to PLRLR- renamed to NLRnormalized_matrix method renamed to print_normalized_matrixmatrix method renamed to print_matrixentropy_calc fixedcross_entropy_calc fixedconditional_entropy_calc fixedprint_table bug for large numbers fixedsave_obj fixedtranspose bug in save_obj fixedPython 3.7 added to .travis.yaml and appveyor.ymlone_vs_alldev-requirements.txtREADME.md modifiedsave_stat modifiedrequirements.txt modifiedREADME.md modifiedsample_weighttransposeREADME.md modifiedOSX env added to .travis.ymlonline_help functionREADME.md modifiedhtml_table function modifiedtable_print function modifiednormalized_table_print function modifiedREADME.md modifiedREADME.md modifiedREADME.md modifiedsetup.py modifiedREADME.md modifiedREADME.md modifiedpycmVectorError classpycmMatrixError classnumpy arrayspycmError classdigit parameter to ConfusionMatrix objectoverall_statstatistic_result to class_statparams to statREADME.mdFAQs
Multi-class confusion matrix library in Python
We found that pycm demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.