Final results

1 Podatki

2 Končni rezultati

To so rezultati na TestSet, ki ni nič popravljena in obdelana.

Test results (optimal models):

Model Accuracy F1 ROC AUC Sensitivity* Specificity*
LogisticRegression 0.789 0.269 0.796 0.710 0.764
DecisionTree 0.797 0.295 0.807 0.739 0.792
RandomForest 0.977 0.751 0.906 0.739 0.956
GradientBoosting 0.974 0.733 0.873 0.717 0.951
AdaBoost 0.945 0.156 0.800 0.710 0.732
XGBoost 0.976 0.747 0.864 0.732 0.931
LightGBM 0.974 0.737 0.872 0.703 0.946
SVC 0.966 0.633 0.879 0.812 0.848
KNN 0.970 0.703 0.896 0.862 0.810
MLP 0.953 0.509 0.842 0.732 0.822

* Sensitivity and Specificity computed at the Youden-optimal threshold (maximizes sensitivity + specificity − 1) from ROC curve.

3 ROC analiza

4 ROC analiza (interaktivno)

5 Primerjava rezultatov

6 AUC z intervali zaupanja

AUC with 95% CI (bootstrap):

Model AUC 95% CI
LogisticRegression 0.796 [0.756, 0.836]
DecisionTree 0.807 [0.765, 0.849]
RandomForest 0.906 [0.872, 0.939]
GradientBoosting 0.873 [0.831, 0.912]
AdaBoost 0.800 [0.757, 0.837]
XGBoost 0.864 [0.823, 0.905]
LightGBM 0.872 [0.832, 0.909]
SVC 0.880 [0.841, 0.915]
KNN 0.896 [0.861, 0.927]
MLP 0.842 [0.800, 0.879]

7 DeLong test (pairwise AUC)

DeLong test (pairwise AUC):

Model A Model B AUC diff z p p_holm
AdaBoost DecisionTree -0.0069 -0.372 0.7102 1.0000
AdaBoost GradientBoosting -0.0726 -3.748 0.0002 0.0057
AdaBoost KNN -0.0966 -5.940 0.0000 0.0000
AdaBoost LightGBM -0.0718 -3.932 0.0001 0.0029
AdaBoost LogisticRegression 0.0043 0.273 0.7848 1.0000
AdaBoost MLP -0.0421 -2.173 0.0297 0.5950
AdaBoost RandomForest -0.1060 -6.827 0.0000 0.0000
AdaBoost SVC -0.0791 -4.032 0.0001 0.0020
AdaBoost XGBoost -0.0642 -3.403 0.0007 0.0173
DecisionTree GradientBoosting -0.0657 -3.736 0.0002 0.0058
DecisionTree KNN -0.0897 -5.239 0.0000 0.0000
DecisionTree LightGBM -0.0649 -4.230 0.0000 0.0009
DecisionTree LogisticRegression 0.0112 0.504 0.6145 1.0000
DecisionTree MLP -0.0352 -1.716 0.0862 1.0000
DecisionTree RandomForest -0.0991 -5.776 0.0000 0.0000
DecisionTree SVC -0.0722 -3.363 0.0008 0.0193
DecisionTree XGBoost -0.0573 -3.681 0.0002 0.0070
GradientBoosting KNN -0.0239 -1.649 0.0991 1.0000
GradientBoosting LightGBM 0.0008 0.111 0.9120 0.9120
GradientBoosting LogisticRegression 0.0769 3.795 0.0001 0.0049
GradientBoosting MLP 0.0306 1.833 0.0667 1.0000
GradientBoosting RandomForest -0.0333 -2.598 0.0094 0.1969
GradientBoosting SVC -0.0065 -0.404 0.6859 1.0000
GradientBoosting XGBoost 0.0084 0.990 0.3223 1.0000
KNN LightGBM 0.0248 1.775 0.0758 1.0000
KNN LogisticRegression 0.1009 5.599 0.0000 0.0000
KNN MLP 0.0545 3.552 0.0004 0.0107
KNN RandomForest -0.0094 -0.968 0.3331 1.0000
KNN SVC 0.0174 1.275 0.2023 1.0000
KNN XGBoost 0.0323 2.082 0.0373 0.7092
LightGBM LogisticRegression 0.0761 4.012 0.0001 0.0021
LightGBM MLP 0.0297 1.927 0.0539 0.9711
LightGBM RandomForest -0.0342 -2.777 0.0055 0.1208
LightGBM SVC -0.0073 -0.455 0.6491 1.0000
LightGBM XGBoost 0.0076 1.072 0.2835 1.0000
LogisticRegression MLP -0.0464 -3.654 0.0003 0.0075
LogisticRegression RandomForest -0.1103 -6.781 0.0000 0.0000
LogisticRegression SVC -0.0834 -4.787 0.0000 0.0001
LogisticRegression XGBoost -0.0685 -3.429 0.0006 0.0164
MLP RandomForest -0.0639 -4.845 0.0000 0.0000
MLP SVC -0.0370 -2.822 0.0048 0.1097
MLP XGBoost -0.0221 -1.335 0.1819 1.0000
RandomForest SVC 0.0269 1.899 0.0575 0.9777
RandomForest XGBoost 0.0417 3.080 0.0021 0.0497
SVC XGBoost 0.0149 0.912 0.3618 1.0000