Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data

Goodness-of-Fit Tests and Calibration Machine-Learning Algorithms for Logistic Regression with Sparse Data

Assessing the goodness-of-fit of a logistic regression model is a critical prerequisite before the model is used for inference. However, goodness-of-fit (GOF) tests such as the chi-square and deviance tests often give invalid results when the data are"sparse"-- a common issue with continuous predictors like age or weight, where the asymptotic distributional assumptions are not satisfied. This thesis studies classical GOF tests for binary logistic regression under both grouped and sparse data, comparing about 30 statistical tests and machine-learning calibration algorithms. These span the classical chi-square and Hosmer-Lemeshow variants, standardized Pearson statistics, covariate-space partitioning, smoothing-based methods, and contemporary calibration machine-learning and bootstrap procedures. At a fixed size, the GiViTI calibration test (2016), McCullagh (1989), Osius-Rojek (1992), le Cessie (1995) and Stute-Zhu (2002) proved empirically powerful, balancing correct identification of bad models (high empirical power) against not raising false alarms on good models (correct empirical Type I error). Relying on formal methods alone is insufficient: visual diagnostics such as calibration plots are a vital exploratory step for detecting model deficiencies that formal tests often overlook. An application to real data (the Low Birth Weight dataset) shows that many of these tests fail to give valid conclusions when exposed to the complexities of actual datasets. The main conclusion is that model assessment requires a combination of several powerful statistical tests alongside careful visual inspection of model calibration.

A directional Hosmer-Lemeshow goodness-of-fit test for sparse logistic regression

Goodness-of-fit assessment for the binary logistic regression model is difficult when covariates are continuous: the data are effectively sparse, the classical Pearson and deviance tests fail, and practitioners rely on partition-based tests, such as the Hosmer-Lemeshow test, that group observations before comparing observed and expected counts. We study a partition test that modifies the Hosmer-Lemeshow statistic with a single directional correction term, weighted by $(1-2\bar\pi_g)$ and referred to a $\chi^2_{G-2}$ distribution. The correction is the grouped form of the Osius-Rojek/Farrington standardization; grouping makes it well defined in the sparse regime, and it targets the asymmetric over- and under-prediction that a misspecified link induces. A single alignment functional captures its effect, predicting where the test gains power (asymmetric-link misspecification) and where it does not (symmetric departures, and covariate-space structure that no probability-grouping test can see). In simulations the test holds its size; no well-calibrated partition test is more sensitive to asymmetric-link misfit, and it clearly exceeds Hosmer-Lemeshow there, most so for the complementary log-log link -- a modest gain that fades as $n$ grows; it ties Hosmer-Lemeshow on an omitted interaction and is less powerful on an omitted quadratic (by about ten percentage points at $n=1000$). A real-data application illustrates its use, and the test is implemented in the R package ebrahim.gof.

A robust combination test for goodness-of-fit

A Statistical Analysis of Classical and Advanced Regression Models in Data Science

Regression models remain foundational tools of both mathematical statistics and modern applied data science for prediction, explanation, and model comparison. In applied statistical modelling there is always a fundamental practical trade-off that must be considered. Classical regression models are attractive mainly because they are easy to interpret, and are built on a well-established statistical foundation. However, they will often describe real world data very poorly when the underlying relationships are curved, interactive, or simply more complex than a simple straight-line structure. More flexible models can often deliver substantially improved prediction performance in these cases, but this improvement almost always comes at the cost of reduced interpretability and transparency. This study presents a systematic comparative analysis of classical, regularized, nonlinear, and ensemble models across two widely used benchmark datasets: the Advertising continuous sales prediction task and the Diabetes binary medical classification task. All models were evaluated using cross-validation and held-out test-set performance. Third-degree Polynomial Regression produced the strongest overall result on the Advertising dataset, with a test R² of 0.9907. Gradient Boosting achieved the highest performance on the Diabetes dataset with a test ROC-AUC of 0.8315, though Logistic Regression remained a highly competitive and far more interpretable baseline. We conclude that advanced models can outperform classical baselines when the underlying data structure is complex, while classical models remain essential for interpretation, comparison, and model selection.

Nonparametric Goodness-of-fit Testing under Covariate Shift

This paper develops procedures for nonparametric goodness-of-fit testing under covariate shift, where labelled data are drawn from a source population but goodness-of-fit is evaluated for a target population. The distribution mismatch is quantified by either a bounded moment condition or a sub-exponential tail condition on the target-to-source density ratio. Our method combines truncated importance-weighting kernel ridge regression with a multiplier bootstrap to construct confidence sets for the regression function. The truncation stabilizes the importance- weighting kernel ridge regression as well as the bootstrap calibration, making our approach applicable even when the density ratio has heavy tails. We prove nonasymptotic validity and sharpness of the resulting confidence sets under suitable operator compatibility conditions, and establish explicit error rates for coverage probability under specific conditions on the target- to-source density ratio and on the spectral decay of the kernel integral operator. Numerical experiments corroborate our theoretical findings.

Power Comparison of Selected Robust Non Parametric Tests for Assessing the Normality of Regression Residuals: A Monte Carlo Simulation Study

Assessing the normality of regression residuals is an important component of regression diagnostics because departures from normality can affect statistical inference, particularly in small and moderate samples. This study compared the empirical Type I error rates and empirical power of six selected normality tests for regression residuals using a Monte Carlo simulation framework with 1,000 replications. Data were generated from a simple linear regression model for sample sizes of 10, 20, 30, 40, 50, and 100 under six residual distributions: normal, exponential, Student’s t, Cauchy, bimodal, and contaminated normal. The Energy, Anderson–Darling, Cramér–von Mises, Lilliefors, Shapiro–Francia, and Random Projection tests were evaluated at a nominal significance level of 5%. Under normal residuals, all six procedures produced empirical Type I error rates close to the nominal level. Under non-normal residuals, empirical power increased with sample size but differed according to the form of departure from normality. The Shapiro–Francia and Energy tests showed the strongest performance under exponential residuals, while the Anderson–Darling test performed best under heavy-tailed and contaminated normal residuals. The Random Projection test achieved the highest empirical power under bimodal residuals. The Lilliefors test generally showed the lowest empirical power. These findings indicate that the relative performance of normality tests depends on residual distributional characteristics and sample size, supporting context-specific test selection in regression diagnostics.