A general framework is proposed that integrates both optimistic and pessimistic optimization approaches in solving the regression problem to address outlier cleaning and robustification in a unified fashion and develops solution methods that can be applied to handle data sets of different scales.
Abstract
In this study, we propose a general framework that integrates both optimistic and pessimistic optimization approaches in solving the regression problem to address outlier cleaning and robustification in a unified fashion. Although data cleaning aims to down-weight the outliers, robustification renders the regression models to heavily rely on extreme data. The main objective of this framework is to construct a new optimization scheme capable of withstanding the influence of outliers without harming the robustness level, by combining these two rather contrasting concepts and operations. In addition to showing its generalization to a few well-known regression models, a set of structural properties of our framework is derived to ensure its statistical significance and to understand its computational demand. Then, we develop solution methods, including mixed integer formulations, alternating direction method of multipliers algorithms, and computation enhancement techniques, that can be applied to handle data sets of different scales. Numerical results on both synthetic and benchmark data sets from the University of California, Irvine (UCI) Machine Learning Repository verify the superiority of our new framework and demonstrate the unified strength to handle complex data sets.
History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Learning.
Funding: X. Qian received financial support from the U.S. National Science Foundation (NSF) [Grants SHF-2215573 and IIS-2212419].
Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0884 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2024.0884 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .
In regression modeling, collinearity among input variables, unevenness in the output observations, and outlier points can affect parameter estimation and reduce the optimality of the models. Several approaches exist to address these problems, including penalized and nonparametric models. However, each has its challenges and performs well only for a specific purpose, leaving the other problems unaddressed. In this paper, with a primary focus on fuzzy regression models, we propose a method based on a new linear uniform model that covers the functions of all these methods, such as reducing and controlling collinearity, coping with unevenness in a dataset, and outlier effects, as well as addressing the problems in their structures, such as the lack of closed form, the nonlinearity of the model parameter formula relation, and the single-purpose nature of the obtained models. Furthermore, when the normal distribution is assumed, it performs better than the best method for model fit, i.e., the least-squares method. In this paper, we demonstrate the optimal performance of the proposed method in addressing the aforementioned problems through various numerical and practical examples and compare it with other existing methods.
M. Kashani, M. Arashi, Mohammad Farshad et al.· International Journal of Unc...· 0 citations
Phase I analysis is essential to understand the variability of the process and determine its stability. Providing Phase II control charts with poor parameters’ estimates leads to a weak performance. In case of incomplete Phase I data, the problem of missing values must be dealt with before estimating the process parameters. Researchers commonly rely on the traditional Mean Substitution (MS) and/or the Stochastic Regression (SRG) imputation methods, whilst the number of studies exploiting machine learning algorithms in the SPC field is rapidly increasing. Accordingly, in this study, we consider two common and powerful machine learning‐based imputation methods; which are the k‐Nearest Neighbors (kNN) and Support Vector Regression (SVR). We compare their effect with the two traditional methods, the MS and the SRG methods, on the performance of the G‐chart designed to monitor the process variability. Our results show that kNN imputation either surpasses the performance of the traditional methods or provides a similar performance. An application of the G‐chart is also illustrated. We recommend the use of the kNN imputation while monitoring the process dispersion.
Dina A. Desoki, Nesma A. Saleh, A. Saad et al.· Quality and Reliability Engi...· 0 citations
Estimating the generalization error (GE) of machine learning models is fundamental, with resampling methods being the most common approach. However, in nonstandard settings, particularly those where observations are not independently and identically distributed, resampling using simple random data divisions may lead to biased GE estimates. This paper strives to present well-grounded guidelines for GE estimation in various such nonstandard settings: clustered data, spatial data, unequal sampling probabilities, concept drift and hierarchically structured outcomes. Our overview combines well-established methodologies with other existing methods that, to our knowledge, have not been frequently considered in these particular settings. A unifying principle among these techniques is that the test data used in each iteration of the resampling procedure should reflect the new observations to which the model will be applied, while the training data should be representative of the entire data set used to obtain the final model. Beyond providing an overview, we address literature gaps by conducting simulation studies and a real data study. These studies assess the necessity of using GE-estimation methods tailored to the respective setting. Our findings corroborate the concern that standard resampling methods often yield biased GE estimates in nonstandard settings, underscoring the importance of tailored GE estimation.
R. Hornung, Malte Nalenz, Lennart Schneider et al.· Statistical Science· 0 citations
As data accumulation continues to expand and information technologies evolve, machine learning methods have become widely adopted, making the effectiveness of learning algorithms crucial. Among the most popular machine learning models is Lasso regression, renowned for its feature selection capabilities and ability to address multicollinearity. This paper introduces novel algorithms for estimating Lasso regression parameters by reformulating the problem as an inverse single-point optimization task. Two algorithms are proposed: Lasso-I, which implements coordinate descent with L1 regularization, and Lasso-H, a hybrid approach that combines Lasso-I with wrapper techniques for feature selection using information criteria. The iterative algorithms involve calculating partial derivatives and selecting arguments for adjustment based on residual sum of squares or information criteria. Algorithm evaluation was performed using linear and logistic regression models across diverse datasets from KEEL and UCI repositories, alongside various metrics including the AIC, MSE, and
R
2
. The experimental results demonstrate that the algorithms effectively address parameter estimation problems, with Lasso-H achieving optimal AIC values in 90% of logistic regression cases. The proposed methods eliminate the need for explicit regularization parameter specification while maintaining robust feature selection capabilities and effective multicollinearity mitigation, demonstrating high accuracy and reliability across high-dimensional datasets.
Recent proliferation of data-optimization integration has led to a range of methods that aim to improve the statistical performance of data-driven optimization decisions. However, while many of these methods are motivated intuitively from a robustness or regularization perspective, their resulting statistical benefits are often unclear and, even if available, are established on a case-by-case basis. We provide a systematic dissection of data-driven optimization formulations using the view of"directionally perturbed"empirical optimization (EO). Specifically, this umbrella of formulations, which we call"EO+", covers many existing data-driven optimization methods, including regularization, distributionally robust optimization, transfer learning, and analogous methods for contextual optimization. On the one hand, we argue that without additional, correctly specified, side information, any EO+ method can result in at most second-order improvements. This provides a negative conclusion, namely ``no free lunch is possible", on the statistical power of EO+. On the other hand, we show that when leveraging side information that is geometrically effective, achieving first-order improvements is possible by choosing hyperparameters that are significantly larger than what is typically suggested in the literature. Moreover, we construct a principled methodology based on excess risk estimation, via either system knowledge or bootstrap resampling, to maximize the first-order gain. We demonstrate how this gain connects to the control-variate principle, a variance reduction technique in the Monte Carlo simulation literature, which helps explain why geometrically effective side information is necessary.
Real-world regression problems often involve noise, redundancy, multicollinearity, and nonlinear relationships that limit the effectiveness of classical models. This study investigates Type-1 Fuzzy Functions (T1FF) combined with several feature selection strategies, with particular emphasis on the integration of Lasso regression into the T1FF framework, which has not been directly examined in prior research. By incorporating Fuzzy C-Means-based membership degrees into the modelling process, T1FF provides a flexible way to capture uncertainty and nonlinear structure without relying on expert-defined fuzzy rules. The proposed framework was evaluated on six datasets, namely Boston, Auto, College, Steel Fatigue Strength, Fish Price, and Smart Pressure Control, using RMSE and MAPE as performance criteria. The results show that T1FF-based models generally outperform classical LM, Ridge, and Lasso models on most datasets, although the best-performing T1FF variant varied depending on dataset characteristics. In particular, the Lasso-based T1FF model yielded competitive results overall and achieved especially strong performance on the College and Fish Price datasets, while Full and Forward T1FF methods showed the most consistent MAPE-based ranking across datasets. Overall, the findings indicate that integrating feature selection and regularization methods into the T1FF framework provides a promising and flexible approach for regression modelling on complex real-world data.
M. Şahin, N. Tak· Karadeniz Fen Bilimleri Derg...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
