Jul 2026· Mathematical biosciences and engineering : MBE· Vol 23 7, pp.
2018-2054
· 0 citations· 25 references
Medicine
Abstract
In this paper, we introduce a new optimization algorithm that is well suited to solve parameter estimation problems that arise when inferring heterogeneous population dynamics. In these estimation problems, parameter estimation is complicated by the presence of two types of constraints: inequality constraints (e.g., non-negativity and boundedness of rates (so-called box-constraints)) and equality constraints that arise due to the need of the population fractions to sum to one. We call our new method cubic regularized Newton with affine scaling (CRNAS). In contrast to so-called first-order methods, which solely rely on the gradient of the objective function, our method utilizes the Hessian of the objective. As a result, it is able to focus on points that satisfy the second-order optimality conditions, as opposed to first-order methods that simply converge to critical points. This is an important feature in parameter estimation problems, where the objective function is often non-convex; as a result, there can be many critical points, which makes it nearly impossible to identify the global minimum. We use an affine scaling approach to handle a wide class of constraints, including equality constraints. We establish that CRNAS identifies a point that satisfies $ \epsilon $-approximate second-order optimality conditions within $ O(\epsilon^{-3/2}) $ iterations. Finally, we compare CRNAS with MATLAB's optimization solver fmincon on three different test problems. These test problems all feature mixtures of heterogeneous populations, a problem setting that CRNAS is particularly well-suited for. Our numerical simulations show that CRNAS has a favorable performance, thereby performing comparable, if not better than, fmincon in accuracy and computational cost for most of our examples.
This paper proposes a novel decentralized stochastic first-order optimization algorithm, which does not require second-order Hessian or Jacobian matrices, for the setting where the lower-level loss function is nonconvex but satisfies the Polyak–Łojasiewicz (PL) condition.
Yihan Zhang, Xinwen Zhang, My T. Thai et al.· 0 citations
This paper proposes a tractable stochastic approach based on an entropic regularization of the distributionally robust value function, which makes it possible to compute stochastic gradient estimators, and the combination of these estimators with a stochastic Frank-Wolfe algorithm, allowing us to optimize the regularized robust objective while naturally handling constraints.
A randomized two-point direct-search algorithm for nonconvex time-varying optimization and derive iteration-complexity bounds under both constant and diminishing probing ratios, which recover the complexity of existing zeroth-order methods in the time-invariant setting while extending direct- search methods beyond static settings.
This work considers a quadratic minmax problem with coupled inner constraints and proposes a method to compute a class of stationary points and shows in particular that the method is polynomial in the special case where the inner feasible set of the authors' constrained minmax problem is independent from outer variables.
S. Cipolla, O. Stein, Alain B. Zemkoho· 0 citations
We consider the task of optimizing smooth, possibly non-convex functions of a matrix variable given access only to directional derivatives rather than full gradients. This setting arises when fine-tuning large neural networks on consumer-grade hardware: the network's weights are matrices, memory constraints rule out backward-mode automatic differentiation, but directional derivatives remain available through forward mode. We frame gradient estimation in this setting as a structured recovery problem, in the spirit of signal processing. From this perspective we provide three contributions. First, we introduce an alternative to the standard random gradient estimator; the difference corresponds to replacing the adjoint of the sampling operator with its pseudoinverse. Second, when the gradient satisfies an approximate low-rank condition, techniques from matrix sensing yield a family of highly accurate gradient estimators that drop into any first-order method. Third, we note that while the computational cost of such estimators is high, this can be amortized by combining them with a matrix-aware optimizer such as spectral descent. Specifically, the gradient estimator computes a factorization of the gradient, allowing for the projection step of spectral descent to be done at no extra cost. We demonstrate our findings with two careful numerical experiments on synthetic functions with approximately low-rank gradients. We show that by exploiting this low-rank property one obtains much faster convergence to good approximate solutions.
Sawyer Allen, Cash Cherry, Aidan Eck et al.· 0 citations
This paper studies the convergence of stochastic gradient descent when the implemented updates are subject to a persistent and state-dependent bias, in which the desired update is scaled by response functions component-wise, and proposes a gradient-based algorithm, termed Residual Learning.
Zhaoxian Wu, Quan Xiao, Tayfun Gokmen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
