Train with Ramps, Deploy with Ranks: Tiny, robust rank-order filters that compete with neural networks

Abstract

A median filter removes impulsive noise by sorting a few neighbouring samples and picking the middle one. Filters of this family (rank-order or order-statistic filters) need no multiplications, cannot be thrown off by a few wild samples, and their output never moves more than their input does. They were popular in the 1980s and 1990s, but they are hard to train: picking “the third largest value” does not change smoothly when the parameters change, so gradient descent gets no signal. We revisit a 1993 thesis on adaptive cascades of such filters with a simple idea: during training each sample is pread as a small ramp between its sorted neighbours, which makes the output smooth; for deployment the ramp is removed and the model is an exact rank filter again. We compare the resulting models (WOS-R) with convolutional networks, multilayer perceptrons and Transformers on signals, images, radar detection and federated learning. With 16 to 213 parameters and almost no multiplications, WOS-R comes within 0.5–1.2dB of networks that are 160 to 1400 times larger, and it is clearly better when the noise changes in ways not seen during training. It is not a universal replacement: for Gaussian noise or operators that need subtraction, networks remain better.