Gil Ben-Artzi

Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

This work introduces Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections and matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.