Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
A depth hierarchy for ReLU neural networks in which every additional ReLU layer can save exponentially many neurons is proved, and the first exponential separation for ReLU networks between two fixed depths whose shallower network has depth at least $3 is proved.
Itay Safran
· 0 citations