A Combinator Library for Neural Network Implementation

Abstract

The field of Neural Networks has evolved quickly, and there is a dizzying variety of algorithm variants. The concepts presented in publications, by comparison, are often diagrammatic, capturing the intention, which leads to a conceptual gap between the implementation and the explanation. This gap in conception and implementation also adds to issues in reproducing the results reported in many papers, due to difficulties in reimplementing the networks. In this paper, we transfer the concept of a combinator from Functional Programming and apply it to architectural concepts in neural network design. We show that many design concepts in neural networks are variations on common patterns and how they can be implemented using common underlying combinators within the PyTorch framework. We discuss the types of these combinators and how understanding these patterns can drive improved reliability in neural network implementation. Finally, we test our combinator-based approach by reimplementing ResNet34 and the Gated Recurrent Unit.