Chris Mingard

Chris Mingard

DPhil student studying generalisation in neural networks, Oxford University

Posts by Collection

publications

Neural networks are a priori biased towards Boolean functions with low entropy

Published in arxiv, 2019

Understanding the inductive bias of neural networks is critical to explaining their ability to generalise. Here, for one of the simplest neural networks – a single-layer perceptron with n input neurons, one output neuron, and no threshold bias term – we prove that upon random initialisation of weights, the a priori probability $P(t)$ that it represents a Boolean function that classifies $t$ points in ${0,1}^n$ as $1$ has a remarkably simple form: $P(t) = 2^{-n}$ for $0\leq t < 2^n$

Recommended citation: Mingard, Chris, et al. "Neural networks are a priori biased towards boolean functions with low entropy." arXiv preprint arXiv:1909.11522 (2019).

Is SGD a Bayesian Sampler?

Published in JMLR, 2021

This paper investigates how similar SGD trained networks are to their Gaussian Processes

Recommended citation: Mingard, Chris, et al. "Is SGD a Bayesian sampler? Well, almost." The Journal of Machine Learning Research 22.1 (2021): 3579-3642.

Feature Learning and Signal Propagation in Deep Neural Networks

Published in ICML 2022, 2022

Recent work by Baratin et al. (2021) sheds light on an intriguing pattern that occurs during the training of deep neural networks: some layers align much more with data compared to other layers

Recommended citation: Lou, Yizhang, Chris E. Mingard, and Soufiane Hayou. "Feature Learning and Signal Propagation in Deep Neural Networks." International Conference on Machine Learning. PMLR, 2022.

Automatic Gradient Descent: Deep Learning without Hyperparameters

Published in arxiv, 2023

Automatic gradient descent trains both fully-connected and convolutional networks out-of-the-box and at ImageNet scale without hyperparameters

Recommended citation: Jeremy Bernstein, Chris Mingard, Kevin Huang, Navid Azizan, Yisong Yue. "Automatic Gradient Descent: Deep Learning without Hyperparameters." arXiv preprint arXiv:2304.05187 (2023).