Page Not Found
Page not found. Your pixels are in another canvas.
Sitemap
A list of all the posts and pages found on the site. For you robots out there is an XML version available for digesting as well.
Pages
Research Interests
About me
Page not in menu
This is a page not in th emain menu
Posts
Neural networks are fundamentally (almost) Bayesian Permalink
Published:
Deep neural networks (DNNs) have been extraordinarily successful in many different situations — from image recognition and playing chess to driving cars and making medical diagnoses. However, in spite of this success, a good theoretical understanding of why they generalise (learn) so well is still lacking. /posts/2020/SGDBayes/
publications
Neural networks are a priori biased towards Boolean functions with low entropy
Published in arxiv, 2019
Understanding the inductive bias of neural networks is critical to explaining their ability to generalise. Here, for one of the simplest neural networks – a single-layer perceptron with n input neurons, one output neuron, and no threshold bias term – we prove that upon random initialisation of weights, the a priori probability $P(t)$ that it represents a Boolean function that classifies $t$ points in ${0,1}^n$ as $1$ has a remarkably simple form: $P(t) = 2^{-n}$ for $0\leq t < 2^n$
Recommended citation: Mingard, Chris, et al. "Neural networks are a priori biased towards boolean functions with low entropy." arXiv preprint arXiv:1909.11522 (2019).
Is SGD a Bayesian Sampler?
Published in JMLR, 2021
This paper investigates how similar SGD trained networks are to their Gaussian Processes
Recommended citation: Mingard, Chris, et al. "Is SGD a Bayesian sampler? Well, almost." The Journal of Machine Learning Research 22.1 (2021): 3579-3642.
Feature Learning and Signal Propagation in Deep Neural Networks
Published in ICML 2022, 2022
Recent work by Baratin et al. (2021) sheds light on an intriguing pattern that occurs during the training of deep neural networks: some layers align much more with data compared to other layers
Recommended citation: Lou, Yizhang, Chris E. Mingard, and Soufiane Hayou. "Feature Learning and Signal Propagation in Deep Neural Networks." International Conference on Machine Learning. PMLR, 2022.
Automatic Gradient Descent: Deep Learning without Hyperparameters
Published in arxiv, 2023
Automatic gradient descent trains both fully-connected and convolutional networks out-of-the-box and at ImageNet scale without hyperparameters
Recommended citation: Jeremy Bernstein, Chris Mingard, Kevin Huang, Navid Azizan, Yisong Yue. "Automatic Gradient Descent: Deep Learning without Hyperparameters." arXiv preprint arXiv:2304.05187 (2023).
Do deep neural networks have an inbuilt Occam’s razor?
Published in arxiv, 2023
Occams razor in neural networks
Recommended citation: Test