Abstract

Machine learning is data hungry. Consider the massive contrast between general relativity, which is still making accurate predictions about the extreme behavior of black holes over a century after it was originally proposed, and the poor out-of-distribution predictions of a deep neural network trained from scratch on a small dataset. While machine learning is having a fantastic year in the physical sciences, picking up two Nobel prizes, many of the scientific problems we wish to solve have very limited training data available.

Now, the boom of Large Language Models from 2022 onwards has demonstrated the power of increased scale, more general models, and more diverse pretraining. Motivated by this ongoing confirmation of Sutton’s bitter lesson, PolymathicAI, a research collaboration mostly split between the University of Cambridge and Flatiron Institute, aims to build industry-scale foundation models for scientific tasks. In particular, our foundation models are specifically pretrained to work on numerical datasets, rather than language. I will present this initiative, describe our ongoing work and upcoming releases, and give some general comments about foundation model-based machine learning for science.