Stanford University have set up a new group to study what they are calling foundation models.

Large scale models, trained on massive datasets, are changing the way ML is done. These include language models like GPT-3 and Bert, vision models like DALLE and audio models like Wav2Vec2. The models have some properties that making them incredibly powerful, such as generalising to new tasks with small amounts of training data.

Stanford recently held a workshop (accompanied by a paper) where these were discussed by some leading academics. All of the talks are available online and are worth a watch. There are multiple issues around these models, including the biases they can perpetuate and questions over who owns and controls them. So far the most advanced models have been created by deep pocketed private companies. It is welcome that academics are beginning to focus on making our understanding of these models more rigorous.