Author: David Poole, Professor Emeritus of Computer Science, University of British Columbia

In 1948, the founder of information theory, Claude Shannon, proposed modelling language in terms of the probability of the next word in a sentence given the previous words. These types of probabilistic language models were largely derided, most famously by linguist Noam Chomsky: “The notion of ‘probability of a sentence’ is an entirely useless one.” In 2022, 74 years after Shannon’s proposal, ChatGPT appeared, which caught the attention of the public, with some even suggesting it was a gateway to super-human intelligence. Going from Shannon’s proposal to ChatGPT took so long because the amount of data and computing time used…