Overall architecture

Today’s LLMs are, in some way or the other, based on an architecture known as the Transformer. This architecture was originally introduced in a paper catchily titled “Attention is all you need” (Vaswani et al. 2017). Of course, this was not the first attempt at automating natural-language generation – not even in deep learning, the sub-type of machine learning whose defining characteristic are many-layered (“deep”) artificial neural networks. But there, in deep learning, it constituted some kind of paradigm change. Before, models designed to solve sequence-prediction tasks (time-series forecasting, text generation…) tended to be based on some form of recurrent architecture, introduced in the 1990’s (eternities ago, on the time scale of deep-learning) by (Hochreiter and Schmidhuber 1997). Basically, the concept of recurrence, with its associated threading of a latent state, was replaced by “attention”. That’s what the paper’s title was meant to communicate: The authors did not introduce “attention”²; instead, they fundamentally expanded its usage so as to render recurrence superfluous.

How did that ancestral Transformer look? – One prototypical task in natural language processing is machine translation. In translation, be it done by a machine or by a human, there is an input (in one language) and an output (in another). That input, call it a code. Whoever wants to establish its counterpart in the target language first needs to decode it. Indeed, one of two top-level building blocks of the archetypal Transformer was a decoder, or rather, a stack of decoders applied in succession. At its end, out popped a phrase in the target language³. What, then, was the other high-level block? It was an encoder, something that takes text (or tokens, rather, i.e., something that has undergone tokenization) and converts it into a form the decoder can make sense of. (Obviously, there is no analogue to this in human translation.)

From this two-stack architecture, subsequent developments tended to keep just one. The GPT family, together with many others, just kept the decoder stack. Now, doesn’t the decoder need some kind of input – if not to translate to a different language, then to reply to, as in the chatbot scenario? Turns out that no, it doesn’t – and that’s why you can also have the bot initiate the conversation. Unbeknownst to you, there will, in fact, be an input to the model – some kind of token signifying “end of input”. In that case, the model will draw on its training experience to generate a word likely to start out a phrase. That one word will then become the new input to continue from, and so forth. Summing up so far, then, GPT-like LLMs are Transformer Decoders.

The question is, how does such a stack of decoders succeed in fulfilling the task?

GPT-type models up close

In opening the black box, we focus on its two interfaces – input and output – as well as on the internals, its core.

Mechanics of text generation

We already know that during model training, predictions are generated word-by-word; at every step, the model’s knowledge about what has been said so far is augmented by one token: the word that really was following at that point. If, making use of a trained model, a bot is asked to reply to a question, its response must by necessity be generated in the same way. However, the actual “correct word” is not known. The only way, then, is to feed back to the model its own most recent prediction. (By necessity, this lends to text generation a very special character, where every decision the bot makes co-determines its future behavior.)

Why, though, talk about decisions? Doesn’t the bot just act on behalf of the core model, the LLM – thus passing on the final output? Not quite. At each prediction step, the model yields a vector, with values as many as there are entries in the vocabulary. As per model design and training rationale, these vectors are “scores” – ratings, sort of, how good a fit a word would be in this situation. Like in life, higher is better. But that doesn’t mean you’d just pick the word with the highest value. In any case, these scores are converted to probabilities, and a suitable probability distribution is used to non-deterministically pick a likely (or likely-ish) word. The probability distribution commonly used is the multinomial distribution, appropriate for discrete choice among more than two alternatives. But what about the conversion to probabilities? Here, there is room for experimentation.

Technically, the algorithm employed is known as the softmax function. It is a simplified version of the Boltzmann distribution, famous in statistical mechanics, used to obtain the probability of a system’s state given that state’s energy and the temperature of the system. But for temperature¹⁰, both formulae are, in fact, identical. In physical systems, temperature modulates probabilities in the following way: The hotter the system, the closer the states’ probabilities are to each other; the colder it gets, the more distinct those probabilities. In the extreme, at very low temperatures there will be a few clear “winners” and a silent majority of “losers”.

In deep learning, a like effect is easy to achieve (by means of a scaling factor). That’s why you may have heard people talk about some weird thing called “temperature” that resulted in [insert adjective here] answers. If the application you use lets you vary that factor, you’ll see that a low temperature will result in deterministic-looking, repetitive, “boring” continuations, while a high one may make the machine appear as though it were on drugs.

That concludes our high-level overview of LLMs. Having seen the machine dissected in this way may already have left you with some sort of opinion of what these models are – not. This topic more than deserves a dedicated exposition – and papers are being written pointing to important aspects all the time – but in this text, I’d like to at least offer some input for thought.

A few facts

I’ll start out briefly mentioning empirical results, conclusive thought experiments, and theoretical considerations. All aspects touched upon (and many more) are more than worthy of in-depth discussion, but such discussion is clearly out of scope for this synoptic-in-character text.

First, while it is hard to put a number on the quality of a chatbot’s answers, performance on standardized benchmarks is the “bread and butter” of machine learning – its reporting being an essential part of the prototypical deep-learning publication. (You could even call it the “cookie”, the driving incentive, since models usually are explicitly trained and fine-tuned for good results on these benchmarks.) And such benchmarks exist for most of the down-stream tasks the LLMs are used for: machine translation, generating summaries, text classification, and even rather ambitious-sounding setups associated with – quote/unquote – reasoning.

How do you assess such a capability? Here is an example from a benchmark named “Argument Reasoning Comprehension Task” (Habernal et al. 2018).

Claim: Google is not a harmful monopoly
Reason: People can choose not to use Google
Warrant: Other search engines don’t redirect to Google
Alternative: All other search engines redirect to Google

Here claim and reason together make up the argument. But what, exactly, is it that links them? At first look, this can even be confusing to a human. The missing link is what is called warrant here – add it in, and it all starts to make sense. The task, then, is to decide which of warrant or alternative supports the conclusion, and which one does not.

If you think about it, this is a surprisingly challenging task. Specifically, it seems to inescapingly require world knowledge. So if language models, as has been claimed, perform nearly as well as humans, it seems they must have such knowledge – no quotes added. However, in response to such claims, research has been performed to uncover the hidden mechanism that enables such seemingly-superior results. For that benchmark, it has been found (Niven and Kao 2019) that there were spurious statistical cues in the way the dataset was constructed – those removed, LLM performance was no better than random.

World knowledge, in fact, is one of the main things an LLM lacks. Bender et al. (Bender and Koller 2020) convincingly demonstrate its essentiality by means of two thought experiments. One of them, situated on a lone island, imagines an octopus¹¹ inserting itself into some cable-mediated human communication, learning the chit-chat, and finally – having gotten bored – impersonating one of the humans. This works fine, until one day, its communication partner finds themselves in an emergency, and needs to build some rescue tool out of things given in the environment. They urgently ask for advice – and the octopus has no idea what to respond. It has no ideas what these words actually refer to.

The other argument comes directly from machine learning, and strikingly simple though it may be, it makes its point very well. Imagine an LLM trained as usual, including on lots of text involving plants. It has also been trained on a dataset of unlabeled photos, the actual task being unsubstantial – say it had to fill out masked areas. Now, we pull out a picture and ask: How many of that blackberry’s blossoms have already opened? The model has no chance to answer the question.

Now, please look back at the Joseph Weizenbaum quote I opened this article with. It is still true that language-generating machine have no knowledge of the world we live in.

Before moving on, I’d like to just quickly hint at a totally different type of consideration, brought up in a (2003!) paper by Spärck Jones (Spaerck 2004). Though written long before LLMs, and long before deep learning started its winning conquest, on an abstract level it is still very applicable to today’s situation. Today, LLMs are employed to “learn language”, i.e., for language acquisition. That skill is then built upon by specialized models, of task-dependent architecture. Popular real-world¹² down-stream tasks are translation, document retrieval, or text summarization. When the paper was written, there was no such two-stage pipeline. The author was questioning the fit between how language modeling was conceptualized – namely, as a form of recovery – and the character of these down-stream tasks. Was recovery – inferring a missing, for whatever reasons – piece of text a good model, of, say, condensing a long, detailed piece of text into a short, concise, factual one? If not, could the reason it still seemed to work just fine be of a very different nature – a technical, operational, coincidental one?

It seems to me that today’s two-stage process notwithstanding, this is still an aspect worth giving some thought.

It’s us: Language learning, shared goals, and a shared world

We’ve already talked about world knowledge. What else are LLMs missing out on?

In our world, you’ll hardly find anything that does not involve other people. This goes a lot deeper than the easily observable facts: our constantly communicating, reading and typing messages, documenting our lives on social networks… We don’t experience, explore, explain a world of our own. Instead, all these activities are inter-subjectively constructed. Feelings are¹⁴. Cognition is; meaning is. And it goes deeper yet. Implicit assumptions guide us to constantly look for meaning, be it in overheard fragments, mysterious symbols, or life events.

How does this relate to LLMs? For one, they’re islands of their own. When you ask them for advice – to develop a research hypothesis and a matching operationalization, say, or whether a detainee should be released on parole – they have no stakes in the outcome, no motivation (be it intrinsic or extrinsic), no goals. If an innocent person is harmed, they don’t feel the remorse; if an experiment is successful but lacks explanatory power, they don’t sense the shallowness; if the world blows up, it won’t have been their world.

Secondly, it’s us who are not islands. In Bender et al.’s octopus scenario, the human on one side of the cable plays an active role not just when they speak. In making sense of what the octopus says, they contribute an essential ingredient: namely, what they think the octopus wants, thinks, feels, expects… Anticipating, they reflect on what the octopus anticipates.

As Bender et al. put it:

It is not that O’s utterances make sense, but rather, that A can make sense of them.

That article (Bender and Koller 2020) also brings impressive evidence from human language acquisition: Our predisposition towards language learning notwithstanding, infants don’t learn from the availability of input alone. A situation of joint attention is needed for them to learn. Psychologizing, one could hypothesize they need to get the impression that these sounds, these words, and the fact they’re linked together, actually matters.

Let me conclude, then, with my final “psychologization”.