divergences

What are Large Language Models? What are they not?

Sigrid Keydana — Tue, 20 Jun 2023 00:00:00 GMT

“At this writing, the only serious ELIZA scripts which exist are some which cause ELIZA to respond roughly as would certain psychotherapists (Rogerians). ELIZA performs best when its human correspondent is initially instructed to”talk” to it, via the typewriter of course, just as one would to a psychiatrist. This mode of conversation was chosen because the psychiatric interview is one of the few examples of categorized dyadic natural language communication in which one of the participating pair is free to assume the pose of knowing almost nothing of the real world. If, for example, one were to tell a psychiatrist “I went for a long boat ride” and he responded “Tell me about boats”, one would not assume that he knew nothing about boats, but that he had some purpose in so directing the subsequent conversation. It is important to note that this assumption is one made by the speaker. Whether it is realistic or not is an altogether separate question. In any case, it has a crucial psychological utility in that it serves the speaker to maintain his sense of being heard and understood. The speaker furher defends his impression (which even in real life may be illusory) by attributing to his conversational partner all sorts of background knowledge, insights and reasoning ability. But again, these are the speaker’s contribution to the conversation.”

Joseph Weizenbaum, creator of ELIZA (Weizenbaum 1966).

Weizenbaum, Joseph. 1966. “ELIZA - a Computer Program for the Study of Natural Language Communication Between Man and Machine.” Commun. ACM (New York, NY, USA) 9 (1): 36–45. https://doi.org/10.1145/365153.365168.

GPT, the ancestor all numbered GPTs, was released in June, 2018 – five years ago, as I write this. Five years: that’s a long time. It certainly is as measured on the time scale of deep learning, the thing that is, usually, behind when people talk of “AI”. One year later, GPT was followed by GPT-2; another year later, by GPT-3. At this point, public attention was still modest – as expected, really, for these kinds of technologies that require lots of specialist knowledge. (For GPT-2, what may have increased attention beyond the normal, a bit, was OpenAI ’s refusal to publish the complete training code and full model weights, supposedly due to the threat posed by the model’s capabilities – alternatively, as argued by others, as a marketing strategy, or yet alternatively, as a way to preserve one’s own competitive advantage just a tiny little bit longer.

As of 2023, with GPT-3.5 and GPT-4 having followed, everything looks different. (Almost) everyone seems to know GPT, at least when that acronym appears prefixed by a certain syllable. Depending on who you talk to, people don’t seem to stop talking about that fantastic [insert thing here] ChatGPT generated for them, about its enormous usefulness with respect to [insert goal here]… or about the flagrant mistakes it made, and the danger that legal regulation and political enforcement will never be able to catch up.

What made the difference? Obviously, it’s ChatGPT, or put differently, the fact that now, there is a means for people to make active use of such a tool, employing it for whatever their personal needs or interests are¹. In fact, I’d argue it’s more than that: ChatGPT is not some impersonal tool – it talks to you, picking up your clarifications, changes of topic, mood… It is someone rather than something, or at least that’s how it seems. I’ll come back to that point in It’s us, really: Anthropomorphism unleashed. Before, let’s take a look at the underlying technology.

¹ Evidently, this is not about singling out ChatGPT as opposed to other chatbots; rather, I’m adopting it as the prototypical such application, since it is the one omnipresent in the media these days.

Large Language Models: What they are

How is it even possible to build a machine that talks to you? One way is to have that machine listen a lot. And listen is what these machines do; they do it a lot. But listening alone would never be enough to attain results as impressive as those we see. Instead, LLMs practice some form of “maximally active listening”: Continuously, they try to predict the speaker’s next utterance. By “continuously”, I mean word-by-word: At each training step, the model is asked to produce the subsequent word in a text.

Maybe in my last sentence, you noted the term “train”. As per common sense, “training” implies some form of supervision. It also implies some form of method. Since learning material is scraped from the internet, the true continuation is always known. The precondition for supervision is thus always fulfilled: A supervisor can just compare model prediction with what really follows in the text. Remains the question of method. That’s where we need to talk about deep learning, and we’ll do that in Model training.

Overall architecture

Today’s LLMs are, in some way or the other, based on an architecture known as the Transformer. This architecture was originally introduced in a paper catchily titled “Attention is all you need” (Vaswani et al. 2017). Of course, this was not the first attempt at automating natural-language generation – not even in deep learning, the sub-type of machine learning whose defining characteristic are many-layered (“deep”) artificial neural networks. But there, in deep learning, it constituted some kind of paradigm change. Before, models designed to solve sequence-prediction tasks (time-series forecasting, text generation…) tended to be based on some form of recurrent architecture, introduced in the 1990’s (eternities ago, on the time scale of deep-learning) by (Hochreiter and Schmidhuber 1997). Basically, the concept of recurrence, with its associated threading of a latent state, was replaced by “attention”. That’s what the paper’s title was meant to communicate: The authors did not introduce “attention”²; instead, they fundamentally expanded its usage so as to render recurrence superfluous.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. Attention Is All You Need. https://arxiv.org/abs/1706.03762.

Hochreiter, Sepp, and Jürgen Schmidhuber. 1997. “Long Short-Term Memory.” Neural Computation 9 (December): 1735–80. https://doi.org/10.1162/neco.1997.9.8.1735.

² I’m using quotes to refer to how attention is operationalized in deep learning, as opposed to how it is conceptualized in cognitive science or psychology.

³ If you’re wondering how that is possible – shouldn’t there be a separate, top-level module for generation? – no, there need not be. That’s because training implies prediction.

How did that ancestral Transformer look? – One prototypical task in natural language processing is machine translation. In translation, be it done by a machine or by a human, there is an input (in one language) and an output (in another). That input, call it a code. Whoever wants to establish its counterpart in the target language first needs to decode it. Indeed, one of two top-level building blocks of the archetypal Transformer was a decoder, or rather, a stack of decoders applied in succession. At its end, out popped a phrase in the target language³. What, then, was the other high-level block? It was an encoder, something that takes text (or tokens, rather, i.e., something that has undergone tokenization) and converts it into a form the decoder can make sense of. (Obviously, there is no analogue to this in human translation.)

From this two-stack architecture, subsequent developments tended to keep just one. The GPT family, together with many others, just kept the decoder stack. Now, doesn’t the decoder need some kind of input – if not to translate to a different language, then to reply to, as in the chatbot scenario? Turns out that no, it doesn’t – and that’s why you can also have the bot initiate the conversation. Unbeknownst to you, there will, in fact, be an input to the model – some kind of token signifying “end of input”. In that case, the model will draw on its training experience to generate a word likely to start out a phrase. That one word will then become the new input to continue from, and so forth. Summing up so far, then, GPT-like LLMs are Transformer Decoders.

The question is, how does such a stack of decoders succeed in fulfilling the task?

GPT-type models up close

In opening the black box, we focus on its two interfaces – input and output – as well as on the internals, its core.

Input

For simplicity, let me speak of words, not tokens. Now imagine a machine that is to work with – more even: “understand”⁴ – words. For a computer to process non-numeric data, a conversion to numbers necessarily has to happen. The straightforward way to effectuate this is to decide on a fixed lexicon, and assign each word a number. And this works: The way deep neural networks are trained, they don’t need semantic relationships to exist between entities in the training data to memorize formal structure. Does this mean they will appear perfect while training, but fail in real-world prediction? – If the training data are representative of how we converse, all will be fine. In a world of perfect surveillance, machines could exist that have internalized our every spoken word. Before that happens, though, the training data will be imperfect.

⁴ Why the quotes? See Large Language Models: What they are not.

⁵ As a fascinating example from dynamical systems theory, take delay coordinate embeddings.

⁶ Suitably named embedding layer.

⁷ See, for example, (Caliskan et al. 2022).

Caliskan, Aylin, Pimparkar Parth Ajay, Tessa Charlesworth, Robert Wolfe, and Mahzarin R. Banaji. 2022. “Gender Bias in Word Embeddings.” Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, July. https://doi.org/10.1145/3514094.3534162.

A much more promising approach than to simply index words, then, is to represent them in a richer, higher-dimensional space, an embedding space. This idea, popular not just in deep learning but in natural language processing overall, really goes far beyond anything domain-specific – linguistic entities, say⁵. You may be able to fruitfully employ it in virtually any domain – provided you can devise a method to sensibly map the given data into that space. In deep learning, these embeddings are obtained in a clever way: as a by-product of sorts of the overall training workflow. Technically, this is achieved by means of a dedicated neural-network layer⁶ tasked with evolving these mappings. Note how, smart though this strategy may be, it implies that the overall setting – everything from training data via model architecture to optimization algorithms employed – necessarily affects the resulting embeddings. And since these may be extracted and made use of in down-stream tasks, this matters⁷.

As to the GPT family, such an embedding layer constitutes part of its input interface – one “half”, so to say. Technically, the second makes use of the same type of layer, but with a different purpose. To contrast the two, let me spell out clearly what, in the part we’ve talked about already, is getting mapped to what. The mapping is between a word index – a sequence 1, 2, …, – on the one hand and a set of continuous-valued vectors of some length – 100, say – on the other. (One of them could like this: ) Thus, we obtain an embedding for every word. But language is more than an unordered assembly of words. Rearranging words, if syntactically allowed, may result in drastically changed semantics. In the pre-transformer paradigma, threading a sequentially-updated hidden state took care of this. Put differently, in that type of model, information about input order never got lost throughout the layers. Transformer-type architectures, however, need to find a different way. Here, a variety of rivaling methods exists. Some assume an underlying periodicity in semanto-syntactic structure. Others – and the GPT family, as yet and insofar we know, has been part of them⁸ – approach the challenge in exactly the same way as for the lexical units: They make learning these so-called position embeddings a by-product of model training. Implementation-wise, the only difference is that now the input to the mapping looks like this: 1, 2, …, where “maximum position” reflects choice of maximal sequence length supported.

⁸ For GPT-4, even high-level model information has not been released.

Summing up, verbal input is thus encoded – embedded, enriched – twofold as it enters the machine. The two types of embedding are combined and passed on to the model core, the already-mentioned decoder stack.

Core Processing

The decoder stack is made up of some number of identical blocks (12, in the case of GPT-2). (By “identical” I mean that the architecture is the same; the weights – the place where a neural-network layer stores what it “knows” – are not. More on these “weights” soon.)

Inside each block, some sub-layers are pretty much “business as usual”. One is not: the attention module, the “magic” ingredient that enabled Transformer-based architectures to forego keeping a latent state. To explain how this works, let’s take translation as an example.

In the classical encoder-decoder setup, the one most intuitive for machine translation, imagine the very first decoder in the stack of decoders. It receives as input a length-seven cypher, the encoded version of an original length-seven phrase. Since, due to how the encoder blocks are built, input order is conserved, we have a faithful representation of source-language word order. In the target language, however, word order can be very different. A decoder module, in producing the translation, had rather not do this by translating each word as it appears. Instead, it would be desirable for it to know which among the already-seen tokens is most relevant right now, to generate the very next output token. Put differently, it had better know where to direct its attention.

Thus, figure out how to distribute focus is what attention modules do. How do they do it? They compute, for each available input-language token, how good a match, a fit, it is for their own current input. Remember that every token, at every processing stage, is encoded as a vector of continuous values. How good a match any of, say, three source-language vectors is is then computed by projecting one’s current input vector onto each of the three. The closer the vectors, the longer the projected vector. ⁹ Based on the projection onto each source-input token, that token is weighted, and the attention module passes on the aggregated assessments to the ensuing neural-network module.

⁹ Mathematically, this is achieved by a pretty standard and pervasively-used, in machine learning, operation, the dot product.

To explain what attention modules are for, I’ve made use of the machine-translation scenario, a scenario that should lend a certain intuitiveness to the operation. But for GPT-family models, we need to abstract this a bit. First, there is no encoder stack, so “attention” is computed among decoder-resident tokens only. And second – remember I said a stack was built up of identical modules? – this happens in every decoder block. That is, when intermediate results are bubbled up the stack, at each stage the input is weighted as appropriate at that stage. While this is harder to intuit than what happened in the translation scenario, I’d argue that in the abstract, it makes a lot of sense. For an analogy, consider some form of hierarchical categorization of entities. As higher-level categories are built from lower-level ones, at each stage the process needs to look at its input afresh, and decide on a sensible way of subsuming similar-in-some-way categories.

Output

Stack of decoders traversed, the multi-dimensional codes that pop out need to be converted into something that can be compared with the actual phrase continuation we see in the training corpus. Technically, this involves a projection operation as well a strategy for picking the output word – that word in target-language vocabulary that has the highest probability. How do you decide on a strategy? I’ll say more about that in the section Mechanics of text generation, where I assume a chatbot user’s perspective.

Model training

Before we get there, just a quick word about model training. LLMs are deep neural networks, and as such, they are trained like any network is. First, assuming you have access to the so-called “ground truth”, you can always compare model prediction with the true target. You then quantify the difference – by which algorithm will affect training results. Then, you communicate that difference – the loss – to the network. It, in turn, goes through its modules, from back/top to start/bottom, and updates its stored “knowledge” – matrices of continuous numbers called weights. Since information is passed from layer to layer, in a direction reverse to that followed in computing predictions, this technique is known as back-propagation.

And all that is not triggered once, but iteratively, for a certain number of so-called “epochs”, and modulated by a set of so-called “hyper-parameters”. In practice, a lot of experimentation goes into deciding on the best-working configuration of these settings.

Mechanics of text generation

We already know that during model training, predictions are generated word-by-word; at every step, the model’s knowledge about what has been said so far is augmented by one token: the word that really was following at that point. If, making use of a trained model, a bot is asked to reply to a question, its response must by necessity be generated in the same way. However, the actual “correct word” is not known. The only way, then, is to feed back to the model its own most recent prediction. (By necessity, this lends to text generation a very special character, where every decision the bot makes co-determines its future behavior.)

Why, though, talk about decisions? Doesn’t the bot just act on behalf of the core model, the LLM – thus passing on the final output? Not quite. At each prediction step, the model yields a vector, with values as many as there are entries in the vocabulary. As per model design and training rationale, these vectors are “scores” – ratings, sort of, how good a fit a word would be in this situation. Like in life, higher is better. But that doesn’t mean you’d just pick the word with the highest value. In any case, these scores are converted to probabilities, and a suitable probability distribution is used to non-deterministically pick a likely (or likely-ish) word. The probability distribution commonly used is the multinomial distribution, appropriate for discrete choice among more than two alternatives. But what about the conversion to probabilities? Here, there is room for experimentation.

Technically, the algorithm employed is known as the softmax function. It is a simplified version of the Boltzmann distribution, famous in statistical mechanics, used to obtain the probability of a system’s state given that state’s energy and the temperature of the system. But for temperature¹⁰, both formulae are, in fact, identical. In physical systems, temperature modulates probabilities in the following way: The hotter the system, the closer the states’ probabilities are to each other; the colder it gets, the more distinct those probabilities. In the extreme, at very low temperatures there will be a few clear “winners” and a silent majority of “losers”.

¹⁰ … and the Boltzmann constant – but that being a constant, we don’t consider it here.

In deep learning, a like effect is easy to achieve (by means of a scaling factor). That’s why you may have heard people talk about some weird thing called “temperature” that resulted in [insert adjective here] answers. If the application you use lets you vary that factor, you’ll see that a low temperature will result in deterministic-looking, repetitive, “boring” continuations, while a high one may make the machine appear as though it were on drugs.

That concludes our high-level overview of LLMs. Having seen the machine dissected in this way may already have left you with some sort of opinion of what these models are – not. This topic more than deserves a dedicated exposition – and papers are being written pointing to important aspects all the time – but in this text, I’d like to at least offer some input for thought.

Large Language Models: What they are not

In part one,describing LLMs technically, I’ve sometimes felt tempted to use terms like “understanding” or “knowledge” when applied to the machine. I may have ended up using them; in that case, I’ve tried to remember to always surround them with quotes. The latter, the adding quotes, stands in contrast to many texts, even ones published in an academic context (Bender and Koller 2020). The question is, though: Why did I even feel compelled to use these terms, given I do not think they apply, in their usual meaning? I can think of a simple – shockingly simple, maybe – answer: It’s because us, humans, we think, talk, share our thoughts in these terms. When I say understand, I surmise you will know what I mean.

Now, why do I think that these machines do not understand human language, in the sense we usually imply when using that word?

A few facts

I’ll start out briefly mentioning empirical results, conclusive thought experiments, and theoretical considerations. All aspects touched upon (and many more) are more than worthy of in-depth discussion, but such discussion is clearly out of scope for this synoptic-in-character text.

First, while it is hard to put a number on the quality of a chatbot’s answers, performance on standardized benchmarks is the “bread and butter” of machine learning – its reporting being an essential part of the prototypical deep-learning publication. (You could even call it the “cookie”, the driving incentive, since models usually are explicitly trained and fine-tuned for good results on these benchmarks.) And such benchmarks exist for most of the down-stream tasks the LLMs are used for: machine translation, generating summaries, text classification, and even rather ambitious-sounding setups associated with – quote/unquote – reasoning.

How do you assess such a capability? Here is an example from a benchmark named “Argument Reasoning Comprehension Task” (Habernal et al. 2018).

Habernal, Ivan, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. “The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants.” Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (New Orleans, Louisiana), June, 1930–40. https://doi.org/10.18653/v1/N18-1175.

Claim: Google is not a harmful monopoly
Reason: People can choose not to use Google
Warrant: Other search engines don’t redirect to Google
Alternative: All other search engines redirect to Google

Here claim and reason together make up the argument. But what, exactly, is it that links them? At first look, this can even be confusing to a human. The missing link is what is called warrant here – add it in, and it all starts to make sense. The task, then, is to decide which of warrant or alternative supports the conclusion, and which one does not.

If you think about it, this is a surprisingly challenging task. Specifically, it seems to inescapingly require world knowledge. So if language models, as has been claimed, perform nearly as well as humans, it seems they must have such knowledge – no quotes added. However, in response to such claims, research has been performed to uncover the hidden mechanism that enables such seemingly-superior results. For that benchmark, it has been found (Niven and Kao 2019) that there were spurious statistical cues in the way the dataset was constructed – those removed, LLM performance was no better than random.

Niven, Timothy, and Hung-Yu Kao. 2019. “Probing Neural Network Comprehension of Natural Language Arguments.” CoRR abs/1907.07355. http://arxiv.org/abs/1907.07355.

¹¹ That choice of species is probably not a coincidence: see https://en.wikipedia.org/wiki/Cephalopod_intelligence.

World knowledge, in fact, is one of the main things an LLM lacks. Bender et al. (Bender and Koller 2020) convincingly demonstrate its essentiality by means of two thought experiments. One of them, situated on a lone island, imagines an octopus¹¹ inserting itself into some cable-mediated human communication, learning the chit-chat, and finally – having gotten bored – impersonating one of the humans. This works fine, until one day, its communication partner finds themselves in an emergency, and needs to build some rescue tool out of things given in the environment. They urgently ask for advice – and the octopus has no idea what to respond. It has no ideas what these words actually refer to.

The other argument comes directly from machine learning, and strikingly simple though it may be, it makes its point very well. Imagine an LLM trained as usual, including on lots of text involving plants. It has also been trained on a dataset of unlabeled photos, the actual task being unsubstantial – say it had to fill out masked areas. Now, we pull out a picture and ask: How many of that blackberry’s blossoms have already opened? The model has no chance to answer the question.

Now, please look back at the Joseph Weizenbaum quote I opened this article with. It is still true that language-generating machine have no knowledge of the world we live in.

Before moving on, I’d like to just quickly hint at a totally different type of consideration, brought up in a (2003!) paper by Spärck Jones (Spaerck 2004). Though written long before LLMs, and long before deep learning started its winning conquest, on an abstract level it is still very applicable to today’s situation. Today, LLMs are employed to “learn language”, i.e., for language acquisition. That skill is then built upon by specialized models, of task-dependent architecture. Popular real-world¹² down-stream tasks are translation, document retrieval, or text summarization. When the paper was written, there was no such two-stage pipeline. The author was questioning the fit between how language modeling was conceptualized – namely, as a form of recovery – and the character of these down-stream tasks. Was recovery – inferring a missing, for whatever reasons – piece of text a good model, of, say, condensing a long, detailed piece of text into a short, concise, factual one? If not, could the reason it still seemed to work just fine be of a very different nature – a technical, operational, coincidental one?

¹² As opposed to the aforementioned problems subsumed under “reasoning”, those having been constructed for research purposes.

[…] the crucial characterisation of the relationship between the input and the output is in fact offloaded in the LM approach onto the choice of training data. We can use LM for summarising because we know that some set of training data consists of full texts paired with their summaries.¹³

¹³ From (Spaerck 2004).

Spaerck, Karen. 2004. “Language Modelling’s Generative Model : Is It Rational?”

It seems to me that today’s two-stage process notwithstanding, this is still an aspect worth giving some thought.

It’s us: Language learning, shared goals, and a shared world

We’ve already talked about world knowledge. What else are LLMs missing out on?

In our world, you’ll hardly find anything that does not involve other people. This goes a lot deeper than the easily observable facts: our constantly communicating, reading and typing messages, documenting our lives on social networks… We don’t experience, explore, explain a world of our own. Instead, all these activities are inter-subjectively constructed. Feelings are¹⁴. Cognition is; meaning is. And it goes deeper yet. Implicit assumptions guide us to constantly look for meaning, be it in overheard fragments, mysterious symbols, or life events.

¹⁴ See https://lisafeldmanbarrett.com/books/how-emotions-are-made/.

How does this relate to LLMs? For one, they’re islands of their own. When you ask them for advice – to develop a research hypothesis and a matching operationalization, say, or whether a detainee should be released on parole – they have no stakes in the outcome, no motivation (be it intrinsic or extrinsic), no goals. If an innocent person is harmed, they don’t feel the remorse; if an experiment is successful but lacks explanatory power, they don’t sense the shallowness; if the world blows up, it won’t have been their world.

Secondly, it’s us who are not islands. In Bender et al.’s octopus scenario, the human on one side of the cable plays an active role not just when they speak. In making sense of what the octopus says, they contribute an essential ingredient: namely, what they think the octopus wants, thinks, feels, expects… Anticipating, they reflect on what the octopus anticipates.

As Bender et al. put it:

It is not that O’s utterances make sense, but rather, that A can make sense of them.

That article (Bender and Koller 2020) also brings impressive evidence from human language acquisition: Our predisposition towards language learning notwithstanding, infants don’t learn from the availability of input alone. A situation of joint attention is needed for them to learn. Psychologizing, one could hypothesize they need to get the impression that these sounds, these words, and the fact they’re linked together, actually matters.

Bender, Emily M., and Alexander Koller. 2020. “Climbing Towards NLU: On Meaning, Form, and Understanding in the Age of Data.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Online), July, 5185–98. https://doi.org/10.18653/v1/2020.acl-main.463.

Let me conclude, then, with my final “psychologization”.

It’s us, really: Anthropomorphism unleashed

Yes, it is amazing what these machines do. (And that makes them incredibly dangerous power instruments.) But this in no way affects the human-machine differences that have been existing throughout history, and continue to exist today. That we are inclined to think they understand, know, mean – that maybe even they’re conscious: that’s on us. We can experience deep emotions watching a movie; hope that if we just try enough, we can sense what a distant-in-evolutionary-genealogy creature is feeling; see a cloud encouragingly smiling at us; read a sign in an arrangement of pebbles.

Our inclination to anthropomorphize is a gift; but it can sometimes be harmful. And nothing of this is special to the twenty-first century.

Like I began with him, let me conclude with Weizenbaum.

Some subjects have been very hard to convince that ELIZA (with its present script) is not human.

Photo by Marjan Blan on Unsplash

Starting to think about AI Fairness

Sigrid Keydana — Thu, 15 Jul 2021 00:00:00 GMT

If you use deep learning for unsupervised part-of-speech tagging of Sanskrit ¹, or knowledge discovery in physics ², you probably don’t need to worry about model fairness. If you’re a data scientist working at a place where decisions are made about people, however, or an academic researching models that will be used to such ends, chances are that you’ve already been thinking about this topic. — Or feeling that you should. And thinking about this is hard.

¹ Srivastava et al. (2018)

Srivastava, Prakhar, Kushal Chauhan, Deepanshu Aggarwal, Anupam Shukla, Joydip Dhar, and Vrashabh Prasad Jain. 2018. “Deep Learning Based Unsupervised POS Tagging for Sanskrit.” Proceedings of the 2018 International Conference on Algorithms, Computing and Artificial Intelligence (New York, NY, USA), ACAI 2018. https://doi.org/10.1145/3302425.3302487.

² Cranmer et al. (2020)

Cranmer, Miles D., Alvaro Sanchez-Gonzalez, Peter W. Battaglia, et al. 2020. “Discovering Symbolic Models from Deep Learning with Inductive Biases.” CoRR abs/2006.11287. https://arxiv.org/abs/2006.11287.

It is hard for several reasons. In this text, I will go into just one.

The forest for the trees

Nowadays, it is hard to find a modeling framework that does not include functionality to assess fairness. (Or is at least planning to.) And the terminology sounds so familiar, as well: “calibration”, “predictive parity”, “equal true [false] positive rate”… It almost seems as though we could just take the metrics we make use of anyway (recall or precision, say), test for equality across groups, and that’s it. Let’s assume, for a second, it really was that simple. Then the question still is: Which metrics, exactly, do we choose?

In reality things are not simple. And it gets worse. For very good reasons, there is a close connection in the ML fairness literature to concepts that are primarily treated in other disciplines, such as the legal sciences: discrimination and disparate impact (both not being far from yet another statistical concept, statistical parity). Statistical parity means that if we have a classifier, say to decide whom to hire, it should result in as many applicants from the disadvantaged group (e.g., Black people) being hired as from the advantaged one(s). But that is quite a different requirement from, say, equal true/false positive rates!

So despite all that abundance of software, guides, and decision trees, even: This is not a simple, technical decision. It is, in fact, a technical decision only to a small degree.

Common sense, not math

Let me start this section with a disclaimer: Most of the sources referenced in this text appear, or are implied on the “Guidance” page of IBM’s framework AI Fairness 360. If you read that page, and everything that’s said and not said there appears clear from the outset, then you may not need this more verbose exposition. If not, I invite you to read on.

Papers on fairness in machine learning, as is common in fields like computer science, abound with formulae. Even the papers referenced here, though selected not for their theorems and proofs but for the ideas they harbor, are no exception. But to start thinking about fairness as it might apply to an ML process at hand, common language – and common sense – will do just fine. If, after analyzing your use case, you judge that the more technical results are relevant to the process in question, you will find that their verbal characterizations will often suffice. It is only when you doubt their correctness that you will need to work through the proofs.

At this point, you may be wondering what it is I am contrasting those “more technical results” with. This is the topic of the next section, where I’ll try to give a birds-eye characterization of fairness criteria and what they imply.

Situating fairness criteria

Think back to the example of a hiring algorithm. What does it mean for this algorithm to be fair? We approach this question under two – incompatible, mostly – assumptions:

The algorithm is fair if it behaves the same way independent of which demographic group it is applied to. Here demographic group could be defined by ethnicity, gender, abledness, or in fact any categorization suggested by the context.
The algorithm is fair if it does not discriminate against any demographic group.

I’ll call these the technical and societal views, respectively.

Fairness, viewed the technical way

What does it mean for an algorithm to “behave the same way” regardless of which group it is applied to?

In a classification setting, we can view the relationship between prediction () and target () as a doubly directed path. In one direction: Given true target , how accurate is prediction ? In the other: Given , how well does it predict the true class ?

Based on the direction they operate in, metrics popular in machine learning overall can be split into two categories. In the first, starting from the true target, we have recall, together with “the rates”: true positive, true negative, false positive, false negative. In the second, we have precision, together with positive (negative, resp.) predictive value.

If now we demand that these metrics be the same across groups, we arrive at corresponding fairness criteria: equal false positive rate, equal positive predictive value, etc. In the inter-group setting, the two types of metrics may be arranged under headings “equality of opportunity” and “predictive parity”. You’ll encounter these as actual headers in the summary table at the end of this text. (Said table organizes concepts from different areas into a three-category format. The overall narrative builds up towards that “map” in a bottom-up way – meaning, most entries will not make sense at this point.}

While overall, the terminology around metrics can be confusing (to me it is), these headings have some mnemonic value. Equality of opportunity suggests that people similar in real life () get classified similarly (). Predictive parity suggests that people classified similarly () are, in fact, similar ().

The two criteria can concisely be characterized using the language of statistical independence. Following Barocas et al. (2019), these are:

Separation: Given true target , prediction is independent of group membership ().
Sufficiency: Given prediction , target is independent of group membership ().

Given those two fairness criteria – and two sets of corresponding metrics – the natural question arises: Can we satisfy both? Above, I was mentioning precision and recall on purpose: to maybe “prime” you to think in the direction of “precision-recall trade-off”. And really, these two categories reflect different preferences; usually, it is impossible to optimize for both. The most famous, probably, result is due to Chouldechova (2016) : It says that predictive parity (testing for sufficiency) is incompatible with error rate balance (separation) when prevalence differs across groups. This is a theorem (yes, we’re in the realm of theorems and proofs here) that may not be surprising, in light of Bayes’ theorem, but is of great practical importance nonetheless: Unequal prevalence usually is the norm, not the exception.

Chouldechova, Alexandra. 2016. “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.” arXiv e-Prints, October, arXiv:1610.07524. https://arxiv.org/abs/1610.07524.

Yeom, Samuel, and Michael Carl Tschantz. 2018. “Discriminative but Not Discriminatory: A Comparison of Fairness Definitions Under Different Worldviews.” CoRR abs/1808.08619. http://arxiv.org/abs/1808.08619.

This necessarily means we have to make a choice. And this is where the theorems and proofs do matter. For example, Yeom and Tschantz (2018) show that in this framework – the strictly technical approach to fairness – separation should be preferred over sufficiency, because the latter allows for arbitrary disparity amplification. Thus, in this framework, we may have to work through the theorems.

What is the alternative?

Fairness, viewed as a social construct

Starting with what I just wrote: No one will likely challenge fairness being a social construct. But what does that entail?

Let me start with a biographical reminiscence. In undergraduate psychology (a long time ago), probably the most hammered-in distinction relevant to experiment planning was that between a hypothesis and its operationalization. The hypothesis is what you want to substantiate, conceptually; the operationalization is what you measure. There necessarily can’t be a one-to-one correspondence; we’re just striving to implement the best operationalization possible.

In the world of datasets and algorithms, all we have are measurements. And often, these are treated as though they were the concepts. This will get more concrete with an example, and we’ll stay with the hiring software scenario.

Assume the dataset used for training, assembled from scoring previous employees, contains a set of predictors (among which, high-school grades) and a target variable, say an indicator whether an employee did “survive” probation. There is a concept-measurement mismatch on both sides.

For one, say the grades are intended to reflect ability to learn, and motivation to learn. But depending on the circumstances, there are influence factors of much higher impact: socioeconomic status, constantly having to struggle with prejudice, overt discrimination, and more.

And then, the target variable. If the thing it’s supposed to measure is “was hired for seemed like a good fit, and was retained since was a good fit”, then all is good. But normally, HR departments are aiming for more than just a strategy of “keep doing what we’ve always been doing”.

Unfortunately, that concept-measurement mismatch is even more fatal, and even less talked about, when it’s about the target and not the predictors. (Not accidentally, we also call the target the “ground truth”.) An infamous example is recidivism prediction, where what we really want to measure – whether someone did, in fact, commit a crime – is replaced, for measurability reasons, by whether they were convicted. These are not the same: Conviction depends on more then what someone has done – for instance, if they’ve been under intense scrutiny from the outset.

Fortunately, though, the mismatch is clearly pronounced in the AI fairness literature. Friedler et al. (2016) distinguish between the construct and observed spaces; depending on whether a near-perfect mapping is assumed between these, they talk about two “worldviews”: “We’re all equal” (WAE) vs. “What you see is what you get” (WYSIWIG). If we’re all equal, membership in a societally disadvantaged group should not – in fact, may not – affect classification. In the hiring scenario, any algorithm employed thus has to result in the same proportion of applicants being hired, regardless of which demographic group they belong to. If “What you see is what you get”, we don’t question that the “ground truth” is the truth.

This talk of worldviews may seem unnecessary philosophical, but the authors go on and clarify: All that matters, in the end, is whether the data is seen as reflecting reality in a naïve, take-at-face-value way.

For example, we might be ready to concede that there could be small, albeit uninteresting effect-size-wise, statistical differences between men and women as to spatial vs. linguistic abilities, respectively. We know for sure, though, that there are much greater effects of socialization, starting in the core family and reinforced, progressively, as adolescents go through the education system. We therefore apply WAE, trying to (partly) compensate for historical injustice. This way, we’re effectively applying affirmative action, defined as

A set of procedures designed to eliminate unlawful discrimination among applicants, remedy the results of such prior discrimination, and prevent such discrimination in the future.

In the already-mentioned summary table, you’ll find the WYSIWIG principle mapped to both equal opportunity and predictive parity metrics. WAE maps to the third category, one we haven’t dwelled upon yet: demographic parity, also known as statistical parity. In line with what was said before, the requirement here is for each group to be present in the positive-outcome class in proportion to its representation in the input sample. For example, if thirty percent of applicants are Black, then at least thirty percent of people selected should be Black, as well. A term commonly used for cases where this does not happen is disparate impact: The algorithm affects different groups in different ways.

Similar in spirit to demographic parity, but possibly leading to different outcomes in practice, is conditional demographic parity ³. Here we additionally take into account other predictors in the dataset; to be precise: all other predictors. The desiderate now is that for any choice of attributes, outcome proportions should be equal, given the protected attribute and the other attributes in question. I’ll come back to why this may sound better in theory than work in practice in the next section.

³ Wachter et al. (2020b)

Summing up, we’ve seen commonly used fairness metrics organized into three groups, two of which share a common assumption: that the data used for training can be taken at face value. The other starts from the outside, contemplating what historical events, and what political and societal factors have made the given data look as they do.

Before we conclude, I’d like to try a quick glance at other disciplines, beyond machine learning and computer science, domains where fairness figures among the central topics. This section is necessarily limited in every respect; it should be seen as a flashlight, an invitation to read and reflect rather than an orderly exposition. The short section will end with a word of caution: Since drawing analogies can feel highly enlightening (and is intellectually satisfying, for sure), it is easy to abstract away practical realities. But I’m getting ahead of myself.

A quick glance at neighboring fields: law and political philosophy

In jurisprudence, fairness and discrimination constitute an important subject. A recent paper that caught my attention is Wachter et al. (2020a) . From a machine learning perspective, the interesting point is the classification of metrics into bias-preserving and bias-transforming. The terms speak for themselves: Metrics in the first group reflect biases in the dataset used for training; ones in the second do not. In that way, the distinction parallels Friedler et al. (2016) ’s confrontation of two “worldviews”. But the exact words used also hint at how guidance by metrics feeds back into society: Seen as strategies, one preserves existing biases; the other, to consequences unknown a priori, changes the world.

To the ML practitioner, this framing is of great help in evaluating what criteria to apply in a project. Helpful, too, is the systematic mapping provided of metrics to the two groups; it is here that, as alluded to above, we encounter conditional demographic parity among the bias-transforming ones. I agree that in spirit, this metric can be seen as bias-transforming; if we take two sets of people who, per all available criteria, are equally qualified for a job, and then find the whites favored over the Blacks, fairness is clearly violated. But the problem here is “available”: per all available criteria. What if we have reason to assume that, in a dataset, all predictors are biased? Then it will be very hard to prove that discrimination has occurred.

A similar problem, I think, surfaces when we look at the field of political philosophy, and consult theories on distributive justice for guidance. Heidari et al. (2018) have written a paper comparing the three criteria – demographic parity, equality of opportunity, and predictive parity – to egalitarianism, equality of opportunity (EOP) in the Rawlsian sense, and EOP seen through the glass of luck egalitarianism, respectively. While the analogy is fascinating, it too assumes that we may take what is in the data at face value. In their likening predictive parity to luck egalitarianism, they have to go to especially great lengths, in assuming that the predicted class reflects effort exerted. In the below table, I therefore take the liberty to disagree, and map a libertarian view of distributive justice to both equality of opportunity and predictive parity metrics.

Heidari, Hoda, Michele Loi, Krishna P. Gummadi, and Andreas Krause. 2018. “A Moral Framework for Understanding of Fair ML Through Economic Models of Equality of Opportunity.” CoRR abs/1809.03400. http://arxiv.org/abs/1809.03400.

In summary, we end up with two highly controversial categories of fairness criteria, one bias-preserving, “what you see is what you get”-assuming, and libertarian, the other bias-transforming, “we’re all equal”-thinking, and egalitarian. Here, then, is that often-announced table.

	Demographic parity	Equality of opportunity	Predictive parity
A.K.A. / subsumes / related concepts	statistical parity, group fairness, disparate impact, conditional demographic parity ⁴	equalized odds, equal false positive / negative rates	equal positive / negative predictive values, calibration by group
Statistical independence criterion ⁵	independence	separation	sufficiency
Individual / group	group	group (most) or individual (fairness through awareness)	group
Distributive Justice	egalitarian	libertarian (contra Heidari et al., see above)	libertarian (contra Heidari et al., see above)
Effect on bias ⁶	transforming	preserving	preserving
Policy / “worldview” ⁷	We’re all equal (WAE)	What you see is what you get (WYSIWIG)	What you see is what you get (WYSIWIG)

⁴ Wachter et al. (2020b)

Wachter, Sandra, Brent D. Mittelstadt, and Chris Russell. 2020b. “Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI.” CoRR abs/2005.05906. https://arxiv.org/abs/2005.05906.

⁵ Barocas et al. (2019)

Barocas, Solon, Moritz Hardt, and Arvind Narayanan. 2019. Fairness and Machine Learning. Fairmlbook.org.

⁶ Wachter et al. (2020a)

Wachter, Sandra, Brent D. Mittelstadt, and Chris Russell. 2020a. “Bias Preservation in Machine Learning: The Legality of Fairness Metrics Under EU Non-Discrimination Law.” West Virginia Law Review, Forthcoming abs/2005.05906. https://ssrn.com/abstract=3792772.

⁷ Friedler et al. (2016)

Friedler, Sorelle A., Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. “On the (Im)possibility of Fairness.” CoRR abs/1609.07236. http://arxiv.org/abs/1609.07236.

(A) Conclusion

In line with its original goal – to provide some help in starting to think about AI fairness metrics – this article does not end with recommendations. It does, however, end with an observation. As the last section has shown, amidst all theorems and theories, all proofs and memes, it makes sense to not lose sight of the concrete: the data trained on, and the ML process as a whole. Fairness is not something to be evaluated post hoc; the feasibility of fairness is to be reflected on right from the beginning.

In that regard, assessing impact on fairness is not that different from that essential, but often toilsome and non-beloved, stage of modeling that precedes the modeling itself: exploratory data analysis.

Thanks for reading!

Photo by Anders Jildén on Unsplash

AI ethics is not an optimization problem

Sigrid Keydana — Tue, 22 Sep 2020 00:00:00 GMT

Why this post, why now

When you work in a field as intellectually-satisfying, challenging and inspiring as software design for machine learning, it is easy to focus on the technical, keeping out of sight the broader context. Some would even say it is required. How else can you keep up the necessary level of concentration?

But even for someone who hasn’t been in the field that long, it is evident that with every year that passes, with deep-learning-based technologies¹ progressing faster and faster over ever-shorter time spans, misuse of these technologies has increased as well, not just in selected countries but all over the world.

¹ For brevity, I’ll be subsuming deep learning and other contemporary machine learning methods under AI, following common(-ish) usage.

² I can’t hope to express this better than Maciej Cegłowski did here, so I won’t elaborate on that topic any further.

To eliminate one possible misunderstanding right from the start: When I’m talking about “faster and faster” progress, I’m not over-hyping things. I’m far from thinking that AI is close to “solving” problems like language understanding, concept learning and their likes — the kind of problems some would argue hybrid models were needed for. The thing is that it doesn’t matter. It is exactly the kinds of things AI does do so well that lend themselves to misuse. It is people we should be afraid of, not machines².

Back to the why. Over time, it increasingly appeared to me that writing regularly about AI in a technical way, but not ever writing about its misuse, was ethically questionable in itself. However, I also became increasingly conscious of the fact that with a topic like this, once you enter the political realm — and that we have to is the main point of this text –, likelihood rises that people will disagree, or worse, feel offended for reasons not anticipated by the writer. Some will find this too radical, some not radical (explicit) enough.

But when the alternative is to stay silent, it seems better to try and do one’s best.

Let’s start with two terms whose recent rise in popularity matches that of AI as a whole: bias and fairness.

But we are doing a lot to improve fairness and remove bias, aren’t we?

Search for “algorithmic fairness”, “deep learning fairness” or something similar, and you’ll find lots of papers, tools, guidelines … more than you have time to read. And bias: Haven’t we all heard about image recognition models failing on black people, women, and black women in particular; about machine translation incorporating gender stereotypes; about search engine results reflecting racism and discrimination? Given enough media attention, the respective algorithms get “fixed”; what’s more, we may also safely assume that overall, researchers developing new models will probe for these exact kinds of failures. So as a community, we do care about bias, don’t we?

Re: fairness. It’s true that there are many attempts to increase algorithmic fairness. But as Ben Green, who I’ll cite a lot in this text, points out, most of this work does so via rigid formalization, as if fairness — its conflicting definitions notwithstanding — were an objective quantity, a metric that could be optimized just like the metrics we usually optimize in machine learning.

Assume we were willing to stay in the formalist frame. Even then there is no satisfying solution to this optimization problem. Take the perspective of group fairness, where a fair algorithm is one that results in equal outcomes between groups. Chouldechova then shows that when an algorithm achieves predictive parity (a.k.a. precision), but prevalence differs between groups, it is not possible to also have both equal false positive and equal false negative rates.

That said, here I mainly want to focus on why pure formalism is not enough.

What with bias? That datasets can be (or rather: are) biased is hardly something anyone working in AI would object to. What amount of bias is being admitted to in other components/stages of the machine learning process varies between people.

Harini and Guttag map sources of bias to the model development and deployment process. Forms of bias they distinguish include representation (think: dataset), measurement (think: metrics³), aggregation (think: model⁴), and evaluation (similar to the two preceding ones, but at test time) bias. The paper is written in a technical style; in fact, there is even a diagram where the authors attempt to formalize the process in a mathematical way. (Personally, I find it hard to see what value is added by this diagram; its existence can probably best be explained by [perceived] requirements of the genre, namely, research paper.)

³ More on that below. Metrics used in machine learning mostly are proxies for things we really care about; there are lots of ways this can go wrong.

⁴ Meaning, a one-model-fits-all approach puts some groups at a disadvantage.

Now, so far I’ve left out the remaining two sources of bias they name. Mapped to the end of the process is deployment bias, when a system “is used or interpreted in inappropriate ways”. That raises a question. Is this about the end of a process, or is what happens now a completely different process (or: processes)? For an in-house data scientist, it may well be the end of a process; for a scientist, or for a consultant, it is not. Once a scientist has developed and published a model, they have no control over who uses it and how. From the collection of humankind’s empirical truths: Once a technology exists, and it is useful, it will be used.

Things get further out of control at the other side of the timeline. Here we have historical bias:

Historical bias arises when there is a misalignment between the world as it is and the values or objectives to be encoded and propagated in a model. It is a normative concern with the state of the world, and exists even given perfect sampling and feature selection.

I’ll get back to why this is called “historical bias” later; the definition is a lot broader though. Now definitely we’re beyond the realm of formalization: We’re entering the realm of normativity, of ways of viewing the world, of ethics.

Please don’t get me wrong. I’m not criticizing the paper; in fact, it provides a useful categorization that may be used as a “check list” by practitioners. But given the formalist style, a reader is likely to focus on the readily-formalizable parts. It is then easy to dismiss the two others as not really being relevant to one’s work, and certainly not something one could exert influence on. (I’ll get back to that later.)

One little aside before we leave formalism: I do believe that a lot of the formalist work on AI ethics is done in good intent; however, work in this area also benefits organizations that undertake it. Citing Tom Slee,

Standards set public benchmarks and provide protection from future accusations. Auditable criteria incorporated into product development and release processes can confirm compliance. There are also financial incentives to adopt a technical approach: standards that demand expertise and investment create barriers to entry by smaller firms, just as risk management regulations create barriers to entry in the financial and healthcare industries.

Not everything in life is an optimization problem

Stephen Boyd, who teaches convex optimization at Stanford, is said to often start theintroductory lecture with the phrase “everything is an optimization problem”. It sounds intriguing at first; certainly a lot of things in my life can be thought of like that. It may become awkward once you start to compare, say, time spent with loved ones and time spent working; it becomes completely unfeasible when comparing across people.

We saw that even under formalist thinking, there is no impartial way to optimize for fairness. But it’s not just about choosing between different types of fairness. How do you weigh fairness against stakeholder interests? No algorithm will be deployed that does not serve an organization’s purpose.

There is thus an intimate link between metrics, objectives and power.

Metrics and power

Even though terminologically, in deep learning, we distinguish between optimization and metrics, the metrics really are what we are optimizing for: Goodhart’s law — When a measure becomes a target, it ceases to be a good measure⁵— does not seem to apply. Still, they deserve the same questioning and inspection as metrics in other contexts.

⁵ cited after Thomas and Uminsky

In AI as elsewhere, optimization objectives are proxies; they “stand in” for the thing we’re really interested in. That proxying process could fail in many ways, but failure is not the only applicable category to think in here.

For one, objectives are chosen according to the dominant paradigm. Dotan and Milli show how a technology’s perceived success feeds back into the criteria used to evaluate other technologies (as well as future instances of itself). Imagine a world where models were ranked not just for classification accuracy, but also for, say, climate friendliness, robustness to adversarial attacks, or reliable quantification of uncertainty.

Objectives thus do not emerge out of nothing. That they reflect paradigms may still sound abstract; that they serve existing power structures less so. Power structures are complex; they reflect more than just who “has the say”. As pointed out in various “classics” of critical race theory, intersectionalist feminism and related areas⁶, we should ask ourselves: Who profits?

⁶ see e.g., DataFeminism and RaceAfterTechnology.

This is an all but simple question, comparable in difficulty, it seems to me, to the request that we question the unspoken premises that underlie our theories. The main point about such premises is that we aren’t aware of them: If we were, we could have stated them explicitly. Similarly, if I noticed I was profiting from someone, I would — hopefully — take some action. Hard as the task may be, though, we have to do our best.

If it’s hard to see how one is privileged, can’t we just be neutral? Objective? Isn’t this getting too political?

There is no objectivity, and all is politics

To some people, the assertion that objectivity cannot exist is too self-evident to require much dwelling on. Are numbers objective? Maybe in some formal way; but once I use them in communication, they convey a message, independently of my intention.

Let’s say I want to convey the fact, taken from the IPCC website, that between 2030 and 2052, global warming is likely to reach 1.5°C. Let’s also assume that I look up the pre-industrial average for the place where I happen to live (15°C, say), and that I want to show off my impressive meteorological knowledge. Thus I say, “… so here, guys, temperature will rise from 288.15 Kelvin to 289.65 Kelvin, on average”. This surely is an objective statement. But what if the people I’m talking to don’t know that — even though the absolute values, when expressed in Kelvin, are so much higher than when expressed in degrees Celsius — the relative differences are the same? They might get the impression, totally unintended, that not much warming is going to happen.

If even numbers, once used in communication, lose their objectivity, this must hold even more for anything that involves more design choices: visualizations, APIs, and, of course, written text. For visualizations, this is nicely illustrated in d’Ignazio and Klein’s Data Feminism.

That book is also an exemplary exercise in what appears like the only way to “deal with” the fact that objectivity is impossible: Trying to be as clear as possible about where we stand, who we are, what are the circumstances that have influenced our way of thinking. Of course, like with the unspoken premises and assumptions discussed above, this is not easy; in fact, it’s impossible to do in perfection. But one can try.

In fact, the above “Trying to be as clear as possible …” is deliberately ambiguous. It refers to two things: For one, to me striving to analyze how I’m privileged, and secondly, to me giving information to others. The first alone is laudable but necessarily limited; the second, as exercised by d’Ignazio and Klein, opens the door not just for better mutual understanding, but also, for feedback and learning. The person I’m talking to might lead me to insights I wouldn’t have gotten otherwise.

Putting things slightly differently, there is no objectivity because there’s always a context. Focus on metrics and formalization detract from that context. Green relates an interesting parallel from American law history. Until the early twentieth century, US law was dominated by a formalist ethos. Ideally, all rules should be traceable to a small number of universal principles, derived from natural rights. Autonomy being such a right, in a famous 1905 case, the U.S. Supreme Court concluded that a law limiting the working hours of employees represented “unreasonable, unnecessary and arbitrary interference with the right and liberty of the individual to contract”. However,

In his dissent, Justice Oliver Wendell Holmes argued that the Court failed to consider the context of the case, noting, “General propositions do not decide concrete cases”.

Thus, the context matters. Autonomy is good; but it can’t be used as an excuse for exploitation.

The same context dependence holds in AI. It is always developed and deployed in a context, — a context shaped by history. History determines what datasets we work with; what we optimize for; who tells us what to optimize for. An daunting example is so-called “predictive policing”. Datasets used to train prediction algorithms incorporate a history of racial injustice: The very definition of “crime” they rely on wasshaped by racist and classist practice. The new method then perpetuates — more than that: exacerbates — the current system, creating a vicious cycle of positive feedback that makes it look like the algorithm was successful.

Summing up: When there is no neutrality, everything is politics. Not acting is acting. Citing Green,

But efforts for reform are no more political than efforts to resist reform or even the choice simply to not act, both of which preserve existing systems.

But I’m just an engineer

When machine learning people, or computer science people in general, are asked about their views on the societal impact of modern AI, an often-heard answer is: “But I’m just an engineer…”. This is completely understandable. Most of us are just tiny cogs in those big machines, wired together in a complex network, that run the world. We’re not in control; how could we be accountable?

For the AI scientist, though normally all but sitting in an “ivory tower”, it’s nevertheless the ML engineers who are responsible of how a model gets deployed, and to what consequences⁷. The ML engineer may delegate to the head of IT, who in turn had no choice but implement what was requested “by business”. And so on and so forth, ad infinitum.

That said, it is hard to come up with a moral imperative here. In the line of thinking exercised above: There is always a context. In some parts of the world, you have more choices than in others. Options vary based on race, gender, abledness, and more. Maybe you can just quit and get another job; maybe you can’t.

There is another — related — point though, on which I’d like to dwell a little longer.

Technology optimism

Sometimes, it’s not that the people working in AI are fully aware of, but don’t see how to counter, the harm that is being done in their field. On the contrary. They are convinced that they’re doing good. Just do a quick search for “AI for good”, and you’ll be presented with a great number of projects and initiatives. But how, actually, is decided what is good? Who decides? Who profits?

Ben Green, again, relates an instructive example:

USC’s Center for Artificial Intelligence in Society (CAIS) is emblematic of how computer science projects labeled as promoting “social good” can cause harm by wading into hotly contested political territory with a regressive perspective. One of the group’s projects involved deploying game theory and machine learning to predict and prevent behavior from “adversarial groups.” Although CAIS motivated the project by discussing “extremist organizations such as ISIS and Jabhat al-Nusra,” it quickly slipped into focusing on “criminal street gangs” [43]. In fact, the project’s only publication was a controversial paper that used neural networks to classify crimes in Los Angeles as gang-related [28, 41].

Predictive policing, already mentioned above, can also be seen in this category. At first thought, isn’t it a good thing? Wouldn’t it be nice if we could make our world a bit more secure?

Phillip Rogaway, who I’ll mention again in the concluding section, talks a lot about how technology optimism dominates among his students; he seems to be as intrigued by it as I am. Personally, I think that whether someone “intrinsically” tends to be a technology optimist or a pessimist is a persistent trait; it seems to be a matter of personality and socialization (or just call it fate: I don’t want to go into any nature-nurture debates here). That glass, is it half full or half empty?

All I could say to a hardcore technology optimist is that their utopia may be another person’s dystopia. Especially if that other person is poor, or black, or poor and a black woman … and so on.

Let me just end this section with a citation from a blog post by Ali Alkhatib centered around the launch of the Stanford institute for human-centered artificial intelligence (HAI). Referring to the director’s accompanying blog post, he writes

James opens with a story of an office that senses you slouching, registers that you’re fatigued, intuits that your mood has shifted, and alters the ambiance accordingly to keep you alert throughout the day.

You can head to Alkhatib’s post (very worth reading) and read the continuation, expanding on the scenario. But for some people, this single sentence may already be enough.

And now?

At some point, an author is expected to wrap up and present ideas for improvement. With most of the topics touched upon here, this is yet another intimidating task. I’ll give it a try anyway. The best synthesis I can come up with at the moment looks about like this.

First, some approaches filed by Green under “formalist” can still make for a good start, or rather, can constitute a set of default measures, to be taken routinely. Most prominently, these include dataset and model documentation.

Beyond the technical, I like the advice given in Baumer and Silberman’s When the Implication is Not to Design. If the consequences, especially on socially marginalized groups, of a technological approach are unforeseeable, think whether the problem can be solved in a “low-tech” way. By all means, do not start with the solution and then, go find a problem.

In some cases, even that may not be enough. With some goals, don’t look for alternative ways to achieve them. Sometimes the goal itself has to be questioned.

The same can be said for the other direction. With some technologies, there is no goal that could justify their application. This is because such a technology is certain to get misused. Facial recognition is one example.

Lastly, let me finish on a speculative note. Rogaway, already referred to above for his comments on technology optimism, calls on his colleagues, fellow cryptographers, to devise protocols in such a way that private communication stays private, that breaches are, in plain terms, impossible. While I personally can’t think see how to port the analogy to AI, maybe others will be able to, drawing inspiration from his text. Until then, changes in politics and legislation seem to be the only recourse.

The hard problem of privacy

Sigrid Keydana — Thu, 27 Feb 2020 00:00:00 GMT

Why “the hard problem” of privacy? Inspired by the study of consciousness, where some authors posit a difference between a “hard” and an “easy” problem, it is tempting to talk about privacy – as perceived by machine learning (ML) people – in a similar way. The “easy” part would be technical realization; including differential privacy (see e.g. Differential privacy with TensorFlow) and federated learning (to be featured on the TensorFlow for R blog soon). The hard part would be to have people think that privacy actually matters.

Of course, it depends on what is understood by privacy, or conversely, by surveillance. Last summer, Rachel Thomas, widely known as co-founder of fast.ai and founding director of the USF Center for Applied Data Ethics, wrote about 8 Things You Need to Know about Surveillance. Everything she writes is highly important and deserves being thought about every single day. However, in a well-meaning community there should be little controversy about how justified these points are. My starting point here is different, and it, like the whole of this post, is highly subjective.

Soul mates

Recently, I listened to an episode of Lex Fridman’s AI Podcast. Depending on one’s interest, this podcast has the most interesting guest list one could imagine: Donald Knuth, Daniel Kahneman, Gilbert Strang, Leonard Susskind, Yoshua Bengio … to name just a few. But the podcast is nice to listen to not just because of the guest list. The interviewer himself impresses by his modest, warm and deeply human ways.

On that backgound, I was immensely surprised when in the episode with Cristos Goodrow, VP of Engineering at Google and head of Search and Discovery at YouTube, Fridman not just highly extolled Youtube’s recommendation algorithm, but also expressed a “dream” that Youtube, besides similar-as-per-the-recommendation-algorithm videos, also suggested similar-as-per-the-algorithm users (min. 30/31):

I know I’m gonna ask for things that are impossible but I would love to cluster human beings, I would love to know who has similar trajectories as me …” .

The algorithm as a helper to find friends, or people like oneself, one would never get to meet otherwise. The most human dream ever, finding soul mates, powered by Youtube.

Incapable to get over my surprise, I started thinking whether I should write this post.

Why aren’t we more scared?

It’s not the most fruitful question, but helpless inability to understand probably starts like this, asking: Why aren’t we more scared?

For starters, we’re used to getting stuff recommended all the time. On Amazon, I may have bought some books that were purchased by similar users. (On the other hand, the overall quality of recommendations is not like I would start to feel threatened by the magic.) So, advantages jump to the eye, while disadvantages are not immediately visible.

Secondly, traditionally, in many cultures, we are accustomed to associate surveillance with the government, not private companies, just simply because historically, it was in fact the government, and the government only, that had the power to surveil and oppress. This is the classic picture of totalitarianism.

However, as described by Shoshana Zuboff in her lengthy-ish, not-without-redundancies, but immensely important book The age of surveillance capitalism, the modern reality – at least in the US and Europe – is less direct and less easy to see through. (Evidently, the story is different in a country like China, where a social credit system has already been established.)

Surveillance capitalism in a nutshell

Zuboff’s book recounts, in great detail and citing a lot of sources public and private, how we ended up living in a world where each of our online actions is tracked by numerous “interested parties”, where our phones send who-knows-what to whichever distant beneficiary, and where more and more of our “real-world”, physical activity is recorded by agents we may not even know of.

In short, the mechanism she describes is this: Companies big and small, the so-called “surveillance capitalists”, have their business based on gathering, extracting, and processing information – more than is needed to improve a product or a service. Instead, this “behavioral surplus” – in processed form – is sold to customers, which may be everything from other businesses over banks and insurance companies to intelligence agencies and governments.

With such a business model, of course a company is interested in optimizing its ML algorithms. The better the algorithms, the better the results that are sold to customers, and the better those results, the more that company is willing to pay. Online ads may look like a triviality, but they are in fact a great illustration of the principle: If our customer pays per click, and our business is “full-service” – “don’t worry about anything, we’ll place your ad for you”, then it’s in my own best interest to show ads to the most susceptible users, in the most favorable circumstances. Of course we’ll get creative, thinking about how we can best determine those users and moments.

As already hinted at above, with private companies in possession of data and inferences not publicly available, governments, intelligence services, and the military re-enter the picture. Zuboff describes how intelligence agencies craved information that was hard to obtain legally, but could be provided by companies operating in a lawless zone, always ahead of legislation. 9/11 was just a door opener; from then on, massive interlacing ensued between politics and companies in the information business, as seen for example, but not only, in the 2012 and 2016 US elections.

Zuboff’s book is full of interesting details, but this post is not meant to be a “best of”. Rather, I’m wondering how the extent, as well as the importance of, what’s going on could be made more salient to ML people – not just to those to whom doubt and skepticism come naturally, but to well-meaning idealists as well.

Why we should care

Trying to make this more salient, I can think of three things:

Concisely stating a few facts that, important though they are in Zuboff’s argumentation, deserve being pointed out even outside the concrete contexts in which they appear in the book;
Calling attention, in a few lines instead of a time-consuming narrative, to what has already happened, in the “real world”, in the very short time span since those new dynamics took off; and (the most impressive to me, personally)
Relating, by citation, the actual “utopias” behind the reality.

(1) A few things to consider

A common way to think about this is: We’re using a free service (like a search engine), we have to pay somehow, so we’re paying with our data. But – data is collected and processed by products we buy, as well, be it phones, wearables or household appliances.

Also, it is not like we get to decide whether and if so, how much privacy we’re willing to give up in exchange for a service. As often noted, no-one has time to read through all the “privacy policies” we’re bombarded with; and sometimes we’re just told “you can disable this; but then feature x [[[some vital feature, probably the one we bought it for]]] can’t be guaranteed to be working anymore”.

(2) Not just online

Even to people who spend most of their day online, being tracked on the web may still seem like a minor inconvenience. Of course, through our smartphones, we carry the web with us. Evidently, depending on what apps one has installed, one may receive friendly “invitations” to, say, pay a visit to McDonald’s just as one passes there on one’s lunchtime run. (Or maybe not, depending on whether the type of ongoing activity is already being taken into account.)

Sure, such notifications disrupt the flow of whatever one’s doing. But still: For some of us, it’s easy to think that who cares? I don’t have those kinds of apps installed anyway. Put more generally: Yes, all this online tracking is annoying. But as long as all that happens is me seeing more “targeted” ads, why bother?

The recent years have seen a consistent and speedy extension of tracking in the real world.

There was Google Street View, filming the outdoors, a space once thought of as ephemeral, and subsequent attempts, by various actors, to map public interiors.

There was Google glass, allowing to video- and audiotape people without their knowledge.

Then, there are all kinds of wearables, also able to record not just the wearer’s, but bystanders’ data. The wearers themselves must be aware that some data is collected (why else buy the thing?), but don’t necessarily know how it’s being made use of, and what other data may be collected.

Then, there are all kinds of IoT devices: TVs, vacuum cleaners, thermostats… Again, they are bought for a reason; but no-one can expect a TV to record what you’re saying, or a vacuum cleaner to transmit back your floor plan.

Then – and this makes for a perfect transition to the last section, “utopias” – there is smart city: not a utopia, but actual reality, already being implemented in Toronto. Here is an excerpt from an interview with CEO Dan Doctoroff:

In effect, what we’re doing is replicating the digital experience in physical space … So ubiquitous connectivity; incredible computing power including artificial intelligence and machine learning; the ability to display data; sensing, including cameras and location data as well as other specialized sensors… We fund it all… through a very novel advertising model… We can actually then target ads to people in proximity, and then obviously over time track them through things like beacons and location services as well as their browsing activity.

From these publicly made statements, it is hard to doubt what The Globe and Mail reported re. a leaked document, the yellow book, containing detailed outlines about how this system would be implemented:

Early on, the company notes that a Sidewalk neighbourhood would collect real-time position data “for all entities” – including people. The company would also collect a “historical record of where things have been” and “about where they are going.” Furthermore, unique data identifiers would be generated for “every person, business or object registered in the district,” helping devices communicate with each other.

The newspaper continues,

The document also describes reputation tools that would lead to a “new currency for community co-operation,” effectively establishing a social credit system. Sidewalk could use these tools to “hold people or businesses accountable” while rewarding good behaviour, such as by rewarding a business’s good customer service with an easier or cheaper renewal process on its licence.

If, to people in “Western cultures”, China, with its open endorsement and implementation of a social credit system, tended to seem “far, far away” – now nothing seems as it did before.

At this point, it’s time to switch gears and look at the world view(s) behind those recent developments.

(3) Utopias

Here is part of the vision of Mark Zuckerberg. Zuboff, aggregating information from several Zuckerberg-quoting, publicly available sources, writes (p.402)

Predictive models would enable the corporation to “tell you to what bar to go to” when you arrive in a strange city. The vision is detailed: when you arrive at the bar, the bartender has your favorite drink waiting, and you’re able to look around the room and identify people just like you.

Extending this to life overall, here we have a vision of total boredom; no exploration, no curiosity, no surprise, no learning. How probable is it that evolutionarily, humans are prepared to live in a world without uncertainty?

How about regulation? Here is Larry Page on laws and experimentation:

If you look at the different kinds of laws we make, they’re very old. The laws when we went public we’re 50 years old. A law can’t be right if it’s 50 years old, like it’s before the Internet, that’s a pretty major change, in how you may go public.[…]

The other thing in my mind is we also haven’t built mechanisms to allow experimentation. There’s many, many exciting and important things you could do that you just can’t do because they’re illegal, or they’re not allowed by regulation, and that makes sense, we don’t want the world to change too fast. Maybe we should set aside a small part of the world …[…]

I think as technologists we should have some safe places where we can try out some new things and figure out what is the effect on society, what’s the effect on people, without having to deploy kind of into the normal world. […]

Or Zuckerberg again, on the ancient social norm of privacy:

“People have really gotten comfortable not only sharing more information and different kinds, but more openly and with more people,” he said. “That social norm is just something that has evolved over time.”

In this light, is a social credit system – perhaps packaged differently – really so far away, in Western cultures? Sometimes, a scenario sounds so silly that it’s hard to see it as a real threat; but that may be just us being naive. Here is another vision, from Rana el Kaliouby, CEO at Affectiva:

We have been in conversations with a company in that space. It is an advertising-rewards company, and its business is based on positive moments. So if you set a goal to run three miles and you run three miles, that’s a moment. Or if you set the alarm for six o’clock and you actually do get up, that’s a moment. And they monetize these moments. They sell them. Like Kleenex can send you a coupon – I don’t know – when you get over a sad moment. Right now, this company is making assumptions about what those moments are. And we’re like, ‘Guess what? We can capture them.’

If that sounds like it needn’t be taken seriously, here is a quote from Social Physics, a highly acclaimed opus by Alex Pentland, director of MIT Media Lab:

The social physics approach to getting everyone to cooperate is to use social network incentives rather than to use individual market incentives or to provide additional information. That is, we focus on changing the connections between people rather than focussing on getting people individually to change their behavior. The logic here is clear: Since exchanges between people are of enormous value to the participants, we can leverage those exchanges to generate social pressure for change.[…]

Social network incentives act by generating social pressure around the problem of finding cooperative behaviors, and so people experiment with new behaviors to find ones that are better.

At this point, I feel like nothing has to be added to those visions. But let’s get back to the beginning, Lex Fridman’s fantasy of clustering users.

Expressing oneself

Zuboff quotes Zuckerberg saying, “humans have such a deep need to express themselves” (p. 403). Evidently, this is what motivates Fridman’s wish, and yes, it’s a, or the, profoundly human thing.

But, doesn’t expressing oneself, if seen as sharing, involve a free decision when, what, and with whom to share?

And what does it do to an experience if it is not just “there”, a thing valuable in itself, but a means for others, to control and profit?

Thanks for reading!