“Toolbox” articles delve into a new way of looking at values, with a view to using these techniques in future articles.
Last week, I introduced the two systems of thinking: System 1, which is based on emotions, intuitions, and heuristics (i.e. rules of thumb), and System 2, which is rational and methodological. Using the Monty Hall problem, a famous brain teaser, I demonstrated how System 1 thinking can lead to mistakes, but also how System 2 can override these errors with time and effort (spoiler alert: it’s extremely uncomfortable, and you would have to be an masochist to want to do that every day).
In this article, I’ll bring the conversation back to values. There is considerable evidence to suggest that value-based judgements are derived from System 1, not System 2. An important piece of proof is “moral dumbfounding”, instances when people have a strong negative moral reaction to a situation (e.g. a moral vignette), but they struggle to explain why they feel that way. If pressed for a justification for their reactions, individuals are left speechless and confused. Sometimes, they conclude that certain acts are just wrong, and no explanation is necessary.
Moral dumbfounding is rare in instances where harm is easily recognized. For example, it is easy to justify why murder is immoral. There are a large number of harm-based arguments that immediately spring to mind (e.g. if everyone murdered with impunity, society would collapse). However, in cases where the direct harms of an action are difficult to discern, moral dumbfounding is more common. For example, situations that violate the Sanctity value (i.e. certain actions are inherently dirty and polluting) often have little to do with harm, so they are fertile ground for exploring moral dumbfounding. Here’s one:
Jennifer works in a medical school pathology lab as a research assistant. The lab prepares human cadavers that are used to teach medical students about anatomy. The cadavers come from people who had donated their body to science for research. One night Jennifer is leaving the lab when she sees a body that is going to be discarded the next day. Jennifer was a vegetarian, for moral reasons. She thought it was wrong to kill animals for food. But then, when she saw a body about to be cremated, she thought it was irrational to waste perfectly edible meat. So, she cut off a piece of flesh, and took it home and cooked it. The person had died recently of a heart attack, and she cooked the meat thoroughly, so there was no risk of disease.
This vignette was created to remove actual harm from the interaction. Even so, people who encounter this vignette intuitively know that Jennifer acted wrongfully, even though they struggle to clearly express why, beyond simply the disgust that the situation elicits. This is where the conflict between System 1 and System 2 begins: System 1 provides an automatic moral judgement. Then, System 2 is activated to provide the justification. If it cannot find a satisfactory explanation, System 2 begins to flail, searching left and right for the appropriate argument to justify and rationalize the initial gut reaction.
Cognitive dissonance often ensues, and there are only two exits: System 2 overrides System 1, or System 2 finds a satisfactory justification through rationalization (i.e. working backwards to find reasons to justify a pre-selected conclusion).
Exit #1 or Exit #2?
In questions of morality, the first exit (painful reflection) is sometimes taken. This is particularly common among left-leaning and highly educated people, who tend to rely heavily on the Care value and who do not believe simple disgust to be a suitable foundation for moral judgement. If they cannot logically justify their moral reactions through the Care value, liberals are more likely than conservatives to override their System 1 responses.
Clearly, values are not completely disconnected from System 2. We have the capacity to observe our moral reaction to a situation, reflect on it, and genuinely change our moral worldview. Shifts in social values, from the breakdown of deference to royalty to the normalization of divorce, benefited from persuasive arguments that activated people’s System 2 thinking. I think we can all look back on a time when we had an immediate moral reaction that we later reconsidered based on reason.
However, this doesn’t happen often. Indeed, few people escape cognitive dissonance through the first exit of painful reflection. This is because reconsidering our System 1 reactions is uncomfortable, as demonstrated by the Monty Hall problem of last week. The vast majority of people who are confronted with this brain teaser struggle to override their intuitions. In fact, even extremely intelligent people throw childish fits over the Monty Hall problem, fighting to preserve and justify their System 1 reactions. And this is a silly little math problem about cars and goats! If reassessing our faulty views of probability is so arduous, imagine the discomfort brought by a clear-eyed re-evaluation of tightly held moral values?
Consequently, the second exit of satisfying rationalization is a far more typical way out of values-based cognitive dissonance. Simply put, people manage to find a satisfactory System 2 explanation for their System 1-based moral judgements. For example, System 2 can find (or invent) some key facts that “prove” the action to be harmful, either directly or after a convoluted series of knock-on effects. With enough time to think through the issue, System 2 will almost always be able to find a sufficiently convincing justification to put our minds at peace. It’s designed to do so.
Going back to the example with Jennifer the Cannibal, when asked to explain why her actions were immoral, many people will revert to an appeal to potential harms. We can imagine how this reasoning might play out: although Jennifer’s actions caused no direct harm, she undermined the taboo against cannibalism, which could cause more people (including Jennifer) to engage in similar behaviour more often. They might even start to murder other people when they’re hungry!
This defense obviously results from rationalization. Imagine an AI who was only fitted with System 2 thought processes. It would probably use the following method to judge Jennifer’s actions:
- Suspend judgement of Jennifer.
- Determine which benefits and harms would reasonably result from Jennifer’s actions. The efficient use of food resources would be considered a benefit. The long-term effects of cannibalism on Jennifer’s outlook might be considered a (weak) harm.
- Make a determination on the (im)morality of the situation by weighing the relative importance of the benefits and harms.
Would following such a process lead this AI to conclude that cannibalism is wrong in all cases? I doubt it. The AI probably wouldn’t find appeals to potential harms to be a particularly satisfactory justification for holding Jennifer morally responsible for eating a person-steak, because Jennifer’s snack is unlikely to appreciably undermine social norms or cause a rise in the rate of cannibalism. I speculate that the AI would find Jennifer’s actions to be permissible.
But we aren’t unemotional robots. Normal people can’t just turn off their System 1 thinking, so they don’t follow such hyper-rational thought processes. They instantly know Jennifer acted incorrectly and make up a reason to justify their gut feelings, showing that System 2 is extremely adaptable when it comes to values. In other words, System 2 can manufacture many potential harms of a given action in an attempt to explain why exactly it is so wrong. This process often doesn’t lead to particularly good justifications for System 1 reactions, but it does lead to good enough reasons to resolve our cognitive dissonance.
The Policy Connection
As public servants, we tend to focus almost exclusively on System 2 reasoning. Briefing notes detail economic costs and benefits, legal precedents, and/or social impacts of policy. Often, policy wonks find solutions that they believe to be superior to alternatives. And if we only factor in System 2 reasoning, if we follow the thought process of an AI, these policies are optimal. They maximize benefits and minimize harms, as System 2 demands.
However, when public servants move to implement these policies, they are sometimes confronted with fierce resistance from certain groups. Opponents often raise incorrect (and sometimes downright bizarre) arguments against reasonable policies. For example, anti-vaxxers posit that vaccination causes autism (it doesn’t), so they resist the requirement for their children to get MMR shots before going to school.
If we believe that anti-vaxxers are being driven by their System 2 thinking, then the solution is obvious: education. Just give them the correct information, they’ll go through a process of painful reflection, and come around to the truth. This is a technocratic problem the demands a technocratic solution.
Sounds great, but there is one problem: anti-vaxxers aren’t being led by their System 2 thinking. No one is. System 1 is the driving force of moral and ethical decisions. Consequently, the specifics of the arguments themselves don’t always matter. Sometimes anti-vaxxers argue that vaccination causes autism, but other times they claim that it changes your DNA, contains 5G tracking chips, or makes people magnetic. The details of each claim are just new rationalizations of a moral gut reaction, so we can expect that educational campaigns will be of questionable effectiveness. Sure, you might convince someone that vaccination doesn’t cause autism, but they may come up with a new reason to hold the same belief.
I don’t mean to suggest that education campaigns are useless. As mentioned earlier, sometimes System 2 can override System 1. But the public service needs to use more tools to communicate and design policy. We need to consider System 1 drivers (especially values) more often and from the earliest stages of policy development. This will allow the public service to tailor policies and communicate more effectively with a population that inevitably reacts to different inputs in different ways.