The Explicit Data for Implicit Biases

In 1995, two psychologists found that if you exposed subjects to images of people of different races, associating them with words that were positive or negative, most people would be biased towards associating black people with negative words, and white people with positive words. This was done via a clever method called the Implicit Association Test (IAT), which you can take here. This was huge, as it seemed to be a window into people’s unconscious minds, and subjects could see exactly which types of people they held unconscious biases towards. Pretty cool! The thinking was that these unconscious biases would manifest as explicit biases in one way or another, and if you were made aware of your implicit bias, you could make an effort to reduce any explicit biases that might bubble up.

The research became very popular, with hundreds of other studies being done, looking for similar effects with various groups of people. Once the general public caught wind of this, an entire cottage industry popped up, with so-called implicit bias experts promising companies that they could educate and dismantle their employees’ implicit biases… for a small fee, of course.

There are four main premises that the IAT is based on:

• IAT is a reliable psychological tool which shows that…
• People often have unconscious biases towards certain groups of people
• These unconscious biases can be used to predict explicitly biased behavior
• Being made aware of these unconscious biases can help mitigate explicitly biased behavior 

Unfortunately, it turns out that every one of the above premises is incorrect--which I will show by citing numerous studies that contradict each claim. The research I will be referring are not just small, one-off studies that I found by combing through the data in an attempt to be a grumpy contrarian. Instead, these are often very large meta-studies that look at the trends of multiple research papers. And while there is always debate over complex scientific topics, these results are not controversial at all among researchers who study implicit biases. 

Premise #1: IAT is a reliable psychological tool

I once had a psychology professor tell me that “psychology is a soft science, but it’s also the hardest science.” She meant that because psychology studies complex human beings, that also makes it the most difficult science. If you kick 10 different soccer balls, you will get similar results, but if you kick 10 different people, your results may vary!

Because humans make things so complicated, researchers have to be careful with their tests, ensuring that their studies aren’t so vague as to elicit wildly different responses from a participant who takes the test multiple times. This idea is called Repeatability, or Test-Retest. If you take some sort of psychological test 10 times and get the same response 9 out of 10 times, researchers can be confident they have tapped into some sort of psychological reality in your mind. But if you get wildly different results each time you take the test, something is wrong. Or at the least, the test is not reliable.

You can read about repeatability and the scores that are used here and here. Though, the basic breakdown is that on a scale of 0 and 1, any score below a 0.5 is unacceptable. The IAT has a score somewhere between 0.44 and 0.5. On its best day, the IAT is unacceptably bad for producing any sort of clear picture of what is supposedly going on in a person’s mind.

This low score (high variation in test results) is mostly attributed to people getting better at the task as they take the test multiple times. Either way, the fact that the IAT’s repeatability coefficient is so low makes it incredibly unlikely that the IAT is telling us anything meaningful or useful about an individual’s mental processes. 

Premise #2: People often have unconscious biases towards certain groups of people 

People absolutely have biases towards different groups of people—this is not in question. The IAT, however, claims to be able to tap in to hidden, unconscious biases that we are not aware of. Though, when subjects were asked to predict the results of their IAT tests, their predictions were quite accurate! How could they be accurately predicting what their unconscious biases are, if the biases are unconscious? This calls into question the “implicit” part of implicit bias.

A 2006 study concluded that while people may not be aware of the origin of their biases, “there is no evidence that people lack conscious awareness of indirectly assessed attitudes.” 

Likewise, a 2014 study reported that “the research findings cast doubt on the belief that attitudes or evaluations measured by the IAT necessarily reflect unconscious attitudes.” 

Another 2014 study found that “there is compelling evidence that people are consciously aware of their implicit evaluations.”

The fact that IAT cannot discover unconscious biases is a problem for the IAT, but it does not mean people do not have biases they may not be aware of. People absolutely do--it is just that the IAT is not a reliable method to discover them. 

Premise #3: These unconscious biases can predict explicitly biased behavior 

This is a fairly intuitive, and very reasonable premise. Thoughts and actions are closely related, so it would make sense that if you had a bias towards a certain group of people, your behavior may reflect that, even subtly. However, this premise is claiming that there is a relationship between the unconscious biases discovered by the IAT, and biased behavior. The evidence for this is not good.

You can find studies here and there that will show predictive power between implicit and explicit biases. However, since the “replication crisis” in psychology started, small studies with small effects are no longer good enough. We need to use large scale, or meta-studies, to see what the larger trends are. In this case, several meta-studies have shown that there is essentially zero correlation between implicit biases and real life behavior or attitudes. Meaning, if the IAT shows you are biased towards a certain group of people, this has no correlation or ability to predict how you actually treat people of that group. 

A 2008 study found that (among other things) “The implicit association test (IAT) is the most widely used measure of implicit attitudes, and strong claims have been made about its ability to reveal high rates of unconscious racism. Empirical evidence does not support these claims.” 

A 2013 study looked at data from an earlier meta-study, concluding “across diverse methods of coding and analyzing the data, IAT scores are not good predictors of ethnic or racial discrimination.”

Another 2013 meta-study found that “The IAT provides little insight into who will discriminate against whom, and provides no more insight than explicit measures of bias.”

A 2016 meta-study found that “there is also little evidence that the IAT can meaningfully predict discrimination, and we thus strongly caution against any practical applications of the IAT that rest on this assumption.” They continue, “the overall effect of discrimination in the literature is virtually zero. There are only a handful studies that in isolation demonstrate clear levels of discrimination, and even fewer do so without having methodological problems that may plausibly have produced the result. Accordingly, there appears to be a very small amount of variance that can reliably be predicted from the IAT.”

The above studies are damming enough as it is. Though, the authors of the original Implicit Bias study stated in a 2014 study that “IAT measures have two properties that render it problematic to use them to classify persons as likely to engage in discrimination. Those two properties are modest test–retest reliability, and small-to-moderate predictive validity effect sizes."

The third premise, in my opinion, is the most important premise of all. If there is no relationship between supposedly implicit and explicit biases, the test is all but useless with regards to its stated purpose.

Premise #4: Being made aware of these unconscious biases can help mitigate explicitly biased behavior 

Since the third premise has failed, the fourth one also fails, as the idea that we can change explicit biases by learning about our implicit biases assumes there is a causal link—which we have seen there is not. However, there is research that looks specifically at the fourth premise, so I think it is important to cover it as well. 

A 2015 meta-study looking at 492 studies with over 87,000 participants found that “changes in implicit measures did not mediate changes in explicit measures or behavior. Our findings suggest that changes in implicit measures are possible, but those changes do not necessarily translate into changes in explicit measures or behavior.”

And with that, the claims of the IAT have completely failed, as none of them are supported by the data. 

Conclusion 

So now what? Probably nothing. This data is not new, is not a secret, and definitely is not sexy. No one is against evolution because they are interested in the debate between the level of selection, or at what point amphibians started to transition into lizards. People who are in denial about evolution are worried about the moral and religious implications.

Similarly, I doubt that many non-psychologists who are interested in the IAT are actually interested in the research. They are interested in eliminating racism, sexism, etc, which is a good thing to be working toward! However, if they have attached too much moral or ideological weight to the IAT, they might deny the evidence above, just like creationists with evolution. Similarly, people who make their living by running anti-bias training programs will never admit that the concepts they base much of their work on are not backed up by the data. To quote Upton Sinclair, “It is difficult to get a man to understand something when his salary depends on his not understanding it."

The IAT is often treated as a magic bullet to uncover unconscious biases and help eliminate them. Science is a wonderful tool—but it can also be a cruel mistress. If you are going to use the findings of science, you have to be willing to change your mind if the evidence later points in another direction.* Despite the IAT’s inability to expose unconscious biases and reduce explicit ones, that doesn’t mean people aren’t biased or bigoted—it just means we have to find other methods and tools to help combat those biases. In order to reach that goal, we need to be honest with ourselves, admit when we are wrong, and utilize methods that actually work.

*The same goes for me and these arguments. I am not an expert, and research could come out tomorrow that completely contradicts me. Double check everything, do your own research and come to your own conclusions!