Bias in research

I got a bit of flak from my last blog for suggesting that some highly experimental studies involving a small number of adults with a particular cognitive profile may not be relevant to the everyday lives of all people with autism. I was accused on Twitter and other places of being ‘biased’. At first I was a bit offended by this, but then I relaxed. One of the first things you learn as a psychology student is that everyone is biased. We can’t help it. It just seems to be part and parcel of the human condition. The second thing you learn though is that biases can be unhelpful and at worst, dangerous. In science biases can be particularly damaging, particularly if they cause scientists to ignore, or worse, suppress evidence that contradicts their biases, or prevents them from collecting or reporting such evidence in the first place (witness for example Susan Greenfield’s spectacular fall from academic grace). Scientists can’t completely eradicate biases, we are after all only human. The best we can do is to try and identify what our biases are, and take steps to minimize the impact of these biases on our scientific endeavours.

Thinking about bias brought to mind one of my favourite journal articles of recent years. The first thing I love about this paper is its wonderfully provocative title – the authors had the audacity to label me and people like me as WEIRD – white, educated, industrialized, rich and democratic. They then go on to make the observation that most psychology research, and more importantly the conclusions and theories about human perception and cognition derived from this research, is based almost entirely on WEIRD people, namely American (and British) university undergraduates. I can see why. I am deeply envious of my colleagues who can run a study on 50 undergrads in a matter of weeks, while it usually takes me a couple of years of travelling all over the place to have enough data to produce a publication! Nevertheless, the authors of this paper ask a simple question ‘are the data derived from an elite, 1% of the world’s population really relevant for the remaining 99% of human experience?’ And they make a convincing case that it is not. Guess what? Not everybody thinks, feels or processes information like a college student. In fact, when considered in relation to a more representative population sample (those in America or Britain who never made it to University or those from different cultures who never even went to a ‘school’ never mind university) these WEIRD people score so far from the population average that they could be considered ‘outliers’ and in some experimental contexts their data would be excluded. Does this mean we should abandon all research with undergraduates? Of course not, many groundbreaking theories have been derived from this subset of the population. Looking at the bigger picture also does not in any way diminish the strengths, talents or processing advantages of American and British undergraduates. It just means that if we want to draw any conclusions about ‘human’ cognition, we have to cast a wider net.

This has some bearing on the SCALES project. Why would I be crazy enough to take on a project of this size, which is a major logistical and intellectual challenge and frequently keeps me awake at night? Simple: I am interested in how language impairments are associated with impairments in other aspects of development (or ‘co-morbidity’). The literature suggests very strong associations: children identified with almost any kind of developmental disorder (ADHD, dyslexia, autism, Down syndrome etc.) are very likely to also have additional language impairments. Likewise, children with language impairments meet criteria for at least one other disorder at higher than expected rates (e.g. while dyslexia affects approximately 5% of the general population, some studies report that as many as 50% of children with ‘specific’ language impairment also meet criteria for dyslexia). The problem is that, with a few notable exceptions, the majority of studies reporting these associations are based on clinically referred samples of individuals, and these samples are inherently biased. In fact, this is so well-recognised that it even has a name, Berkson’s bias. The bias is that children who are having multiple development challenges (delayed language, clumsy, problems with attention and behaviour) are more likely to be noticed by parents and teachers and then referred for extra help from therapy or education services. However, there may be lots of children in mainstream classrooms who have whopping deficits in marking verb tenses (i.e. ‘he walk to school’ instead of ‘he walks or walked to school’) but are getting on just fine. This would not only be theoretically interesting (is language ‘special’ in the developing brain?), but practically important. If the ‘deficit’ we see doesn’t interfere with learning, making friends, or the child’s self-esteem, should we spend limited resources ‘fixing’ it? The only way to find out the extent to which language impairments are associated with other developmental concerns is to reduce the effects of Berkson’s bias, and the only way to do that is to recruit a more representative, population sample. That is what the SCALES project is trying to do.

I certainly have a few ideas about what I think we will find. I would prefer to call these ideas hypotheses, based on the extant literature and my own observations of children with language impairments over many years and across many clinical and educational settings. No doubt these hypotheses also reflect some my biases. I guess the test comes when we look at the data. If, in four years time, we find that my hypotheses are completely unsupported will I hang my head in shame, or try to hide the fact that I was wrong in case people think I am ignorant? No – I’d certainly be intrigued but also I think quite excited. Being wrong is often what moves science forward, challenges the dominant view and inspires new ideas and theories. How WEIRD is that? Well, based on the scientists I have known, I think not very. They may represent a biased sample, but I hope not.


Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 1-75.

  1. I’ve often wondered about that whole tense marking thing. SLI researchers make it out to be the most important thing ever, but as far as I can tell it’s pretty much redundant in terms of conveying meaning – which is what really matters. I guess the argument would be that it’s symptomatic of some deeper language problem that does have a functional impact.

    • Tense marking is interesting for lots of reasons – theoretically because it has been taken as evidence of modularity of language. It can be a marker of more pervasive language impairment, but it is certainly rarely what gets kids referred and in my experience, very way down on the list of things to target in intervention. Also interesting is the fact that this in not a marker of language impairment in non-English speaking languages, highlighting the importance of language input and the ways that environment can interact with biological predispositions to disorder.

  2. “for suggesting that some highly experimental studies involving a small number of adults with a particular cognitive profile may not be relevant to the everyday lives of all people with autism”

    Not what I for one responded to. But autism research–all of it, of any quality in any area–does affect all autistics, so it is relevant to our everyday lives. As I learned, we cannot get away from it. We live its consequences every day.

    I did use what you wrote in the seminar by the way, with its complete context (as you asked), and in the further context of the long history of similar statements in the autism literature.

  3. This may seem like an obvious question, but as a student interested in research, bias is something I’ve always been both wary and curious of. Hence my question is, how can someone reduce bias if they’re not even conscious of their bias or its impact? Maybe sometimes the challenge is not in reducing bias, but is in fact recognising it in the first place- even if more researchers simply acknowledged and made others following on from their work aware of the inherent biases involved, this in itself would have an impact in whatever field they’re in. But, once again, sorry for my naivety.

    • Thanks for the question! A good starting place is to think critically about possible influences on your variable of interest. This will come from lots of reading and looking at examples of what other people have done and what they try to ‘control’ for. Another test is to ask ‘what do I think the result will be?’ then ‘why do I think that?’ are your hypotheses based on evidence, or opinions? Sometimes research questions are fuelled by observations and these observations can lead you to expect a certain outcome, but that is not the same as evidence. Finally, if you haven’t picked up the biases, you can be pretty sure that someone else will! We all have a love/hate relationship with reviewers, but they usually make good points and can point out alternative explanations worth exploring. Hope that helps!

      • That helps a lot, thank you for the reply!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: