Archive for March, 2012

Bias in research

I got a bit of flak from my last blog for suggesting that some highly experimental studies involving a small number of adults with a particular cognitive profile may not be relevant to the everyday lives of all people with autism. I was accused on Twitter and other places of being ‘biased’. At first I was a bit offended by this, but then I relaxed. One of the first things you learn as a psychology student is that everyone is biased. We can’t help it. It just seems to be part and parcel of the human condition. The second thing you learn though is that biases can be unhelpful and at worst, dangerous. In science biases can be particularly damaging, particularly if they cause scientists to ignore, or worse, suppress evidence that contradicts their biases, or prevents them from collecting or reporting such evidence in the first place (witness for example Susan Greenfield’s spectacular fall from academic grace). Scientists can’t completely eradicate biases, we are after all only human. The best we can do is to try and identify what our biases are, and take steps to minimize the impact of these biases on our scientific endeavours.

Thinking about bias brought to mind one of my favourite journal articles of recent years. The first thing I love about this paper is its wonderfully provocative title – the authors had the audacity to label me and people like me as WEIRD – white, educated, industrialized, rich and democratic. They then go on to make the observation that most psychology research, and more importantly the conclusions and theories about human perception and cognition derived from this research, is based almost entirely on WEIRD people, namely American (and British) university undergraduates. I can see why. I am deeply envious of my colleagues who can run a study on 50 undergrads in a matter of weeks, while it usually takes me a couple of years of travelling all over the place to have enough data to produce a publication! Nevertheless, the authors of this paper ask a simple question ‘are the data derived from an elite, 1% of the world’s population really relevant for the remaining 99% of human experience?’ And they make a convincing case that it is not. Guess what? Not everybody thinks, feels or processes information like a college student. In fact, when considered in relation to a more representative population sample (those in America or Britain who never made it to University or those from different cultures who never even went to a ‘school’ never mind university) these WEIRD people score so far from the population average that they could be considered ‘outliers’ and in some experimental contexts their data would be excluded. Does this mean we should abandon all research with undergraduates? Of course not, many groundbreaking theories have been derived from this subset of the population. Looking at the bigger picture also does not in any way diminish the strengths, talents or processing advantages of American and British undergraduates. It just means that if we want to draw any conclusions about ‘human’ cognition, we have to cast a wider net.

This has some bearing on the SCALES project. Why would I be crazy enough to take on a project of this size, which is a major logistical and intellectual challenge and frequently keeps me awake at night? Simple: I am interested in how language impairments are associated with impairments in other aspects of development (or ‘co-morbidity’). The literature suggests very strong associations: children identified with almost any kind of developmental disorder (ADHD, dyslexia, autism, Down syndrome etc.) are very likely to also have additional language impairments. Likewise, children with language impairments meet criteria for at least one other disorder at higher than expected rates (e.g. while dyslexia affects approximately 5% of the general population, some studies report that as many as 50% of children with ‘specific’ language impairment also meet criteria for dyslexia). The problem is that, with a few notable exceptions, the majority of studies reporting these associations are based on clinically referred samples of individuals, and these samples are inherently biased. In fact, this is so well-recognised that it even has a name, Berkson’s bias. The bias is that children who are having multiple development challenges (delayed language, clumsy, problems with attention and behaviour) are more likely to be noticed by parents and teachers and then referred for extra help from therapy or education services. However, there may be lots of children in mainstream classrooms who have whopping deficits in marking verb tenses (i.e. ‘he walk to school’ instead of ‘he walks or walked to school’) but are getting on just fine. This would not only be theoretically interesting (is language ‘special’ in the developing brain?), but practically important. If the ‘deficit’ we see doesn’t interfere with learning, making friends, or the child’s self-esteem, should we spend limited resources ‘fixing’ it? The only way to find out the extent to which language impairments are associated with other developmental concerns is to reduce the effects of Berkson’s bias, and the only way to do that is to recruit a more representative, population sample. That is what the SCALES project is trying to do.

I certainly have a few ideas about what I think we will find. I would prefer to call these ideas hypotheses, based on the extant literature and my own observations of children with language impairments over many years and across many clinical and educational settings. No doubt these hypotheses also reflect some my biases. I guess the test comes when we look at the data. If, in four years time, we find that my hypotheses are completely unsupported will I hang my head in shame, or try to hide the fact that I was wrong in case people think I am ignorant? No – I’d certainly be intrigued but also I think quite excited. Being wrong is often what moves science forward, challenges the dominant view and inspires new ideas and theories. How WEIRD is that? Well, based on the scientists I have known, I think not very. They may represent a biased sample, but I hope not.

Reference:

Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 1-75.

A week in March

Sunday was Mother’s Day. I woke up in my mother-in-law’s house with a horrendous cold and conjunctivitis, my left eye swollen and weeping. My daughter was glued to the enormous television in the lounge (a rare treat for her) and I could hear my usually patient husband, now weary with ill wife, elderly mother and active 3-year-old, imploring the latter to come a wish me a Happy Mother’s Day. NO! was the resounding response. My mother-in-law admired the bouquet of flowers resting in water in the kitchen. ‘They’re for Courtenay’ my husband said. She came in to thank me for them. Not the restful start to the week I had in mind.

By Monday lunch time my eye had recovered enough to let me get on with some work. The main challenge for the SCALES project at the moment is to recruit as many schools as possible to take part in the screening phase. Ideally all 300-odd schools would say yes, but that is unlikely to happen. Realistically, we are aiming for 80% of all schools. In reality, just over 40% have agreed so far. This is over 5,000 children, and we’ve still got time, but recruitment is starting to worry me. Only 3 schools have said no, many just haven’t responded. I understand this – schools are busy places and my research project just isn’t so high on the list of priorities. But for a population sample, we need them. So I spend a fair bit of time drafting more emails, outlining what is involved and what we can offer in return. I also work on updating the database, redrafting the advertisement for the graduate research assistant and phoning various schools that have requested further information. When that is done I deal with other admin –scheduling student meetings, booking travel for the week ahead, submitting a paper revision, and writing three references for current and former students.

Tuesday I leave very early to make a 9am meeting at RHUL. I see my MSc student first – she too is having trouble with recruitment and we are discussing the options, plus possible changes to the experiment following pilot testing. I then have end of term meetings with undergraduate personal advisor students. I think this is what GPs must feel like – 10 minute slots to see how people are doing, with the serious issues raised about two minutes before the next person is due to arrive. I’m always behind schedule but today I need to leave promptly as I have a ferry to catch.

I’m visiting a special school on the Isle of Wight and arrive at the end of the school day. I’m here to collect data for an ESRC funded project we’re running that is using eye-tracking to explore language production in children with language disorders and/or children with autism spectrum disorders (ASD). I’m also here to celebrate with two of the speech-language therapists – we’ve just submitted a paper together! I have an hour or two to meet with my PhD student and the post-doc working on the project. Then we go to dinner to meet the therapists. When I came to RHUL, I set up the LiLaC (Literacy, Language and Communication) network, with the express aim of getting researchers and clinicians together to foster collaboration and make research less mysterious to the people most likely to use it. These therapists were the first to take up the LiLaC challenge – they wanted to evaluate a therapy programme they offered to teenagers with severe language and communication disorders and we helped them to do this. They are very excited about the submission and I feel a bit of a killjoy as I try to manage expectations. After all, they have never experienced the joys of peer review and I don’t want them to be too bruised by the experience. By the way, those of you dealing with REF and impact (see Dorothy Bishop’s excellent comments on this) may be interested to know that despite the fact that this collaboration has clearly changed working practice, encouraged publication of a manual that will be used by other therapy teams, and has direct implications for employability of young people with communication disorders, this does not meet the REF definition of impact – extremely annoying!

Anyway, on Wednesday I finally get to see the kids. I don’t really have time to do lots of testing anymore, but I do love it and make time whenever I can. It really keeps me grounded; the kids in this school are at the severe end of the spectrum and a day with them is very humbling. Of the vast number of research papers published on ASD, I wonder how many have any relevance to the lives of these people. For example, a study is published this week suggesting that people with autism have ‘an information processing advantage’. I haven’t read the paper in detail and unlike the participants of that study, we are investigating children with a wide range of abilities, but I just don’t see the advantages of being able to name all the racetracks in Europe, of bringing Mr Men into every conversational exchange, of asking each new person met for their list of personal dislikes, or being able to notice and then label every tiny detail on the screen when asked for a simple picture description. I am equally intrigued by the difficulties faced by the non-autistic children with language impairment. We met a bright boy of almost 14 who was unable to name any letters of the alphabet, despite extensive intervention – what is the limitation? None of our current theories seems to capture the complicated picture of strength and difficulty we are presented with.

We leave the island on Thursday afternoon. I haven’t slept much as I’m still coughing and still puzzling over the assessments we’ve done and how we’ll make sense of the data we’re gathering. It has been good to spend some time with the team – the PhD student took the opportunity to outline her analysis and writing plans for the next 6 months; the post-doc is coming to the end of his contract and is applying for a first lectureship in the most brutal academic climate. We get back to RHUL in time to drop off the testing kit and I neglect running club to race home and collect my daughter from nursery.

Friday sees me in a mainstream primary school, not for assessment, but because BBC London want to do a piece on the SCALES project and are filming us in Cranmere School. Jo Slessor is the wonderful Special Educational Needs Co-ordinator there and one of the first to sign up to the project. She has kindly agreed to speak on camera about the educational implications of language difficulties and has even persuaded a very articulate and thoughtful mum to come in with her son. A few years ago I did a Media Fellowship with the British Science Association, hoping that this would put me more at ease when speaking to the press. I spent a splendid few weeks in Dublin writing for the Irish Times, phoning scientists and chatting to them about work in progress. Unfortunately we rarely hear about work in-progress in the mainstream British press, instead we just get the headline news when results are finally published. That is why I’m so pleased the journalist from the BBC has taken an interest in our story. It takes a very long time to get the footage they need and the school staff and children are incredibly patient. I’m impressed with the journalist and how she has listened to what I said I wanted to emphasize when we spoke about it last week. Everyone is nervous, but they do a brilliant job and I am optimistic that it will be a good piece that raises awareness of language impairment (you may be able to see for yourselves – it should go out on Monday, 26 March at 6.30pm).

After filming I go back to RHUL to meet with the post-doc who is my right-hand woman on the SCALES project. We review all the hard work she’s done while I’ve been away and she adds to my long list of things to do. I realise that I will have to negotiate a few hours with my husband on Sunday to get it all done, but she has definitely lightened the load.

As I drive home exhausted on Friday, I think about all the different roles I have assumed in the week: teacher, mentor, clinician, public speaker, administrator, wife and mother. Sometimes, especially weeks like this when I’m run down and not feeling my best, I think it is all too much. It’s an impossible job with too many different strands to manage at all well. But then I reflect that this variety is actually one of the things I love most about my job. Every day is different and every day brings me into contact with fascinating people leading challenging but fascinating lives. I am never bored – my brain is always ringing with questions and ideas and this helps me to keep going when it is a bit overwhelming. And so now I look at the coming week and see that it is just as manic, but just as interesting. My cold has pretty much gone and yesterday my husband and daughter took me for afternoon tea on the rooftop terrace of the Ashmolean to celebrate Mother’s Day. Things are looking up.

A lesson in persistence

Academic life can be brutal at times. I became an academic because there were questions that I wanted to answer about why some children fail to learn their native language as expected. The only way to pursue this research question is to convince another group of academics that my ideas are valuable and worth funding. This involves writing a grant – a proposal that details how I will answer the question, why it is important, what methods and materials will be used, how many research staff will be required, how the findings will be communicated and what the impact of those findings will be. Anyone who has ever written a grant will know that grants represent a tremendous amount of work and a huge personal investment.

The seeds of the SCALES grant were sown way back in 2007. Gillian Baird and I were attending the 4th International Afasic Symposium and spent a pleasant evening discussing Bruce Tomblin’s talk and the epiSLI study (more about that later). We discussed the possibility of a UK population study of risk for language impairment, one that would take co-morbidity into account from the earliest stages and one that measured the impact of language impairment on the child’s everyday life. Conclusion: let’s write a grant…

We got a fantastic team together (more about them later!) but the first hurdle we faced was that a population study would require screening a large number of children in order to identify our target sample and none of the available screening measures were really up to scratch. So we decided to write a preliminary grant that would test two screening  methods, direct child assessment or teacher report of everyday language and communication. In testing the screen, we would also assess how many identified children had ‘specific’ language impairments and how many had additional developmental concerns.  It seemed a very straightforward proposal: theoretically interesting, clear hypothesis, clear implications for policy and practice.

The rejection letter and five sets of reviewer comments of varying degrees of venom and hostility arrived four days after the birth of my first baby. (Some of you may be wondering why on earth I checked my email four days after giving birth – I can only claim post-partum denial of reality).  Now every academic has experienced grant rejection and I suspect most find it quite upsetting. But in my hormonal, sleep-deprived state I was completely inconsolable. Convinced that my research career, indeed my life as I’d known it was over, I resolved to spend the rest of my days trying to feed my screaming baby. I sent my co-applicants another email. It went something like ‘our brilliant grant has been completely rubbished. Oh, and I’ve had a beautiful baby girl.’ Fortunately they are a wise and wonderful team and helped me to put things in perspective!

I put it behind me, wrote a different grant (that got funded), adjusted to parenthood and went back to work. About a year later we reconvened and decided to go with what we believed was the best available screening measure and write the grant we really wanted to do. Now, we knew that ideally we would follow children over time, but longitudinal projects are very expensive to run and don’t fit into the standard 3-year grant scheme. I was also a relatively new and untested investigator, so it was unlikely that anyone would trust me with such a large project. So we played it conservatively –screen at reception, assess in Year 1 and investigate language deficit and co-occurring difficulties.

This time I was more mentally prepared for the rejection letter and was somewhat cheered by the fact that the five reviewers were for the most part extremely positive. But my good cheer soon gave way to deep frustration – unlike other countries, here there is usually no option to revise and resubmit a grant. It now looked like our project would fester in the filing cabinet.

I don’t give up easily though – this was a project that I really believed in, we had terrific support in Surrey and it just had to happen. So I phoned Ailish Murray at the Wellcome Trust and I asked for her advice about what to do next. She was fantastic- she told me that the panel had been enthusiastic about the project but felt that a longitudinal design would be best suited to answer our primary research questions. Although they would not take a resubmission, we could submit a new, 4-year grant with different questions and a different method.

One more chance. As luck would have it, our last chance coincided with the last ever round of Wellcome project grants. Half of England would be putting a grant in to this round. I found myself in a big grant-writing slump – it seemed a completely hopeless endeavour and a total waste of time. I’m also not great at endlessly revising and rewriting things, but my very thoughtful and experienced collaborators sent back draft after draft with comments and questions and pushed me to keep going.

A particularly challenging aspect of this process is that we all know when the panel meets, so there is a period of time in which a hand full of people know your fate, but you are not one of those people. This purgatory was longer than usual this time, given the large number of applications received. At the very same time, my husband was waiting to hear about a grant submitted to a different funding body. We looked at the month ahead and realised we’d either be drowning in gin or swimming in champagne. We tried not to dwell on the other possibility that one of us would get funded while the other would be disappointed (again).

My husband heard first – good news! I was thrilled for him, but figured that was the death knell for my grant. (I know the two events are causally unrelated, but it didn’t seem possible that we’d both get funded). I stopped holding my breath every time I opened my email. Then one day I was on a bus in London, looked at my phone, and saw an email from Ailish Murray followed by messages from every one of the co-applicants. I didn’t want to cry on the bus, so for a split second thought I might wait until I got home to open them. Of course I didn’t do that. I must have shouted at an inappropriate bus volume because the woman next to me said ‘is everything alright?’ And it was! We did have lots of champagne and I was elated for weeks. Then I read the grant again and realised I would now have to do all the things I said I was going to do…

Surrey Communication and Language in Education Study (SCALES): an introduction

Many years ago I had the amazing good fortune to work as a research assistant for Dorothy Bishop. My job was to drive the length and breadth of Britain assessing six-year-old twins, many of whom had language impairment. I was a speech-language therapist at the time, so I knew a fair bit about assessing young children with language and communication difficulties. But I knew nothing about research and even less about the reality of gathering the amount of data necessary to explore the genetic and environmental contributions to language impairment. I was, however, up for a new challenge and keen to work with Dorothy and to travel around this fair island in the name of scientific discovery.

It was an amazing experience. Certainly there were times when it was monotonous and I never wanted to see another child or motorway again. As I look back though, I just remember how much fun I had and how much I learned. My only regret is that I never wrote any of it down. Most of the time I was just too busy entering data or figuring out how to get to my next destination (in the days before SatNav) to keep a journal. Yet just about every day brought an experience that I wanted to share with someone else.

A great example was my first home visit – a story I tell to all new students and research assistants in my lab. I went to see twin boys, one of whom had been expelled from school for his behaviour (at the age of six remember). When I arrived, mum explained that he had been diagnosed with dyspraxia and was often unintelligible. He was also not to keen on my visit. In fact, he was under his bed and no amount of temptation with stickers and Pokemon cards could entice him to come out. In desperation, I suggested to mum that I would just carry on setting up my equipment and see if he might become interested in me. When I got to the bottom of the stairs he appeared, hurling shoes and toilet rolls at me and shouting with great clarity “fucking off, you fucking idiot.” His poor mum pleaded with him to stop swearing while I dodged missiles and questioned the wisdom of my career change. Happily though, once his brother started my activities, he became very interested in what we were doing and eager to demonstrate that he could do it too. He tried extremely hard and we managed to complete the session successfully.

Fortunately not every home visit was like that, but I did face many quirky challenges such as chickens in the kitchen, finding the small difference that distinguished identical twins, engaging the child who just wanted to bite my ankles and recognising the signs that the loo is required by an over excited child who does not want to be parted from the dinosaur game. And those were the participant challenges – I also had to endure a number of colourful accommodations and over time perfected the art of dining alone. In total, I personally saw about 200 twin pairs and travelled as far north as Newcastle and as far south as the Cornish border. That’s more than 1,000 hours of testing and many thousands of driving miles (I’ve probably been in more motorway service stations than Alan Partridge). Thanks to Dorothy’s brilliance, we published numerous journal articles and made exciting new discoveries, though I often feel that the publications just don’t do justice to the stories behind the numbers.

So now I find myself the Principal Investigator on SCALES, a large project funded by the Wellcome Trust. SCALES will be the first epidemiological study of risk for language impairment at school entry in the UK. It will involve screening a population cohort, in-depth assessment of approximately 500 children and a longitudinal follow-up of those children over a four year period. It involves a large research team and extensive collaboration with Surrey County Council, local schools and families. My life circumstances have changed since the twin days – I’m a lecturer in Psychology and have a 3-year-old at home, so there are many more demands on my time. But I am keen not to make the same mistake twice and allow my challenges and experiences to go unrecorded. SCALES is the perfect opportunity to record what actually goes in to making projects like this succeed and to highlight the personal stories that make science work. I hope the blog will be fun and informative, documenting both the drudgery and exuberance of a life in science.