Doing Applied Linguistics

Groom, N and Littlemore, J (2011). Routledge: London

A.K.A. Sampling Bias A-Go-Go

Some of the most common advice on structuring a speech is as follows: tell them what you are going to say, then say it, then tell them what you said. The authors of this book have taken that advice very much to heart, so much so that I’m half tempted to use the Introduction and Conclusion sections of each chapter as a basic grammar exercise for some of my students, comparing the future and past tenses in English in otherwise unchanged passages.

It is an introductory text though, so there’s value in covering the basics thoroughly. The opening chapters (1-4), specifically addressing the field of Applied Linguistics – what it is and does – are something I think I’ll returning to repeatedly as I get up to speed with the terminology. I have a bit of prior experience with social science (and indeed physical science) research methodologies though, so the chapters addressing these were less personally helpful. Not aimed at the likes of me perhaps. Fair enough.

However, “[i]t is also an important part of the applied linguist’s remit to go about creating problems – or more precisely, to go about identifying problems which have hitherto gone unnoticed” (p12). So let’s take the authors at their word and pick some holes in this baby, shall we?

Hand biting to commence in 3, 2, 1…

  

The biggest issues I had were with the chapters 5-8, on collecting and analysing qualitative and quantitative data. We can start right there with the titles: “Collecting Qualitative Data” and “Collecting Quantitative Data.”

Collecting? Data are produced, they aren’t ‘collected’. The act of research is an explicitly productive act on the part of the researcher, and the data thus produced may or may not be more or less reflective of the researcher’s biases and prejudices, both conscious and unconscious.

Data don’t grow on trees like apples. There isn’t a data orchard you can wander merrily through, plucking a shiny ripe datum as you please to assuage the thirst you worked up that morning whilst laboring in the field harvesting questionnaires. But wait! What’s this, why if it isn’t another of those pesky data scrumpers from the next village! No time to enjoy that juicy datum, quick, get the shotgun! Only for a warning shot, mind, but the little tearaways have to learn. You’ll put the fear of God into the little blighters. There’s no more fearsome sight in all of academia than a data farmer chasing a group of raggedy-kneed scamps across the questionnaire fields, straw hat on head, shotgun in hand, hurling forth a spittle-flecked bellow of “GET OFF MY LAAAAAND!” (cf. MacDonald, O. 1917).

Groom and Littlemore also seem curiously down on quantitative research. I’m not sure how the same authors in the same book can state both this on quantitative data:

              “…questions were the researcher’s questions and were not put together by the participants themselves, and this may therefore shape the answers that are given.’ (p105, emphasis in original)

This on text analysis:

              “No matter how rigorously it is carried out, text analysis is fundamentally an act of interpretation.” (p148)

And this on qualitative:

              “In this approach [grounded theory], the researcher starts off with no preconceptions at all about the data.” (p87)

No preconceptions about the data at all? Then why, pray, did they bother to ‘collect’ the fucking stuff in the first place? Last I checked, the Monte Carlo method was a pretty firmly established quantitative approach, and even that ultimate expression of randomness in research methodology is essentially grounded in what the researcher expects to find.

              “… unmotivated looking… means studying a transcript completely inductively, which is to say, without having any prior conception of what might be found within it.” (p88)

In the spirit of inquiry, last night I tried some unmotivated cooking for my wife, producing dinner completely inductively, that is to say, without having any prior etc etc and so on:

              -“What are we having for dinner, Honey?”

-“Well, I thought we’d have par-boiled popcorn garnished with kimchi and dishwasher powder, served with a side of tinfoil and the abstract concept of ‘doubt.’ Sound good to you?”

She, apparently, will get custody of the children.

There are serious ontological and epistemological points here. While it’s a pleasant change to see social scientists showing a bit of confidence in their methodology, the claim that anyone, anywhere – let alone a researcher with a background in the specific field that they are researching (and is there any other kind?) – can view anything ‘without having any prior conception of what might be found within it’ is demonstrable nonsense. To let comments like this stand unqualified or unmitigated in what is supposed to be an introductory text seems like an obvious and glaring failing. Let’s cherry-pick some more quotes, shall we? Note the page numbers.

              “Factor analysis… thus provides an objective manner of identifying the way in which certain items group together.” (p120, emphasis in original)

              “Labelling the factors in a more meaningful way is the researcher’s job.” (p121)

              “This… prevents the researcher from imposing his or her own categories on the research.” (p122)

Hmmm. So for all that the categories themselves have been arrived at ‘objectively’ (and where to start with that claim?) it’s still the act of labelling them which provides most of their meaning. That’s hardly a watertight method of eliminating bias. In fact I would respectfully suggest that as preventative measures in general go, the withdrawal method is more effective and, er, rigorously applied.

* * * * *

              “[There is] …a widespread dilemma in social science research: the trade-off between a desire to collect data that can be analysed statistically and then generalized to a wider population, and the need to get accurate in-depth data about what people actually think.” (p102)

This seems a little apples and oranges to me. I refer you back to my query regarding motivations for data collection; why exactly would one feel they ‘need to get accurate in-depth data about what people actually think’? What’s the point? How many ‘people’? The implication is that it’s a small number, because surely at some point ‘people’ must necessarily bleed into the ‘wider population’. Individuals? Small groups? Battalions? It’s the apocryphal Amazonian tribe whose numbers consist of ‘one, two, few, many…’

I’d dearly love some in-depth data regarding my wife’s thought processes, and I’m pretty sure that right now certain world leaders would like the same for Kim-Jong Un and the politbureau of the CCP, but what value is there in gaining an in-depth understanding of an ordinary, anonymous E2L learner if not because you hope it might have wider applications? There are obviously serious issues to consider when scaling up, but if the understanding of an individual is only and exclusively applicable to that individual wouldn’t it be a hideous waste time and effort only to arrive at conclusions with such narrowly specific applicability?

Which leads me to that sampling bias. The vast majority of research examples given in this book concern E2l learners. Given AL’s roots in TEFL that seems entirely appropriate, but the few examples that aren’t from English Language classrooms are of academics or university students. The tendency for researchers to plump for the most easily available study group is endemic across all disciplines, but seems especially relevant considering those questions of applicability and bias. At this point I can’t decide whether it’s a bug or a feature.