Research Methods for Applied Linguistics: Scope, Characteristics, and Standards

Over the past two decades, books on applied linguistics research have often failed to define the term research, perhaps because the focus of such books was so narrow that a definition of research seemed self-evident. Some books (Anshen, 1978; Hatch & Farhady, 1982; Butler, 1985; Woods, Fletcher, & Hughes, 1986; Brown, 1988; Hatch & Lazaraton, 1991; Rietveld & van Hout, 1993; Scholfield, 1995) focused almost exclusively on statistical research. Other books focused on the research methodology involved in language teacher research (Freeman, 1998), language classroom research (van Lier, 1988; Allwright & Bailey, 1991), action research (Wallace, 1998; Burns, 1999), survey research (Brown, 2001), research methods in text and discourse analysis (Jenner, 2000), or research methods in pragmatics (Kasper & Dahl, 1991). Recently, collections of articles have also begun to appear that are focused on a specific type of research, like teacher research (Griffee & Nunan, 1997; Hornberger & Corson, 1997), or qualitative research (Bailey & Nunan, 1997).

Other research-oriented books in applied linguistics have been more inclined to grapple with the notion of what research is, perhaps because they were somewhat more general in nature, usually surveying a variety of different types of research. The earliest of these explored the differences between qualitative and quantitative research (Seliger & Shohamy, 1989). A few years later, Johnson (1992) covered correlational approaches, case-study approaches, survey research, ethnographic research, experimental research, and what she called multisite/multimethod research, and Nunan (1992) explained experimental method, ethnography, case study, classroom-observation research, introspection methods, elicitation techniques, interaction analysis, and program evaluation. Still more recently, McDonough and McDonough (1998) dealt with observation, introspection, diary studies, experiments, interviews, questionnaires, numerical techniques, and case study research, and Brown and Rodgers (2002) included chapters on case study research, introspection research, classroom observation and interaction research, descriptive statistics research, correlational research, quasi-experimental research, and course or program evaluation.

In Brown (1992), I reported on a survey of hundreds of members of the Teachers of English to Speakers of Other Languages (TESOL) organization around the world. One of the questions I asked them was how they defined research. The diversity of answers was staggering, ranging from short, idealistic answers about what research is (e.g., “Careful, thorough study” and “The search for the truth”) to very cynical answers (e.g., “Something that profs at universities that grant advanced degrees do because they don’t teach and need to publish” and “Ignoring the obvious”). The approaches to defining research differed in fairly systematic ways, falling generally into four categories as follows:

1. Definitions that listed the types of research (e.g., “An investigation of a particular topic, or problem, through a document search and/or empirical study (the conducting of experiments) and analysis” and “Investigation through the reading of literature, experimentation and/or any other type of data gathering . . .”)

2. Definitions that listed the topics of research (e.g., “In its widest sense, to seek new ways to improve language education and intercultural communication training” and “Searching for information on how students process information, internalize data and retain it for communicative purposes.”)

3. Definitions that covered the purpose of research (e.g., “The search for information that will help practitioners (in this case, teachers) better carry out their jobs . . .” and “Systematic study of language issues and use in order to improve delivery of services to our students.”)

4. Definitions that enumerated the steps in the process of research (e.g., “Working toward truth, proving theories, trying out new approaches – and then compiling results, analyzing results and sharing with colleagues” and “Stating a hypothesis; gathering data; testing the hypothesis; relating the conclusions to issues at hand.”)

In Brown (1988), I naively categorized the different types of research as shown in Figure 19.1 into two categories that my librarian mother taught me years ago: secondary research (derived from the research and writings of others) and primary research (derived from original data of some sort). I further subdivided primary research into case studies and statistical research and then subdivided statistical research into survey and experimental research. In the mid-eighties, I apparently viewed the types of research going on in applied linguistics in such simplistic binary terms.

Applied linguistics research can be described from many different perspectives including at least (1) the contextual factors involved in applied linguistics research, (2) van Lier’s parameters of educational research design, (3) Grotjahn’s data collection methods, data types, and data analysis procedures, (4) other sets of research characteristics, and (5) the qual–quant continuum.

Brown and Rodgers (2002, pp. 14–16) summarized a number of contextual factors that influence applied linguistics research at international, national, professional, institutional, local, and personal levels:

1. International and national contexts. International and national organizations and governmental bodies support a fair amount of applied linguistics research. Since they control the money, their political priorities tend to influence who will do such research and how.

2. Professional contexts. Within applied linguistics, the types of research that are popular at any given time vary; for a few years, interest in one type of research may increase at the expense of another, then interest may be rekindled for that latter type. In other words, even research can have its trends and fads.

3. Institutional contexts. Institutional contexts in applied linguistics research can refer to everything from entire school districts, to individual language programs, or even to very specific individual tutoring situations. Within these institutional contexts a number of factors can influence the type and quality of research: the size of the institution, availability of resources to support research, institutional policies and priorities, the institution’s past experiences with researchers, and even the personalities of the various administrators and teachers involved.

4. Local contexts. Local contexts refer to the specific circumstances in which the research will take place. The context may be a classroom, laboratory, private home, Internet bulletin board, or even a coffee shop. A number of factors in the local context may turn out to be important to the success or failure of a research study:

a). physical context (e.g., class size, layout of the school, etc.),

b). time context (e.g., minutes per class, classes per day, etc.),

c). social context (e.g., language backgrounds, ethnic mix of the students,

etc.),

d). pedagogical context (e.g., teaching methods used, preferred learning

styles of the students and teachers, etc.),

e). psychological context (e.g., comfort level of participants with regard to research studies, etc.).

5. Personal contexts. Individual researchers have certain preconceptions about the aspects of applied linguistics that ought to be researched, the form that research should take, and their role in the research process. Such preconceptions and preferences arise from individual differences in abilities, personalities, motivations, priorities, training, etc. and may influence the types of research a particular individual or group will be interested in doing.

On the selectivity axis, research can be highly selective in focus (e.g., obligatory use of the definite article by immigrant Chinese adolescent men from Guangdong province), or non-selective (e.g., all language related behaviors observed of all participants in a population of students). The selectivity and intervention axes also create what van Lier calls four territories:

1. controlling, in which the researcher conducts a carefully planned experiment restricted both in participants and content focus;

2. measuring, in which intervention is minimal but the data focus is highly restricted;

3. asking/doing, in which the researcher might intervene to ask participants to talk about what they are thinking;

4. watching, in which the researcher just observes with both intervention and selectivity kept to a minimum.

Each of these territories accounts for different types of research and has different potential audiences (what he calls audience inhabitants).

According to Grotjahn (1987), research can be classified in terms of data collection methods (i.e., experimental vs. non-experimental), data types (i.e., qualitative vs. quantitative), and data analysis procedures (statistical vs. interpretive).

The topics discussed above (i.e., contextual factors, intervention and selectivity, data collection methods, data types, and data analysis procedures) are all interesting and useful ways of characterizing applied linguistics research, but unfortunately, such research is far more complicated than any of those sets of categories would suggest because it also includes at least the following additional concerns: time orientation, theory generation, variable description, and researcher perspective.

Another way to classify research is according to time orientation, or the amount of time invested in gathering data. Along those lines, studies are sometimes classified as either cross-sectional or longitudinal (e.g., see Larsen-Freeman & Long, 1991). Cross-sectional studies are those conducted over a short period of time, often with a relatively large number of participants. For instance, a study might gather language proficiency data (using a test), motivation data (using a questionnaire), and personal information data (on the same questionnaire) from 300 students in a one-shot cross-sectional study.

Studies can also differ in terms of theory generation, which can take two forms: hypothesis forming and hypothesis testing. Hypothesis forming research may begin with some very general framing questions, but will typically have no hypotheses to start with. In such a study, the researcher will make every effort to keep an open mind and form hypotheses about what is going on only after a great many observations of various sorts. The resulting hypotheses are typically considered part of the interpretation, and the researcher often goes back to the participants to ask them if the hypotheses are reasonable (in a process called member checking). One of the great strengths often cited for qualitative research is its potential for forming new

hypotheses.

Similarly, variable description can take two forms in applied linguistics research: variable definition and variable operationalization. In variable definition research, the researcher attempts to begin with no preconceived notions of what the important variables in the study will be, or how they will be defined. As the study progresses, the process of discovering and describing variables serves to gradually define them.

The researcher’s perspective is sometimes described as either emic or etic. Researchers adopting the emic perspective make every effort to understand the point of view of the participants and to examine how the interpretations drawn from the research relate to those views through practices such as member checking (getting the participants’ reactions to the interpretations drawn from the research). Researchers assuming the etic perspective take an outsider’s view during the data gathering process, often attempting to be as objective as possible (for more on the emic/etic distinction, see Davis, 1995, p. 433).

For some applied linguistics the key distinction among the different types of research is that between qualitative and quantitative research. Indeed, in the Department of Second Language Studies at the University of Hawai’i, where I work, the research courses begin with an introductory course that introduces the basic concepts of qualitative and quantitative research methods. Then the students can take courses in one of two strands: one series for those interested in qualitative research and another for those who want to learn about quantitative research methods.

Reichardt and Cook (1979, p. 10) summarize the differences between qualitative and quantitative research as shown in Table 19.2. Notice that the column on the left in Table 19.2 is labeled “Qualitative paradigm” and that the column to the on the right is labeled “Quantitative paradigm,” with each column containing adjectives and adjectival phrases describing each “paradigm.” A number of these distinctions are uncontroversial and make eminent sense; for example, the fact that the qualitative paradigm advocates qualitative methods and the quantitative paradigm advocates quantitative methods and the fact that the qualitative paradigm is typically naturalistic where the quantitative paradigm would more accurately be characterized as controlled.

Such a qualitative versus quantitative approach also has a number of general problems:

1. Dichotomizing qualitative versus quantitative research leaves out altogether secondary research types like literature reviews.

2. It treats as monolithic at least seven very distinct qualitative research techniques (case study research; introspection research; discourse analysis research; interactional analysis research; classroom observation research; interviews; and questionnaires).

3. It represents as monolithic at least ten qualitative research traditions that come from a variety of other fields like anthropology and theology.

4. It presents as monolithic at least six very different quantitative research techniques (interviews; questionnaires; descriptive; exploratory; quasiexperimental; and experimental).

5. It ignores the way survey research, including interviews and questionnaires, is both qualitative and quantitative.

6. It ignores the ways researchers often combine qualitative and quantitative research techniques.

7. It confuses research methods (interpretive, survey, and statistical) and research techniques (like those listed in the second and fourth points above).

Perhaps a more constructive and accurate approach would be to view qualitative and quantitative research as a matter of degrees, a continuum, rather than a clear-cut dichotomy.

In short, any particular research project can be viewed as combining any possible combinations of the points along the 12 research characteristics continua. For example, one research project might be characterized as being in the middle of the data type continuum because it uses both quantitative and qualitative data in equal proportions; the same research project might be leaning toward the non-experimental end of the data collection methods (say 20 percent along it) because it is largely observational but uses one set of test scores; the project might also be considered 60 percent along the data analysis methods continuum because the analyses are somewhat more statistical than interpretive; and so forth.

Notice that the tree above the research characteristics continua restricts itself to primary research (because secondary research has a whole set of different characteristics), then shows three general primary research methodologies (interpretive, survey, and statistical) and the many different research techniques (ranging from case studies to experimental) in applied linguistics that tend to array themselves from the qualitative-exploratory to the quantitative experimental end of the qual–quant continuum. I emphasize the word tend in order to allow for the possibility that any of these various research techniques could have research characteristics anywhere along the various continua. In short, the research techniques ranging from case studies to experimental are convenient categories that can help us to understand the world of applied linguistics research, but at the same time, they may lead us to oversimplify the true state of affairs.

The standard of reliability in quantitative research requires researchers to demonstrate both (1) the reliability of the instruments used in their studies and (2) the reliability of the results of their studies. Reliability of instruments is concerned with the degree to which the results of a questionnaire, test, or other measuring instrument are consistent. Addressing this issue typically means answering the question: To what degree would the results be the same if the instrument were administered repeatedly? Similarly, the reliability of the results of a study is concerned with the degree to which the results would be likely to reappear if the study were replicated under the same conditions. Any expectation that the instruments or the results of a study would ever be 100 percent reliable would be unreasonable. However, the reliability of quantitative research can be maximized by carefully designing, piloting, and validating any measures involved and by carefully planning and designing the research from the beginning. In addition, ample statistical tools exist to help researchers study and report the degree to which their measures are consistent.

The standard of replicability in quantitative research requires researchers to provide enough information about a study to allow other researchers to replicate or repeat the study exactly as it was originally conducted. The replicability of a quantitative study can be improved by thorough and complete descriptions of: (1) the participants in the study and how they were selected, (2) the instruments used in the study as well as arguments for their reliability and validity, and (3) the procedures followed in collecting the data, scoring or coding the instruments, and analyzing the results (see Brown, 1988, ch. 5 for a more complete discussion of what should be included to make a study adequately replicable).

The standard of validity in quantitative research requires researchers to demonstrate both internal and external validity. Internal validity is the degree to which the results of a study can be accurately interpreted as meaning what they appear to mean. External validity is the degree to which the results of a study are contrived or artificial, or put another way, the degree to which the results apply to the outside world. Listed below are a number of issues identified by Campbell and Stanley (1963) as potential threats to the internal and external validity of a study:

1. History

2. Maturation

3. Testing

4. Instrumentation

5. Statistical regression

6. Selection bias

7. Experimental mortality

8. Selection-maturation interaction

9. Reactive effects of testing

10. Interaction of selection biases and the treatment

11. Reactive effects of experimental arrangements

12. Multiple treatment interference.

As mentioned above, the standard of generalizability in quantitative research requires researchers to show the degree to which the results of a study can justifiably be generalized, or applied, to a larger population or to other similar groups. A study can be extremely well designed, controlled, and internally valid, but lack external validity. The problem is that controls, when introduced, may make the study artificial and thus limit the external validity. In one sense of the word, generalizability is clearly related to the concept of external validity discussed in the previous paragraph.

The standard of dependability in qualitative research requires that researchers account for (1) any shifting conditions directly related to the people and things they are studying and (2) any modifications they have made in the design of their study as it has progressed. The purpose of such accounting is to help researchers and their readers get a more exact understanding of the context. Dependability is roughly analogous to the concept of reliability (described above) in quantitative studies. The dependability of a qualitative study can be improved by using such techniques as stepwise replications, overlapping methods, and/or inquiry audits (for more on these concepts, see Davis, 1992, 1995; or Brown, 2001).

The standard of confirmability in qualitative research requires that researchers fully reveal the data they are basing their interpretations on, or at least make those data available. The point is that, whether or not anybody actually takes the researchers up on it, they should make their data available so that other researchers could examine them and confirm, reject, or modify the original interpretations. The confirmability of a qualitative study is approximately analogous to the concept of replicability (described above) in quantitative studies. Confirmability can be improved by using audit trails (as described briefly in Davis, 1992, 1995; Brown, 2001; or in more detail in Lincoln & Guba, 1985).

The standard of credibility in qualitative research requires researchers to show that they maximized the accuracy of their definitions and their characterizations of the people or things under investigation – especially as the various participants in the study judged those interpretations. Credibility is more or less analogous to the concept of internal validity (described above) in quantitative studies.

The standard of transferability in qualitative research requires researchers to describe the research design, context, and conditions so well that the readers can decide for themselves if the interpretations apply to another context with which they are familiar. Transferability in qualitative research is approximately analogous to the concept of generalizability (described above) in quantitative studies.

A study may need to focus entirely on the standard at the qualitative-exploratory end of the continuum or the standard at the quantitative-experimental end. In other cases, researchers may need to balance their concern for standards at both ends of the continuum to varying degrees. For example, one team of researchers might need to demonstrate the dependability, confirmability, credibility, and transferability of their study, while another might need to stress the reliability, replicability, validity, and generalizability their study. A third team might need to combine all eight standards, while a fourth team might choose to stress primarily reliability, replicability, validity, and generalizability, but also feel that confirmability and transferability should be addressed briefly. And so forth.

Kimmel (1988) discussed some of the sorts of ethical problems that arise in

social sciences research:

1. The complexity of a single research problem can give rise to multiple questions of proper behavior.

2. Sensitivity to ethical issues is necessary but not sufficient for solving them.

3. Ethical problems are the results of conflicting values.

4. Ethical problems can relate to both the subject matter of the research and the conduct of the research.

5. An adequate understanding of an ethical problem sometimes requires a broad perspective based on the consequences of research.

6. Ethical problems involve both personal and professional elements.

7. Ethical problems can pertain to science (as a body of knowledge) and to research (conducted in such a way as to protect the rights of society and research participants).

8. Judgments about proper conduct lie on a continuum ranging from the clearly unethical to the clearly ethical.

9. An ethical problem can be encountered as a result of a decision to conduct a particular study or a decision not to conduct the study.

Instead of dwelling on the potential problems that unethical behavior can cause in research, I would prefer to simply delineate some of the steps that can be taken to avoid ethical pitfalls in applied linguistics research anywhere along the qual–quant continuum. Some of the most important ethical and professional responsibilities fall into three categories (adapted from Brown, 1997): participant issues, analysis responsibilities, and concerns for the audience of a study:

With regard to participant issues, it is important to:

1. Avoid abusing the participants in a study in any way, including at least abuses of their persons, time, or effort; it may also be important to obtain the participants’ informed consent in writing.

2. Avoid abusing any colleagues by collecting data from their students without their permission or by using too much of their precious class time.

3. Reward the cooperation and efforts of all participants and colleagues, at very least by giving them feedback or information on what happened in the study.

4. Guard against consciously or subconsciously modifying data or interpretations so they support personal views and prejudices.

5. Select the appropriate research method and technique(s) and best possible interaction of research characteristics for the purposes of the particular research project involved.

6. Select the most appropriate interaction of standards possible for the purposes of the particular research project involved.

As for concerns for the audience of a study, it is essential to:

7. Explain the research clearly so it can be understood by the readers.

8. Organize the report using traditional sections, headings, and other conventions (e.g., see American Psychological Association, 1994; or Brown, 2001, ch. 6) so readers can easily follow the study.

9. Interpret results very carefully while guarding against any temptation to

over-interpret, or generalize beyond what the data and results can support.

Since ethics is an area where all research methods and techniques come together and tend to agree, I will end here. However, I would like to mention one further set of considerations that is seldom listed in discussions of research ethics. In my view, all researchers in applied linguistics have two overriding ethical responsibilities: (1) to continue reading, learning, and growing as researchers in order to better serve the field, and (2) to design research that is effective and fits well into the particular institutional contexts involved by selecting those characteristics and standards along the qual–quant continuum that will best interact to provide systematic and principled answers to the many important questions that remain to be answered in applied linguistics.

Baca: Six Propositions in Search of a Methodology

Davies, Alan & Catherine Elder (2004). The Handbook of Applied Linguistics. USA: Blackwell Publishing Ltd