Promising feedback on the results

Christopher Ryan (personal communication) has always maintained that survey researchers can do great damage if they pursue what he called a 'slash and burn' strategy. By this he meant that surveyors typically exploit their participants without offering anything in return - as soon as the data have been gathered, they disappear. On the other hand, if someone puts reasonable effort into answering the questions, this involvement will create a natural curiosity about the project and its outcome. It...

Strategies for getting around anonymity

We saw in Section 2.1.3 that - from the researcher's point of view -respondent anonymity is often undesirable in survey research because without proper identification we cannot match survey data with other sources of information obtained about the same participants (e.g., course marks or other questionnaires). The other side of the coin, however, is that with certain sensitive questions anonymity may be desirable from the respondents' point of view because they may feel safer this way to...

Content Analysis Of Openended Questions

Although it was argued in Sections 1.3 and 2.5 that wide-open, essaylike questions do not work well in questionnaires and therefore should be avoided, questions that are slightly 'less open' can have some merits and are well worth experimenting with as long as this does not exist at the expense of the closed questions (in terms of response time or willingness). Because open-ended questions do not have precoded response options, their processing is less straightforward than that of closed items....

What do questionnaires measure

Broadly speaking, questionnaires can yield three types of data about the respondent factual, behavioral, and attitudinal. 1. Factual questions (also called 'classification' questions or 'subject descriptors') are used to find out about who the respondents are. They typically cover demographic characteristics (e.g., age, gender, and race), residential location, marital and socioeconomic status, level of education, religion, occupation, as well as any other background information that may be...

Main Types Of Questionnaire Administration

In social research the most common form of administering questionnaires is by mail. Educational research is different in this respect because administration by hand is just as significant (if not more) as postal surveys. Within non-postal surveys, we can distinguish two distinct subtypes, one-to-one administration and group administration. Because the administration method has a significant bearing on the format and to some extent also on the content of the questionnaire, we need to examine...

Introduction

One of the most common methods of data collection in second language (L2) research is to use questionnaires of various kinds. The popularity of questionnaires is due to the fact that they are easy to construct, extremely versatile, and uniquely capable of gathering a large amount of information quickly in a form that is readily process-able. Indeed, the frequency of use of self-completed questionnaires as a research tool in the L2 field is surpassed only by that of language proficiency tests....

Questionnaires in Second Language Research

Asking questions is one of the most natural ways of gathering information and, indeed, as soon as babies have mastered the basics of their mother tongue they launch into a continuous flow of questions, and keep going throughout the rest of their lives. Some people such as reporters actually make a living of this activity and survey polling organizations can base highly successful businesses on it. Because the essence of scientific research is trying to find answers to questions in a systematic...

Using multiitem scales

Multi-item scales are the key components to scientific questionnaire design, yet this concept is surprisingly little known in the L2 profession. The core of the issue is that when it comes to assessing attitudes, beliefs, opinions, interests, values, aspirations, expectations, and other personal variables, the actual wording of the questions assumes an unexpected importance minor differences in how the question is formulated and framed can produce radically different levels of agreement or...

Processing Questionnaire Data

Having designed a questionnaire and administered it to an appropriate sample is half the battle. Now comes the final phase of our research, the processing of the data. The starting point of this phase is the very salient presence of stacks of completed questionnaires taking up what little empty space there is in our office. Accordingly, our initial priority is to get rid of these stacks and transform the information that is hidden in these piles of questionnaires into a more useful form that we...

Sensitive topics

'Sensitive' topics are not confined to explicitly illegal or embarrassing subjects but also include basic demographic items such as age or marital status. Indeed, various facts of life can carry such a prominent social and emotional loading that questions targeting them often fall prey to the respondents' 'social desirability' bias (cf. Section 1.2.2). Depending on our core values, we are likely to overreport on what we conceive as a positive aspect and underreport on a negative one....

Disadvantages

Although the previous description of the virtues of questionnaires might suggest that they are perfect research instruments, this is not quite so. Questionnaires have some serious limitations, and some of these have led certain researchers to claim that questionnaire data are not reliable or valid. I do not agree with this claim in general, but there is no doubt that it is very easy to produce unreliable and invalid data by means of ill-constructed questionnaires. In fact, as Gillham (2000, p....

Simplicity and superficiality of answers

Because respondents are left to their own devices when filling in self-completed questionnaires, the questions need to be sufficiently simple and straightforward to be understood by everybody. Thus, this method is unsuitable for probing deeply into an issue (Moser & Kalton, 1971) and it results in rather superficial data. The necessary simplicity of the questions is further augmented by the fact that the amount of time respondents are usually willing to spend working on a questionnaire is...

On length and crowdedness

Perhaps the most common mistake of the beginner in questionnaire construction is to crowd questions together in the hope of making the questionnaire look short. While length is important, the respondent's perception of the difficulty of the task is even more important on self-administered questionnaires. A less crowded questionnaire with substantial white space looks easier and generally results in higher cooperation and fewer errors. (Sudman & Bradburn, 1983, p. 244) Orderly layout. Even if...

Questionnaires In Quantitative And Qualitative Research

The typical questionnaire is a highly structured data collection instrument, with most items either asking about very specific pieces of information (e.g., one's address or food preference) or giving various response options for the respondent to choose from, for example by ticking a box. This makes questionnaire data particularly suited for quantitative, statistical analysis. After all, the essential characteristic of quantitative research is that it employs categories, viewpoints, and models...

Complementing Questionnaire Data With Other Information

Having discussed how to construct and administer questionnaires, and then how to analyze and report the responses we have obtained, the final section of this book addresses ways of proceeding toward a Sample Table 1. Descriptive statistics of the content of this book Sample Table 1. Descriptive statistics of the content of this book fuller understanding of the content area targeted by our survey. As discussed in Chapter 1, although questionnaires offer a versatile and highly effective means of...

The coding frame and the codebook

Having marked each questionnaire with an identification number, we are ready to embark on the coding of the items. Except for extensive texts obtained by open-ended questions (which require special content analysis - cf. Section 4.4), the coding process for each item involves converting the answer into a numerical score. Because numbers are meaningless themselves and are also easy to mix up, a major element of the coding phase is to compile (a) a coding frame that specifies the meaning of the...

Conclusion and Checklist

The previous four chapters have provided a summary of questionnaire theory. Hopefully, they have also made a strong case for basing questionnaire design and processing on scientific principles rather than merely on the researcher's common sense. As emphasized in the Introduction, this book has been intended to serve practical purposes and therefore in this concluding section I will draw up a checklist of what I consider the most important points and recommendations for every phase of the...

Avoid ambiguous or loaded words and sentences

It goes without saying that any elements that might make the language of the items unclear or ambiguous need to be avoided. The most notorious of such elements are Nonspecific adjectives or adverbs e.g., good, easy, many, sometimes, often . Items containing universals such as 'all,' 'none,' 'never.' Modifying words such as 'only,' 'just,' 'merely' - these should be used with moderation. Words having more than one meaning. Loaded words e.g., 'democratic,' 'modern,' 'natural,' 'free,' etc. ,...

Inferential statistics

Descriptive statistics are useful, for example, to describe the achievement of a particular class of learners. What happens, however, if we notice that, say, the L2 learning achievement of boys and girls shows a remarkable difference in our sample, with girls outperforming boys which is often the case Can we draw the inference that girls are better language learners No. Based on descriptive statistics all we can say is that in this class girls did better than boys. In order to venture any...

Group administration

In L2 research, group administration is the most common method of having questionnaires completed. One reason for this is that the typical targets of the surveys are language learners studying within institutional contexts, and it is often possible to arrange to administer the instrument to them while they are assembled together, for example, as part of a lesson or slotted between certain other organized activities. The other reason for the popularity of this administration format is that it...

Initial piloting of the item pool

The first time in the questionnaire construction process that some external feedback is indispensable is when we have prepared an initial item pool cf. Section 2.6.1 , that is, a large list of possible items, and we are ready to reduce the number of questions to the intended final number. The initial piloting of the item pool usually consists of the following steps Select three or four people who are motivated to spend some time to help you and whose opinion you value. Some of them should not...

Onetoone administration

One-to-one administration refers to a situation when someone delivers the questionnaire by hand to the designated person and arranges the completed form to be picked up later e.g., handing out questionnaires to colleagues at work . This is a much more personal form of administration than mail surveys and therefore the chances for the questionnaires to be returned are significantly better. The personal contact also allows the questionnaire administrator to create rapport with the respondent, to...

Rank order items

It is a common human mental activity to rank order people, objects, or even abstract concepts according to some criterion, and rank order items in questionnaires capitalize on our familiarity with this process. As the name suggests, these items contain some sort of a list and respondents are asked to order the items by assigning a number to them according to their preferences. Wilson and McClean 1994 warn us that it may be very demanding to arrange items in order of importance whenever there...

Sensitive topics and anonymity

It was mentioned in Section 1.2.2 that respondents are sometimes reluctant to give honest answers to sensitive questions. Questionnaire items differ greatly in terms of how threatening imposing sensitive embarrassing they feel. It requires little justification that we need to approach the issue of constructing and administering the questionnaire in a very different way if it concerns, for example, the evalua tion of the L2 teacher or the school rather than one's interest in travelling abroad....

Clear and orderly structure

The most important aspect of sequencing questions is to ensure that the respondents' overall impression is that the structure is well-organized and orderly. If the ordering of questions is unpredictable or seemingly haphazard, it will frustrate respondents and make the study appear ill-considered and amateurish Newell, 1993 . Neither the content nor the style of the questionnaire should jump around Aiken, 1997 - the items should seem as a series of logically organized sequences. To achieve...

Convenience or opportunity sampling

The most common sample type in L2 research is a convenience or opportunity sample, where an important criterion of sample selection is the convenience for the researcher Members of the target population will be selected for the purpose of the study if they meet certain practical criteria, such as geographical proximity, availability at a certain time, or easy accessibility. If we decide, for example, to study a class group because we have good contacts with the particular school, that would be...

The coding frame

The coding frame is a classification scheme that offers a numerical score for every possible answer to an item see Sample 4.2 on page 100 . The minimum number of categories is two, as with yes no questions or gender data 'Yes' and 'male' are usually coded '1,' whereas 'No' and 'female' are coded '2.' For some open-ended questions e.g., What foreign languages have you learned in the past the coding frame can have many more categories - in fact, as many as the number of different answers in all...

Factual or personal or classification questions at the end

As Oppenheim 1992 concludes, novice researchers typically start to design a questionnaire by putting a rather forbidding set of questions at the top of a blank sheet of paper, asking for name, address, marital status, number of children, religion, and so on. These personal classification questions tend to be very off-putting Having been through the various introductory phases, respondents are now ready to look at some interesting questions dealing with the topic of the study. Instead, they are...

Constructing the Questionnaire

Section 1.2.2 contained a long list of potential problems with self-completed questionnaires. My goal was not to dissuade people from using such instruments but rather to raise awareness of these possible shortcomings. It is true that respondents are often unmotivated, slapdash, hasty, and insincere, yet it is also an established fact that careful and creative questionnaire construction can result in an instrument that motivates people to give relatively truthful and thoughtful answers, which...

Layout

Sanchez 1992 points out that the design of the questionnaire layout is frequently overlooked as an important aspect of the development of the instrument. This is a mistake Because in surveys employing self-completed questionnaires the main interface between the researcher and the respondent is the hard copy of the questionnaire the format and graphic layout carry a special significance and have an important impact on the responses. Over the past 15 years I have increasingly come to the belief...

References

Rating scales and checklists Evaluating behavior, personality, and attitudes. New York John Wiley. Aiken, L. 1997 . Questionnaires and inventories Surveying opinions and assessing personality. New York John Wiley. Anderson, L. W. 1985 . Attitudes and their measurement. In T. Hus n amp T. N. Postlethwaite Eds. , The international encyclopedia of education Vol. 1, pp. 352-58 . Oxford Pergamon. Bardovi-Harlig, K. 1999 . Researching method. In L. F. Bouton Ed. , Pragmatics and...

General Features

Between the initial idea of preparing a questionnaire for the purpose of our research and actually getting down to writing the first draft, a number of important decisions need to be taken regarding the general features of the would-be instrument. First of all, we need to specify the maximum length of time that the completion of the questionnaire could take then we need to consider general format characteristics', and finally we need to think about the issue of anonymity, particularly if we are...

The behavior of the survey administrator

After all the preliminary considerations, we have finally arrived at the actual day of the survey. The survey administrator is facing the participants obviously, this section does not apply to postal surveys and is ready to launch into his her pep talk. However, in line with the saying, 'Actions speak louder than words,' we need to be aware that our behavior is also conveying important messages to the respondents. The administrators of the questionnaire are, in many ways, identified with the...

Piloting The Questionnaire And Conducting Item Analysis

Because in questionnaires so much depends on the actual wording of the items even minor differences can change the response pattern an integral part of questionnaire construction is 'field testing,' that is, piloting the questionnaire at various stages of its development on a sample of people who are similar to the target sample the instrument has been designed for. These trial runs allow the researcher to collect feedback about how the instrument works and whether it performs the job it has...

Social desirability or prestige bias

The final big problem with regard to questionnaires is that people do not always provide true answers about themselves that is, the results represent what the respondents report to feel or believe, rather than what they actually feel or believe. There are several possible reasons for this, and the most salient one is what is usually termed the social desirability or prestige bias. Questionnaire items are often 'transparent,' that is, respondents can have a fairly good guess about what the...

Openended questions at the end

As discussed in Section 2.5, if we include real open-ended questions that require substantial and creative writing, it is preferable to place them near the end rather than at the beginning of the questionnaire. In this way, their potential negative consequences e.g., the required work can put some people off others might get bogged down and spend most of the available time and mental energy agonizing over what they should write will not affect the previous items. In addition, some people find...

Variations on Likert scales

Likert scales use response options representing the degree of agreement. This standard set of responses i.e., strongly agree - gt strongly disagree can be easily replaced by other descriptive terms that are relevant to the target. For example, Oxford's 1990 Strategy Inventory in Language Learning uses categories ranging from 'Never or almost never true of me' to 'Always or almost always true of me.' Or, in D rnyei and Cl ment's 2001 Language Orientation Question naire a five-point scale ranging...

Sample Instructions for semantic differential scales

The following section of the questionnaire aims at finding out about your ideas and impressions about SOMETHING. In answering the questions we would like to ask you to rate these concepts on a number of scales. These all have pairs of opposites at each end, and between these there are 7 dashes. You are to place a check mark on one of the seven positions, indicating how you feel about the particular concept in view of the two poles. For example, if the scales refer to listening comprehension...

Internal consistency reliability

In order to meet internal consistency reliability requirements, a questionnaire must satisfy two conditions Instead of single items, multi-item scales cf. Section 2.3.2 are to be used wherever it is possible. Multi-item scales are only effective if the items work together in a homogeneous manner, that is, if they measure the same target area. In psychometric terms this means that each item on a scale should correlate with the other items and with the total scale score, which has been referred...

Semantic differential scales

Instead of Likert scales we can also use semantic differential scales for certain measurement purposes. These are very useful in that by using them we can avoid writing statements which is not always easy instead, respondents are asked to indicate their answers by marking a continuum with a tick or an 'X' between two bipolar adjectives on the extremes. For example These scales are based on the recognition that most adjectives have logical opposites and where an opposing adjective is not...

Acquiescence bias

Another common threat inherent to self-completed questionnaires is acquiescence, which refers to the tendency for people to agree with sentences when they are unsure or ambivalent. Acquiescent people include yeasayers, who are ready to go along with anything that sounds good Robinson, Shaver, amp Wrightsman, 1991, p. 8 , and the term also covers those who are reluctant to look at the negative side of any issue and are unwilling to provide strong negative responses.

Undesirable social behavior

With regard to responses that might be felt will meet with disapproval, several strategies have been suggested in the literature. Wilson and McClean 1994 recommend that they can be diffused by the use of categories, or brands, for respondents to tick. In their seminal book on questionnaire design, Sudman and Bradburn 1983 devote a great deal of space to discussing sensitive items. Their practical suggestions to mitigate the undesirable nature of certain behaviors include Wording the question in...

Statistical procedures to analyze data

The standard method of analyzing quantitative questionnaire data is by means of submitting them to various statistical procedures. These involve a range of different techniques, from calculating item score means on a pocket calculator to running complex statistical analyses. As mentioned earlier, it is beyond the scope of this book to provide a detailed analysis of the available procedures. Instead, I would like to emphasize one crucial aspect of statistical data analysis that is so often...

Sentence completion items

A simple question is often less effective in eliciting a meaningful answer than an unfinished sentence beginning that the respondent needs to complete. I have successfully used this technique on various feedback forms in particular. A good completion item should be worded so that it directs the respondent's attention to a well-defined issue area. Sometimes respondents are asked not to 'agonize' over the answers but jot down the first thing that comes to mind. For example One thing I liked about...

Drawing up an item pool

It is generally recommended by survey specialists that when we get down to writing the actual items, we should start doing so without restricting ourselves to any number limitations. Let our imagination go free and create as many potential items as we can think of - this collection is referred to as the item pool. At this stage, successful item designers rely heavily on their own verbal creativity, but they also draw on two additional sources 1. Qualitative, exploratory data gathered from...

Likert scales

The most commonly used scaling technique is the Likert scale, which has been named after its inventor, Rensis Likert. Over the past 70 years Likert's original article came out in 1932 the number of research studies employing this technique has certainly reached a six-digit figure, which is due to the fact that the method is simple, versatile, and reliable. Likert scales consist of a series of statements all of which are related to a particular target which can be, among others, an individual...

Respectable sponsorship

A further factor that might work positively for survey administrators before they have even opened their mouths is some respectable and impressive institutional sponsorship of the study. If we can claim to represent an organization that is esteemed highly by the respondents, the positive reputation is likely to be projected onto the survey. If our institution is less known among the participants, a short leaflet describing its main features and its strengths might tip the balance in favor of...

Computer Programs For Constructing Questionnaires

Because market research - a booming business area - utilizes questionnaires for various types of surveys, several software companies have developed commercial computer programs to cater to these needs Currently there are over 30 available desktop packages that combine questionnaire design, data collection, and data analysis. However, as Macer 1999 summarizes, few packages rise to the challenge of each stage in the process with the same degree of accomplishment, and development effort often...

Selected List Of Published L Questionnaires

Please note that the use of the term 'questionnaires ' in this book does not include 'tests', 'production questionnaires' e.g., DCTs or classroom observation schemes cf. Section 1.1 . I would like to thank all my friends and colleagues who have helped me to compile this list. I am certain that I have unintentionally omitted several valuable published instruments from the list below. I apologize for this. Attitudes see also 'language learning motivation' Wenden 1991 Attitudes questionnaire for...