Further reading

There is no shortage of books on questionnaires many relevant and useful works have been written on the topic in such diverse disciplines as psychology, measurement theory, statistics, sociology, educational studies, and market research. In the L2 field a very recent volume by J. D. Brown (2001) provides a comprehensive account of survey research (which uses questionnaires as one of the main data gathering instruments), offering a detailed account of how to process questionnaire data either...

Quite so

The problem is that the types of respondents who return questionnaires may be a specific type of 'eager-beaver' or 'gung-ho' respondent. Thus the results of the survey can only be generalized to 'eager-beaver' or 'gung-ho' people in the population rather than to the entire population. The scope of the self-selection problem can be illustrated by the fact that 'impersonal' questionnaires (e.g., mail surveys) typically attract an initial response rate of only around 30 , and over 50 can already...

Followup letters

After you have posted the questionnaires, an anxious period of waiting begins. Based on his experience, Gillham (2000) provides a rule-of-thumb estimate that the response you have received by the end of 10 days will be about half of what you can expect to get back in the long run. In order to receive the other half, you need to send a follow-up letter (about 2Vi to 3 weeks after the original mailing). This second mailing is well worth the effort as it can increase the response rate by as much...

Administration by mail

The unique characteristic of postal administration is that the researcher has no contact with the respondent except for a cover letter he she has written to accompany the questionnaire. In addition, mailed questionnaires are often in competition for the addressee's attention with various sorts of circulars, catalogues, and junk mail also received through mail, and the two factors together largely explain why the return rate of such surveys is often well below 30 . Such a low return rate, of...

Advance notice

Surprising as it may sound, the administration of the questionnaire really does not start when the survey administrator first appears on the scene with a bundle of sheets in his her hand. In most cases several important things about the survey have been determined in the respondent by that time. For example, Sudman and Bradburn (1983) conclude that most refusals to cooperate occur before the interviewers have had a chance to explain fully the purposes of the survey. In a paper entirely devoted...

Strategies To Increase The Quality And Quantity Of Participant Response

The main message of this section can be summarized in three words Administration procedures matter It was emphasized more than once in the previous two chapters that the 'Achilles heel' of questionnaires as measuring instruments is that it is difficult to get respondents to spend enough time and effort completing them. Educational researchers are in a slightly better position in this respect because school children are often willing to work hard on a task simply because it is assigned to them,...

Guidelines for increasing mail survey return rates

How can we increase the willingness of the recipients to take the time and trouble to complete and return the postal survey The strategies most frequently mentioned in the measurement literature are as follows (see also Section 3.3, which offers general - i.e., not restricted to postal surveys in particular - strategies to promote respondent attitudes) Pre-survey letters give advance notice about the purpose and nature of the forthcoming questionnaire and can create a favorable climate for the...

The cover letter

In the absence of a 'live' contact person, the cover letter has the difficult job to 'sell' the survey, that is, to create rapport with the respondents and to convince them about the importance of the survey and of their role in contributing to it. In addition to this public relations function, the cover letter also needs to provide certain specific information and directions. To write a letter that meets all these requirements is not easy, particularly in view of the fact that it needs to be...

A selfgenerated identification coding procedure

The identification procedure just described does not ensure anonymity but only saves the salient act of students' writing their name on the questionnaire. A more complex method of ensuring identifiable anonymity has been piloted by Kearney et al. (1984) with some success. This method involves students' generating for themselves a unique personal code number and including this on every document they complete - hence the possibility for data linkability. Of course, no one except them would know...

Absolutely

Questionnaires do not emerge fully-fledged they have to be created or adapted, fashioned and developed to maturity after many abortive test flights. In fact, every aspect of a survey has to be tried out beforehand to make sure that it works as intended. So when and what shall we pilot While it is useful to have 'ongoing piloting' by continuously discussing every aspect of questionnaire design with a colleague or a friend, there are two points where a more formal trial run is needed (1) Once the...

Administering The Questionnaire

3.1.2 How large should the sample be 73 3.1.3 The problem of respondent self-selection 75 3.2 Main types of questionnaire administration 76 3.2.1 Administration by mail 77 3.2.2 One-to-one administration 81 3.3 Strategies to increase the quality and quantity of participant response 83 3.3.2 Attitudes conveyed by teachers, parents, and other authority figures 85 3.3.3 Respectable sponsorship 85 3.3.4 The behavior of the survey administrator 86 3.3.5 Communicating the purpose and significance of...

Appropriate sampling of the content

Ad hoc questionnaire design involves jotting down a number of relevant questions without any rigorous procedure to ensure that the coverage is comprehensive. The problem with this method, as Davidson (1996, p. 10) highlights, is that You cannot analyze what you do not measure. That is, not even the most sophisticated data analysis techniques will be able to compensate for leaving out some important questions from the data collection by accident. Certain omissions are bound to occur even in...

Attitudes conveyed by teachers parents and other authority figures

Data gathering often takes place in someone's 'home ground.' In school settings, for example, students usually hear about the survey first from their teachers. The important thing to note with respect to this is that participants are rather quick to pick up their superiors' (e.g., teachers' or bosses') attitude toward the survey and only acquiesce if the message they receive is positive. Similarly, parental disposition can also have a major impact on students' willingness to respond. It is...

Avoid items that are likely to be answered the same way by everybody

In rating scales we should avoid statements that are likely to be endorsed by almost everyone or almost no one. In most cases these items are not informative and they are certainly difficult if not impossible to process statistically. Here is a recent example from my own research (D rnyei & Cl ment, 2001) A questionnaire item asked students to rate the international role importance of six countries, including the United States. As can be imagined, most respondents gave the U.S. the top...

Basic ethical principles of data collection

Drawing on Oppenheim's (1992) and Sudman and Bradburn's (1983) discussion of ethical issues in survey research, the following five principles can be compiled Principle 1 No harm should come to the respondents as a result of their participation in the research. This is the primary ethical principle governing data collection and it overrides all other considerations. Principle 2 The respondent's right to privacy should always be respected, and no undue pressure should be brought to bear. That is,...

Closedended Questionnaire Items

Let us start our exploration of the various types of questionnaire items by first examining the most frequent question type closed-ended (or simply 'closed) questions. Although this category subsumes several very different item types, these all share in common the fact that the respondent is provided with ready-made response options to choose from, normally by encircling or ticking one of them or by putting an 'X' in the appropriate slot box. That is, these items do not require the respondents...

Coding Questionnaire Data

Most data analysis software handles data in a numerical rather than in an alphabetic form, and even with programs that allow the storage of information recorded as letters, the procedures that are available for handling such data are limited compared to the vast arsenal of statistical techniques to be used with numerical responses. Therefore, the first step of data processing usually involves converting the respondents' answers to numbers by means of coding procedures. As we will see, these...

Communicating the purpose and significance of the survey

Although actions may speak louder than words, this does not mean that words don't matter. An important element in 'selling' the survey to the participants is communicating to them the purpose of the survey and conveying to them the potential significance of the results. People tend not to mind answering questions if they see the point. We should also be aware of the fact that, as Gillham (2000) warns us, in our information-conscious age there is a general suspicion that much more data are...

Computer Programs For Processing Questionnaire Data

There are numerous statistical software packages that can be used to process quantitative questionnaire data. Personally, I have always used, and been satisfied with, 'SPSS' (Statistical Package for the Social Sciences), which is one of the market leaders in this category. There are also various computer programs to facilitate the qualitative analysis of transcribed texts (e.g., NUD*IST, NVivo). From a survey researcher's point of view, programs that can handle quantitative and qualitative...

Data cleaning

The initial data file will always contain mistakes. Some of these are the result of human error occurring during the data entry phase (e.g., typing the wrong number) and some are mistakes made by the respondent when filling in the questionnaire. Data cleaning involves correcting as many of these errors and inaccuracies as possible before the actual analyses are undertaken. The main checks and techniques are as follows Correcting impossible data. Most items have a specific range, determined by...

Detachment from real life

Researchers often note how ironical it is that months of hard labor can sometimes be summarized in one or two tables. While this may not be a problem in basic research - after all, Einstein's theory of relativity did not exactly take up several volumes either - in more applied studies when we are looking at concrete questions concerning real people, a primarily quantitative summary may lose some of the edge and flavor of the original issue. This is when a few open-ended items in the...

Entering The Data Into A Computer File

With the coding frames and the codebook ready, we need to get down to the rather tedious and time-consuming process of keying in the data. All of us involved in survey research have spent countless number of hours in front of a computer screen typing seemingly endless rows of figures. However boring and mechanical this job may be, it is essential that we maintain concentration because every mistyped number will be a contamination of the dataset. In agreement with Brown (2001), I have found that...

Examining the reliability and validity of the data

Reliability and validity are two key concepts in measurement theory, referring to the psychometric properties of the measurement techniques and the data obtained by them. The reliability of a psychometric instrument refers to the extent to which scores on the instrument are free from errors of measurement. For example, bathroom scales are not reliable if they show different figures depending on how steamy the air in the bathroom is, and neither are proficiency test raters if their evaluation...

Final piloting dress rehearsal

Based on the feedback received from the initial pilot group we can normally put together a near-final version of the questionnaire that 'feels' OK and that does not have any obvious glitches. However, we still do not know how the items will work in actual practice, that is, whether the selected respondents will reply to the items in the manner intended by the questionnaire designers. There is only one way to find out by administering the questionnaire to a group of respondents who are in every...

First things first Identification code and Research Logbook

Before we get down to actual coding, there are two things to be done. One is compulsory, the other highly recommended. The compulsory task is to give each questionnaire a unique identification code. In practice this may involve taking each questionnaire one by one and numbering them sequentially by writing a code number in one of the top corners of the front page. Questionnaires coming from the same group (e.g., same school or class) should be kept together and marked with a special code for...

Fred Davidsons plea for backing up data

Law 2 Do it now. - Anonymous I have noticed that nobody seems to learn to back up data until he or she has a serious disaster with data that are personally quite important. Please prove me wrong and abide by these two laws as early in your data career as possible. The traditional (and still frequently used) method of entering data into a computer file involves creating a rectangular text file (e.g., a word-processing file saved in a text only with line breaks file format) in...

General instruction

As a minimum, the general instruction (or 'opening greeting') should cover the following points (see also Section 3.2.1, for special instructions for mail surveys) What the study is about and why it is important or socially useful. The organization responsible for conducting the study. Emphasizing that there are no right or wrong answers requesting honest answers and trying to elicit integrity in general. For better readability and emphasis, the instructions should be graphically highlighted,...

Grouping And Ordering Items

Once all the items to be included in the questionnaire have been written or collected, we need to decide on their order. Item sequence is a significant factor because the context of a question can have an im pact on its interpretation and the response given to it. Indeed, the meaning of almost any question can be altered by the adjacent questions. However, it is usually acknowledged that research has not as yet generated any specific theoretical rules to order questions, beyond some broad...

Handling missing data

One issue that should definitely be resolved before the analyses is deciding what to do with missing values. They are a nuisance in many ways. First, it is not always clear whether the lack of any useful response is meaningful or not. For example, if Rupert is asked about what foreign languages he speaks and he leaves the question unanswered, would this mean that (a) Rupert only speaks his mother tongue, or (b) he has skipped the question by mistake, or (c) he has intentionally refused to...

How much to generalize

With regard to the issue of generalizing, it is so easy to offer the broad and rather unhelpful guideline Do not over generalize However, research in most cases is all about the need to produce generalizable findings. After all, with the exception of 'action research,' researchers in the L2 field rarely investigate a sample with the sole purpose of wanting to know more only about the particular people under investigation. Instead, what we normally want to do is find out more about the...

How To Write Good Items

Over the past 50 years, survey researchers have accumulated a considerable body of knowledge and experience about what makes a questionnaire item good and what the potential pitfalls are. However, most specialists also emphasize that question design is not a 100 scientific activity because in order to write good items one also needs a certain amount of creativity and lots of common sense. Furthermore, alternative versions of questions must be rigorously piloted because in the absence of hard...

Include both positively and negatively worded items

In order to avoid a response set in which the respondents mark only one side of a rating scale, it is worth including in the questionnaire both positively and negatively worded items. In addition, a balanced mixture might also reduce the harmful effects of the 'acquiescence bias' (cf. Section 1.2.2). The term 'negatively worded item' means that it focuses on negative rather than positive aspects of the target, and we should note that it is all too easy to fall into the trap of trying to express...

Indeed

If respondents are clear about what you are trying to find out and why, they are much more likely to respond appropriately and helpfully, or, indeed, at all. There is a curious convention that if you tell respondents what you are trying to find out, this will 'bias' them. It might simply make them more helpful. If you are mysterious about the purpose of the questionnaire they may be disinclined to answer or misunderstand the purpose, and so bias their answers in that way. Just like the cover...

Little or no opportunity to correct the respondents mistakes

Questionnaire items focus on information which the respondents know best, and therefore the researcher has little opportunity to double-check the validity of the answers. Sometimes respondents deviate from the truth intentionally (see further) but it is also common that -as just mentioned - they simply misunderstand or forget something, or do not remember it correctly. Another fairly common situation is when informants do not know the exact response to a question yet answer it without...

Not indeed

The world is full of well-meaning people who believe that everyone who can write plain English and has a modicum of common sense can produce a good questionnaire. This book is not for them. As already mentioned in the Introduction, my interest in questionnaires is pragmatic and practice-driven. I use them all the time and I would like the measures obtained by them to meet high research standards. Having fallen into many of the existing pitfalls several times, I intend for this book to offer...

Numerical rating scales

Teenagers sometimes play a rating game whereby they evaluate the appearance and 'sexiness' of the various girls boys they see passing by in the street on a scale of 1-10. They would be surprised to hear that what they are doing is applying numerical rating scales. These scales involve 'giving so many marks out of so many,' that is, assigning one of several numbers corresponding to a series of ordered categories describing a feature of the target. The popularity of this scaling technique is due...

Opening questions

Just like with any other piece of writing, the initial section of a questionnaire is particularly important in that it sets the tone. This is partly the reason that instructions (cf. Sections 2.2.2 and 3.3.7) play a significant role, and this is also why the first few 'opening' questions should be carefully selected. In order to create a pleasant first impression, the starter questions need to be interesting, relatively simple yet at the same time focused on some important and salient aspect,...

Potential threat

With regard to items in which an honest answer can pose some real threat to the respondent (e.g., questions about illegal activities, or asking students to evaluate their language teacher), the main task is to convince the respondents that their answers will remain confidential. Obviously, offering complete anonymity in such cases might be helpful, but this may not be feasible in certain complex research projects where we need to match the data with information obtained from other sources (cf....

Questionnaire items

After the instructions comes the central part of the questionnaire, the actual items. They will be discussed in detail in Sections 2.3 - 2.7. Questionnaire items rarely take the form of actual questions that end with a question mark. The item type found in Sample 2.3, for example, is far more common than that in Sample 2.2 (which is a real question). The items need to be veiy clearly separated from the instructions. This is where different typefaces and font styles come in handy. Unless the...

Random sampling

The key component of scientific sampling procedures is random sampling. This involves selecting members of the population to be included in your sample on a completely random basis, a bit like drawing numbers from a hat (e.g., by numbering each member and then asking the computer to generate random numbers). The assumption underlying this procedure is that it minimizes the effects of any extraneous or subjective variables that might affect the outcome of the survey study. Combining random...

Recoding values

It has been recommended earlier (in Section 2.6.2) that in order to avoid a response set where the respondents mark only one side of a rating scale, it is worth including in the questionnaire both positively and negatively worded items this may also reduce the harmful effects of the acquiescence bias. However, if we have such negatively worded items, we must not forget to reverse the scoring for these be fore including them in multi-item scales. This may sound like an obvious and trivial...

Reducing the number of variables in the questionnaire

Once we have completed data cleaning and data manipulation, we are ready to embark on the examination of the obtained data. The first step in analyzing questionnaire data is always to reduce to manageable proportions the number of variables measured by the question naire so that the mass of details does not prevent us from seeing the forest through the trees. The rationale for this is that - in accordance with the arguments in Section 2.3.2 - a well-designed questionnaire contains multiple...

Regrettably

People tend not to take questionnaires seriously their answers may be frankly frivolous. In view of these handicaps, the researcher's task to motivate the respondents to give truthful and thoughtful answers to all the relevant items on the questionnaire might seem daunting if not impossible. The good news, however, is that people in general like to express their opinions and do not mind answering questions as long as they think that the survey is related to a worthy cause and that their opinion...

Sample General instruction

We would like to ask you to help us by answering the following questions concerning foreign language learning. This survey is conducted by the Language Research Group of the University of X to better understand This is not a test so there are no right or wrong answers and you don't even have to write your name on it. We are interested in your personal opinion. Please give your answers sincerely as only this will guarantee the success of the investigation. Thank you very much for your help.

Sampling procedures

Broadly speaking, the sample is the group of people whom researchers actually examine and the population is the group of people whom the survey is about. For example, the population in a study might be EFL learners in Taiwanese secondary schools and the actual sample might involve three Taiwanese secondary classes. That is, the target population of a study consists of all the people to whom the survey's findings are to be applied or generalized. Why don't we include every member of the...

Standardizing the data

When we use pooled results from various subsamples, one way of controlling for the heterogeneous nature of the subgroups is to use standardized scores. The standardization of raw scores involves the conversion of the distribution within a sample in a way that the mean will be 0 and the standard deviation 1. Thus, standard scores express how much each raw value is different from the subgroup mean, and by equalizing the means, scores obtained from different subsamples (e.g., different classes in...

Technical information to accompany survey results

Novice researchers often make the mistake of concentrating only on the presentation of their actual findings in their survey reports. While this approach may appear logical, it fails to take into account the fact that in order to be able to interpret (and believe) the claims made, readers will have to be convinced that the methodology used to produce the particular findings was appropriate. This does not mean that we can only report results if our study did not have any methodological...

The problem of respondent selfselection

To conclude the discussion of the various sampling issues for research purposes in general, we need to highlight a potential pitfall that might put the validity of the survey at risk the problem of participant self-selection. This refers to cases when for various reasons the actual composition of the sample is not only the function of some systematic selection process but also of factors related to the respondents' own willingness to participate. Problems can arise, for example, when...

The style and layout of the questionnaire

As argued earlier, respondents are normally willing to spend time and effort on a questionnaire if they believe that they are contributing to a serious investigation. One factor that plays an important role in convincing them about this is the professional quality of the questionnaire. The tone and content of the printed instructions, the layout and typesetting of the items, and small details such as thanking the participants for their cooperation, can all contribute to the formation of a...

Truefalse items

In some scales the designers only set two response options true versus false (or 'yes' or 'no'), resulting in what is usually referred to as a 'true-false item' While generally it is true that the more options an item contains, the more accurate evaluation it yields, there might be cases when only such a polarized, yes-no decision can be considered reliable. For example, little children are sometimes seen as incapable of providing more elaborate ratings, and some personality test items also...

Use simple and natural language

As a rule, in questionnaire items we should always choose the simplest way to say something. Items need to be kept clear and direct, without any acronyms, abbreviations, colloquialisms, proverbs, jargon, or technical terms. We should try to speak the 'common language' and find synonyms for the polysyllabic and Latinate constructions that come easily to the tongue of the college educated (Converse & Presser, 1986, p. 10). Oppenheim (1992) argues that the most important rule in writing rating...

Tests are not questionnaires

Written, self-completed (or self-report) questionnaires are very similar to written tests, yet there is a basic difference between them. A 'test' takes a sample of the respondent's behavior knowledge and, on the basis of this sample, inferences are made about the degree of the development of the individual's more general underlying competence abilities skills (e.g., overall L2 proficiency). Thus, a test measures how well someone can do something. In contrast, questionnaires do not have good or...

Summarizing And Reporting Questionnaire Data

Survey data can be used for a great variety of purposes and each of these might require somewhat different types of summaries and reports of the findings. It is obvious, for instance, that a PhD dissertation will have to meet criteria that are very different from the presentation requirements of a summary of student achievement at a school staff meeting. Rather than attempting to provide templates for all the diverse applications (such templates are readily available in various writers'...

Openended Questions

Open-ended questions include items where the actual question is not followed by response options for the respondent to choose from but rather by some blank space (e.g., dotted lines) for the respondent to fill. As we have seen in the previous chapter (in Section 1.3), questionnaires are not particularly suited for truly qualitative, exploratory research. Accordingly, they tend to have few open-ended questions and even the ones included are relatively short, with their 'openness' somehow...

Measuring and ensuring internal consistency reliability

Internal consistency reliability is measured by the Cronbach Alpha coefficient (named after its introducer, L. J. Cronbach). This is a figure ranging between zero and +1, and if it proves to be very low, either the particular scale is too short or the items have very little in common. Internal consistency estimates for well-developed attitude scales containing as few as 10 items ought to approach 0.80. Because of the complexity of the second language acquisition process, L2 researchers...

Respondent literacy problems

Questionnaire research makes the inherent assumption that the respondents can read and write well. Even in the industrialized world this is not necessarily the case with regard to the whole population Statistics of about 5 -7 are regularly quoted when estimating the proportion of people who have difficulty reading, and the number of those who are uncomfortable with writing is even bigger. It is therefore understandable that for respondents with literacy problems, filling in a questionnaire can...

Ellard and Rogers p Ten Commandments of Question Writing

Thou shalt not create double-barreled items. II. Thou shalt not use 'no' and 'not' or words beginning with 'un.' III. Thou shalt match the vocabulary used in items to the vocabulary of those who will respond to them. IV. Thou shalt not use complex grammatical forms. V. Thou shalt have 40 to 60 true- or agree-keyed items. VI. Thou shalt not use redundant or irrelevant items. VII. Thou shalt not permit any loaded questions to appear in your questionnaire. VIII. Thou shalt not mix response...

Avoid doublebarreled questions

Double-barreled questions are those that ask two (or more) questions in one while expecting a single answer. For example, the question How are your parents asks about one's mother and father, and cannot be answered simply if one of them is well and the other unwell. Indeed, questions dealing with pluralisms (children, students) often yield double-barreled questions, but compound questions also often fall into this category (e.g., Do you always write your homework and do it thoroughly ). With...

Unreliable and unmotivated respondents

Most people are not very thorough in a research sense, and this is all the more true about dealing with questionnaires - an activity which typically they do not enjoy or benefit from in any way. Thus, the re- suits may vary greatly from one individual to another, depending on the time and care they choose or are able to give (Hopkins, Stanley, & Hopkins, 1990). Respondents are also prone to leave out some questions, either by mistake or because they did not like them, and Low (1999) presents...

Promising feedback on the results

Christopher Ryan (personal communication) has always maintained that survey researchers can do great damage if they pursue what he called a 'slash and burn' strategy. By this he meant that surveyors typically exploit their participants without offering anything in return - as soon as the data have been gathered, they disappear. On the other hand, if someone puts reasonable effort into answering the questions, this involvement will create a natural curiosity about the project and its outcome. It...

Strategies for getting around anonymity

We saw in Section 2.1.3 that - from the researcher's point of view -respondent anonymity is often undesirable in survey research because without proper identification we cannot match survey data with other sources of information obtained about the same participants (e.g., course marks or other questionnaires). The other side of the coin, however, is that with certain sensitive questions anonymity may be desirable from the respondents' point of view because they may feel safer this way to...

Content Analysis Of Openended Questions

Although it was argued in Sections 1.3 and 2.5 that wide-open, essaylike questions do not work well in questionnaires and therefore should be avoided, questions that are slightly 'less open' can have some merits and are well worth experimenting with as long as this does not exist at the expense of the closed questions (in terms of response time or willingness). Because open-ended questions do not have precoded response options, their processing is less straightforward than that of closed items....

What do questionnaires measure

Broadly speaking, questionnaires can yield three types of data about the respondent factual, behavioral, and attitudinal. 1. Factual questions (also called 'classification' questions or 'subject descriptors') are used to find out about who the respondents are. They typically cover demographic characteristics (e.g., age, gender, and race), residential location, marital and socioeconomic status, level of education, religion, occupation, as well as any other background information that may be...

Main Types Of Questionnaire Administration

In social research the most common form of administering questionnaires is by mail. Educational research is different in this respect because administration by hand is just as significant (if not more) as postal surveys. Within non-postal surveys, we can distinguish two distinct subtypes, one-to-one administration and group administration. Because the administration method has a significant bearing on the format and to some extent also on the content of the questionnaire, we need to examine...

Introduction

One of the most common methods of data collection in second language (L2) research is to use questionnaires of various kinds. The popularity of questionnaires is due to the fact that they are easy to construct, extremely versatile, and uniquely capable of gathering a large amount of information quickly in a form that is readily process-able. Indeed, the frequency of use of self-completed questionnaires as a research tool in the L2 field is surpassed only by that of language proficiency tests....

Questionnaires in Second Language Research

Asking questions is one of the most natural ways of gathering information and, indeed, as soon as babies have mastered the basics of their mother tongue they launch into a continuous flow of questions, and keep going throughout the rest of their lives. Some people such as reporters actually make a living of this activity and survey polling organizations can base highly successful businesses on it. Because the essence of scientific research is trying to find answers to questions in a systematic...

Using multiitem scales

Multi-item scales are the key components to scientific questionnaire design, yet this concept is surprisingly little known in the L2 profession. The core of the issue is that when it comes to assessing attitudes, beliefs, opinions, interests, values, aspirations, expectations, and other personal variables, the actual wording of the questions assumes an unexpected importance minor differences in how the question is formulated and framed can produce radically different levels of agreement or...

Processing Questionnaire Data

Having designed a questionnaire and administered it to an appropriate sample is half the battle. Now comes the final phase of our research, the processing of the data. The starting point of this phase is the very salient presence of stacks of completed questionnaires taking up what little empty space there is in our office. Accordingly, our initial priority is to get rid of these stacks and transform the information that is hidden in these piles of questionnaires into a more useful form that we...

Sensitive topics

'Sensitive' topics are not confined to explicitly illegal or embarrassing subjects but also include basic demographic items such as age or marital status. Indeed, various facts of life can carry such a prominent social and emotional loading that questions targeting them often fall prey to the respondents' 'social desirability' bias (cf. Section 1.2.2). Depending on our core values, we are likely to overreport on what we conceive as a positive aspect and underreport on a negative one....

Disadvantages

Although the previous description of the virtues of questionnaires might suggest that they are perfect research instruments, this is not quite so. Questionnaires have some serious limitations, and some of these have led certain researchers to claim that questionnaire data are not reliable or valid. I do not agree with this claim in general, but there is no doubt that it is very easy to produce unreliable and invalid data by means of ill-constructed questionnaires. In fact, as Gillham (2000, p....

Simplicity and superficiality of answers

Because respondents are left to their own devices when filling in self-completed questionnaires, the questions need to be sufficiently simple and straightforward to be understood by everybody. Thus, this method is unsuitable for probing deeply into an issue (Moser & Kalton, 1971) and it results in rather superficial data. The necessary simplicity of the questions is further augmented by the fact that the amount of time respondents are usually willing to spend working on a questionnaire is...

On length and crowdedness

Perhaps the most common mistake of the beginner in questionnaire construction is to crowd questions together in the hope of making the questionnaire look short. While length is important, the respondent's perception of the difficulty of the task is even more important on self-administered questionnaires. A less crowded questionnaire with substantial white space looks easier and generally results in higher cooperation and fewer errors. (Sudman & Bradburn, 1983, p. 244) Orderly layout. Even if...

Questionnaires In Quantitative And Qualitative Research

The typical questionnaire is a highly structured data collection instrument, with most items either asking about very specific pieces of information (e.g., one's address or food preference) or giving various response options for the respondent to choose from, for example by ticking a box. This makes questionnaire data particularly suited for quantitative, statistical analysis. After all, the essential characteristic of quantitative research is that it employs categories, viewpoints, and models...

Complementing Questionnaire Data With Other Information

Having discussed how to construct and administer questionnaires, and then how to analyze and report the responses we have obtained, the final section of this book addresses ways of proceeding toward a Sample Table 1. Descriptive statistics of the content of this book Sample Table 1. Descriptive statistics of the content of this book fuller understanding of the content area targeted by our survey. As discussed in Chapter 1, although questionnaires offer a versatile and highly effective means of...

The coding frame and the codebook

Having marked each questionnaire with an identification number, we are ready to embark on the coding of the items. Except for extensive texts obtained by open-ended questions (which require special content analysis - cf. Section 4.4), the coding process for each item involves converting the answer into a numerical score. Because numbers are meaningless themselves and are also easy to mix up, a major element of the coding phase is to compile (a) a coding frame that specifies the meaning of the...

Conclusion and Checklist

The previous four chapters have provided a summary of questionnaire theory. Hopefully, they have also made a strong case for basing questionnaire design and processing on scientific principles rather than merely on the researcher's common sense. As emphasized in the Introduction, this book has been intended to serve practical purposes and therefore in this concluding section I will draw up a checklist of what I consider the most important points and recommendations for every phase of the...

Avoid ambiguous or loaded words and sentences

It goes without saying that any elements that might make the language of the items unclear or ambiguous need to be avoided. The most notorious of such elements are Nonspecific adjectives or adverbs e.g., good, easy, many, sometimes, often . Items containing universals such as 'all,' 'none,' 'never.' Modifying words such as 'only,' 'just,' 'merely' - these should be used with moderation. Words having more than one meaning. Loaded words e.g., 'democratic,' 'modern,' 'natural,' 'free,' etc. ,...

Inferential statistics

Descriptive statistics are useful, for example, to describe the achievement of a particular class of learners. What happens, however, if we notice that, say, the L2 learning achievement of boys and girls shows a remarkable difference in our sample, with girls outperforming boys which is often the case Can we draw the inference that girls are better language learners No. Based on descriptive statistics all we can say is that in this class girls did better than boys. In order to venture any...

Group administration

In L2 research, group administration is the most common method of having questionnaires completed. One reason for this is that the typical targets of the surveys are language learners studying within institutional contexts, and it is often possible to arrange to administer the instrument to them while they are assembled together, for example, as part of a lesson or slotted between certain other organized activities. The other reason for the popularity of this administration format is that it...

Initial piloting of the item pool

The first time in the questionnaire construction process that some external feedback is indispensable is when we have prepared an initial item pool cf. Section 2.6.1 , that is, a large list of possible items, and we are ready to reduce the number of questions to the intended final number. The initial piloting of the item pool usually consists of the following steps Select three or four people who are motivated to spend some time to help you and whose opinion you value. Some of them should not...

Onetoone administration

One-to-one administration refers to a situation when someone delivers the questionnaire by hand to the designated person and arranges the completed form to be picked up later e.g., handing out questionnaires to colleagues at work . This is a much more personal form of administration than mail surveys and therefore the chances for the questionnaires to be returned are significantly better. The personal contact also allows the questionnaire administrator to create rapport with the respondent, to...

Rank order items

It is a common human mental activity to rank order people, objects, or even abstract concepts according to some criterion, and rank order items in questionnaires capitalize on our familiarity with this process. As the name suggests, these items contain some sort of a list and respondents are asked to order the items by assigning a number to them according to their preferences. Wilson and McClean 1994 warn us that it may be very demanding to arrange items in order of importance whenever there...

Sensitive topics and anonymity

It was mentioned in Section 1.2.2 that respondents are sometimes reluctant to give honest answers to sensitive questions. Questionnaire items differ greatly in terms of how threatening imposing sensitive embarrassing they feel. It requires little justification that we need to approach the issue of constructing and administering the questionnaire in a very different way if it concerns, for example, the evalua tion of the L2 teacher or the school rather than one's interest in travelling abroad....

Clear and orderly structure

The most important aspect of sequencing questions is to ensure that the respondents' overall impression is that the structure is well-organized and orderly. If the ordering of questions is unpredictable or seemingly haphazard, it will frustrate respondents and make the study appear ill-considered and amateurish Newell, 1993 . Neither the content nor the style of the questionnaire should jump around Aiken, 1997 - the items should seem as a series of logically organized sequences. To achieve...

Convenience or opportunity sampling

The most common sample type in L2 research is a convenience or opportunity sample, where an important criterion of sample selection is the convenience for the researcher Members of the target population will be selected for the purpose of the study if they meet certain practical criteria, such as geographical proximity, availability at a certain time, or easy accessibility. If we decide, for example, to study a class group because we have good contacts with the particular school, that would be...

The coding frame

The coding frame is a classification scheme that offers a numerical score for every possible answer to an item see Sample 4.2 on page 100 . The minimum number of categories is two, as with yes no questions or gender data 'Yes' and 'male' are usually coded '1,' whereas 'No' and 'female' are coded '2.' For some open-ended questions e.g., What foreign languages have you learned in the past the coding frame can have many more categories - in fact, as many as the number of different answers in all...

Factual or personal or classification questions at the end

As Oppenheim 1992 concludes, novice researchers typically start to design a questionnaire by putting a rather forbidding set of questions at the top of a blank sheet of paper, asking for name, address, marital status, number of children, religion, and so on. These personal classification questions tend to be very off-putting Having been through the various introductory phases, respondents are now ready to look at some interesting questions dealing with the topic of the study. Instead, they are...

Constructing the Questionnaire

Section 1.2.2 contained a long list of potential problems with self-completed questionnaires. My goal was not to dissuade people from using such instruments but rather to raise awareness of these possible shortcomings. It is true that respondents are often unmotivated, slapdash, hasty, and insincere, yet it is also an established fact that careful and creative questionnaire construction can result in an instrument that motivates people to give relatively truthful and thoughtful answers, which...

Layout

Sanchez 1992 points out that the design of the questionnaire layout is frequently overlooked as an important aspect of the development of the instrument. This is a mistake Because in surveys employing self-completed questionnaires the main interface between the researcher and the respondent is the hard copy of the questionnaire the format and graphic layout carry a special significance and have an important impact on the responses. Over the past 15 years I have increasingly come to the belief...

References

Rating scales and checklists Evaluating behavior, personality, and attitudes. New York John Wiley. Aiken, L. 1997 . Questionnaires and inventories Surveying opinions and assessing personality. New York John Wiley. Anderson, L. W. 1985 . Attitudes and their measurement. In T. Hus n amp T. N. Postlethwaite Eds. , The international encyclopedia of education Vol. 1, pp. 352-58 . Oxford Pergamon. Bardovi-Harlig, K. 1999 . Researching method. In L. F. Bouton Ed. , Pragmatics and...

The behavior of the survey administrator

After all the preliminary considerations, we have finally arrived at the actual day of the survey. The survey administrator is facing the participants obviously, this section does not apply to postal surveys and is ready to launch into his her pep talk. However, in line with the saying, 'Actions speak louder than words,' we need to be aware that our behavior is also conveying important messages to the respondents. The administrators of the questionnaire are, in many ways, identified with the...

Piloting The Questionnaire And Conducting Item Analysis

Because in questionnaires so much depends on the actual wording of the items even minor differences can change the response pattern an integral part of questionnaire construction is 'field testing,' that is, piloting the questionnaire at various stages of its development on a sample of people who are similar to the target sample the instrument has been designed for. These trial runs allow the researcher to collect feedback about how the instrument works and whether it performs the job it has...

Social desirability or prestige bias

The final big problem with regard to questionnaires is that people do not always provide true answers about themselves that is, the results represent what the respondents report to feel or believe, rather than what they actually feel or believe. There are several possible reasons for this, and the most salient one is what is usually termed the social desirability or prestige bias. Questionnaire items are often 'transparent,' that is, respondents can have a fairly good guess about what the...

Openended questions at the end

As discussed in Section 2.5, if we include real open-ended questions that require substantial and creative writing, it is preferable to place them near the end rather than at the beginning of the questionnaire. In this way, their potential negative consequences e.g., the required work can put some people off others might get bogged down and spend most of the available time and mental energy agonizing over what they should write will not affect the previous items. In addition, some people find...

Variations on Likert scales

Likert scales use response options representing the degree of agreement. This standard set of responses i.e., strongly agree - gt strongly disagree can be easily replaced by other descriptive terms that are relevant to the target. For example, Oxford's 1990 Strategy Inventory in Language Learning uses categories ranging from 'Never or almost never true of me' to 'Always or almost always true of me.' Or, in D rnyei and Cl ment's 2001 Language Orientation Question naire a five-point scale ranging...

Sample Instructions for semantic differential scales

The following section of the questionnaire aims at finding out about your ideas and impressions about SOMETHING. In answering the questions we would like to ask you to rate these concepts on a number of scales. These all have pairs of opposites at each end, and between these there are 7 dashes. You are to place a check mark on one of the seven positions, indicating how you feel about the particular concept in view of the two poles. For example, if the scales refer to listening comprehension...

Internal consistency reliability

In order to meet internal consistency reliability requirements, a questionnaire must satisfy two conditions Instead of single items, multi-item scales cf. Section 2.3.2 are to be used wherever it is possible. Multi-item scales are only effective if the items work together in a homogeneous manner, that is, if they measure the same target area. In psychometric terms this means that each item on a scale should correlate with the other items and with the total scale score, which has been referred...

Semantic differential scales

Instead of Likert scales we can also use semantic differential scales for certain measurement purposes. These are very useful in that by using them we can avoid writing statements which is not always easy instead, respondents are asked to indicate their answers by marking a continuum with a tick or an 'X' between two bipolar adjectives on the extremes. For example These scales are based on the recognition that most adjectives have logical opposites and where an opposing adjective is not...

Acquiescence bias

Another common threat inherent to self-completed questionnaires is acquiescence, which refers to the tendency for people to agree with sentences when they are unsure or ambivalent. Acquiescent people include yeasayers, who are ready to go along with anything that sounds good Robinson, Shaver, amp Wrightsman, 1991, p. 8 , and the term also covers those who are reluctant to look at the negative side of any issue and are unwilling to provide strong negative responses.

Undesirable social behavior

With regard to responses that might be felt will meet with disapproval, several strategies have been suggested in the literature. Wilson and McClean 1994 recommend that they can be diffused by the use of categories, or brands, for respondents to tick. In their seminal book on questionnaire design, Sudman and Bradburn 1983 devote a great deal of space to discussing sensitive items. Their practical suggestions to mitigate the undesirable nature of certain behaviors include Wording the question in...