A selfgenerated identification coding procedure

The identification procedure just described does not ensure anonymity but only saves the salient act of students' writing their name on the questionnaire. A more complex method of ensuring identifiable anonymity has been piloted by Kearney et al. (1984) with some success. This method involves students' generating for themselves a unique personal code number and including this on every document they complete - hence the possibility for data linkability. Of course, no one except them would know...

Absolutely

Questionnaires do not emerge fully-fledged they have to be created or adapted, fashioned and developed to maturity after many abortive test flights. In fact, every aspect of a survey has to be tried out beforehand to make sure that it works as intended. So when and what shall we pilot While it is useful to have 'ongoing piloting' by continuously discussing every aspect of questionnaire design with a colleague or a friend, there are two points where a more formal trial run is needed (1) Once the...

Administering The Questionnaire

3.1.2 How large should the sample be 73 3.1.3 The problem of respondent self-selection 75 3.2 Main types of questionnaire administration 76 3.2.1 Administration by mail 77 3.2.2 One-to-one administration 81 3.3 Strategies to increase the quality and quantity of participant response 83 3.3.2 Attitudes conveyed by teachers, parents, and other authority figures 85 3.3.3 Respectable sponsorship 85 3.3.4 The behavior of the survey administrator 86 3.3.5 Communicating the purpose and significance of...

Anonymity

One frequent method used to diffuse sensitive items is to make the questionnaire anonymous. For example, in a student questionnaire that asked the learners to evaluate their language teacher and the course (Cl ment, D rnyei, & Noels, 1994), using similar items to the ones employed in the Gliksman et al. (1982) study just mentioned, we felt it unlikely that the 16 17-year-old teenagers in the sample were going to agree to give us honest answers without being assured about the anonymity of the...

Appropriate sampling of the content

Ad hoc questionnaire design involves jotting down a number of relevant questions without any rigorous procedure to ensure that the coverage is comprehensive. The problem with this method, as Davidson (1996, p. 10) highlights, is that You cannot analyze what you do not measure. That is, not even the most sophisticated data analysis techniques will be able to compensate for leaving out some important questions from the data collection by accident. Certain omissions are bound to occur even in...

Attitudes conveyed by teachers parents and other authority figures

Data gathering often takes place in someone's 'home ground.' In school settings, for example, students usually hear about the survey first from their teachers. The important thing to note with respect to this is that participants are rather quick to pick up their superiors' (e.g., teachers' or bosses') attitude toward the survey and only acquiesce if the message they receive is positive. Similarly, parental disposition can also have a major impact on students' willingness to respond. It is...

Avoid items that are likely to be answered the same way by everybody

In rating scales we should avoid statements that are likely to be endorsed by almost everyone or almost no one. In most cases these items are not informative and they are certainly difficult if not impossible to process statistically. Here is a recent example from my own research (D rnyei & Cl ment, 2001) A questionnaire item asked students to rate the international role importance of six countries, including the United States. As can be imagined, most respondents gave the U.S. the top...

Basic ethical principles of data collection

Drawing on Oppenheim's (1992) and Sudman and Bradburn's (1983) discussion of ethical issues in survey research, the following five principles can be compiled Principle 1 No harm should come to the respondents as a result of their participation in the research. This is the primary ethical principle governing data collection and it overrides all other considerations. Principle 2 The respondent's right to privacy should always be respected, and no undue pressure should be brought to bear. That is,...

Checklists

Checklists are similar to rank order items in that they consist of a list of descriptive terms, attributes, or even objects, and respondents are instructed to mark the items on the list that apply to the particular question. For example, students might be asked to mark all the adjectives in a list of personality characteristics that describe their teacher. This evaluation would, then, yield a score for the teacher on each characteristic, indicating how many raters checked the particular...

Clarification questions

Certain answers may be potentially so important that it is worth attaching a clarification question to them, for example in a 'routed' form If you rated the coursebook you are using as poor or very poor, please briefly explain why. Write your answer here Clarification questions are also appropriate when there is an Other category in a multiple-choice item. Typically, Please specify is used and some space is left for the respondent to provide a statement.

Closedended Questionnaire Items

Let us start our exploration of the various types of questionnaire items by first examining the most frequent question type closed-ended (or simply 'closed) questions. Although this category subsumes several very different item types, these all share in common the fact that the respondent is provided with ready-made response options to choose from, normally by encircling or ticking one of them or by putting an 'X' in the appropriate slot box. That is, these items do not require the respondents...

Coding Questionnaire Data

Most data analysis software handles data in a numerical rather than in an alphabetic form, and even with programs that allow the storage of information recorded as letters, the procedures that are available for handling such data are limited compared to the vast arsenal of statistical techniques to be used with numerical responses. Therefore, the first step of data processing usually involves converting the respondents' answers to numbers by means of coding procedures. As we will see, these...

Communicating the purpose and significance of the survey

Although actions may speak louder than words, this does not mean that words don't matter. An important element in 'selling' the survey to the participants is communicating to them the purpose of the survey and conveying to them the potential significance of the results. People tend not to mind answering questions if they see the point. We should also be aware of the fact that, as Gillham (2000) warns us, in our information-conscious age there is a general suspicion that much more data are...

Computer Programs For Processing Questionnaire Data

There are numerous statistical software packages that can be used to process quantitative questionnaire data. Personally, I have always used, and been satisfied with, 'SPSS' (Statistical Package for the Social Sciences), which is one of the market leaders in this category. There are also various computer programs to facilitate the qualitative analysis of transcribed texts (e.g., NUD*IST, NVivo). From a survey researcher's point of view, programs that can handle quantitative and qualitative...

Data cleaning

The initial data file will always contain mistakes. Some of these are the result of human error occurring during the data entry phase (e.g., typing the wrong number) and some are mistakes made by the respondent when filling in the questionnaire. Data cleaning involves correcting as many of these errors and inaccuracies as possible before the actual analyses are undertaken. The main checks and techniques are as follows Correcting impossible data. Most items have a specific range, determined by...

Descriptive statistics

Descriptive statistics are used to summarize sets of numerical data in order to conserve time and space. It is obvious that providing the mean and the range (i.e., minimum and maximum values) of a variable is a more professional way of describing the respondents' answers than listing all the scores that have been obtained. And if we also include the standard deviation of the results (which is an index of the average disparity among the scores), we have achieved a well-rounded description of the...

Detachment from real life

Researchers often note how ironical it is that months of hard labor can sometimes be summarized in one or two tables. While this may not be a problem in basic research - after all, Einstein's theory of relativity did not exactly take up several volumes either - in more applied studies when we are looking at concrete questions concerning real people, a primarily quantitative summary may lose some of the edge and flavor of the original issue. This is when a few open-ended items in the...

Entering The Data Into A Computer File

With the coding frames and the codebook ready, we need to get down to the rather tedious and time-consuming process of keying in the data. All of us involved in survey research have spent countless number of hours in front of a computer screen typing seemingly endless rows of figures. However boring and mechanical this job may be, it is essential that we maintain concentration because every mistyped number will be a contamination of the dataset. In agreement with Brown (2001), I have found that...

Examining the reliability and validity of the data

Reliability and validity are two key concepts in measurement theory, referring to the psychometric properties of the measurement techniques and the data obtained by them. The reliability of a psychometric instrument refers to the extent to which scores on the instrument are free from errors of measurement. For example, bathroom scales are not reliable if they show different figures depending on how steamy the air in the bathroom is, and neither are proficiency test raters if their evaluation...

Figures

Figures are methods to visualize various characteristics of the data. I have used two types of figures in the past, charts diagrams and schematic representations. Charts diagrams offer a useful way of describing the size strength of variables in relation to each other. Bar charts and line charts use a vertical 7-axis and a horizontal X-axis to present data (see Figures 1 and 2). The vertical axis usually represents the unit of measurement (or dependent variable) and the horizontal axis the...

Final piloting dress rehearsal

Based on the feedback received from the initial pilot group we can normally put together a near-final version of the questionnaire that 'feels' OK and that does not have any obvious glitches. However, we still do not know how the items will work in actual practice, that is, whether the selected respondents will reply to the items in the manner intended by the questionnaire designers. There is only one way to find out by administering the questionnaire to a group of respondents who are in every...

First things first Identification code and Research Logbook

Before we get down to actual coding, there are two things to be done. One is compulsory, the other highly recommended. The compulsory task is to give each questionnaire a unique identification code. In practice this may involve taking each questionnaire one by one and numbering them sequentially by writing a code number in one of the top corners of the front page. Questionnaires coming from the same group (e.g., same school or class) should be kept together and marked with a special code for...

Fred Davidsons plea for backing up data

Law 2 Do it now. - Anonymous I have noticed that nobody seems to learn to back up data until he or she has a serious disaster with data that are personally quite important. Please prove me wrong and abide by these two laws as early in your data career as possible. The traditional (and still frequently used) method of entering data into a computer file involves creating a rectangular text file (e.g., a word-processing file saved in a text only with line breaks file format) in...

General instruction

As a minimum, the general instruction (or 'opening greeting') should cover the following points (see also Section 3.2.1, for special instructions for mail surveys) What the study is about and why it is important or socially useful. The organization responsible for conducting the study. Emphasizing that there are no right or wrong answers requesting honest answers and trying to elicit integrity in general. For better readability and emphasis, the instructions should be graphically highlighted,...

Grouping And Ordering Items

Once all the items to be included in the questionnaire have been written or collected, we need to decide on their order. Item sequence is a significant factor because the context of a question can have an im pact on its interpretation and the response given to it. Indeed, the meaning of almost any question can be altered by the adjacent questions. However, it is usually acknowledged that research has not as yet generated any specific theoretical rules to order questions, beyond some broad...

Handling missing data

One issue that should definitely be resolved before the analyses is deciding what to do with missing values. They are a nuisance in many ways. First, it is not always clear whether the lack of any useful response is meaningful or not. For example, if Rupert is asked about what foreign languages he speaks and he leaves the question unanswered, would this mean that (a) Rupert only speaks his mother tongue, or (b) he has skipped the question by mistake, or (c) he has intentionally refused to...

How much to generalize

With regard to the issue of generalizing, it is so easy to offer the broad and rather unhelpful guideline Do not over generalize However, research in most cases is all about the need to produce generalizable findings. After all, with the exception of 'action research,' researchers in the L2 field rarely investigate a sample with the sole purpose of wanting to know more only about the particular people under investigation. Instead, what we normally want to do is find out more about the...

How To Write Good Items

Over the past 50 years, survey researchers have accumulated a considerable body of knowledge and experience about what makes a questionnaire item good and what the potential pitfalls are. However, most specialists also emphasize that question design is not a 100 scientific activity because in order to write good items one also needs a certain amount of creativity and lots of common sense. Furthermore, alternative versions of questions must be rigorously piloted because in the absence of hard...

IlllllJlll

Pie charts are used to describe proportions and percentages. The first pie chart in Figure 3 can display, for example, the proportion of focusing on three L2 skills in a language course. If we want to highlight changes, we can use two pies. Thus, the second pie below can be seen, for example, to represent the revised curriculum after some significant educational reform. By changing the overall size of the second pie we can also indicate growth or shrinkage (Fink, 1995) - the pie charts in...

Include both positively and negatively worded items

In order to avoid a response set in which the respondents mark only one side of a rating scale, it is worth including in the questionnaire both positively and negatively worded items. In addition, a balanced mixture might also reduce the harmful effects of the 'acquiescence bias' (cf. Section 1.2.2). The term 'negatively worded item' means that it focuses on negative rather than positive aspects of the target, and we should note that it is all too easy to fall into the trap of trying to express...

Indeed

If respondents are clear about what you are trying to find out and why, they are much more likely to respond appropriately and helpfully, or, indeed, at all. There is a curious convention that if you tell respondents what you are trying to find out, this will 'bias' them. It might simply make them more helpful. If you are mysterious about the purpose of the questionnaire they may be disinclined to answer or misunderstand the purpose, and so bias their answers in that way. Just like the cover...

Instructions

The title of the questionnaire is followed by instructions. These cannot be too long and yet need to be informative and well pitched because they play an important role in determining the respondents' feelings toward the questionnaire and in specifying how they should go about answering the items. Instructions are of two types General instruction (or 'opening greeting') at the beginning of the questionnaire. Specific instructions introducing each new task.

Item analysis

Item analysis can be conducted at two different points in the survey process After the final piloting stage - in this case the results are used to fine-tune and finalize the questionnaire. After the administration of the final questionnaire - after such a 'post hoc analysis' the results are used to screen out any items that have not worked properly. The procedures in both cases are similar. They usually involve checking three aspects of the response pattern (1) Missing responses and possible...

Little or no opportunity to correct the respondents mistakes

Questionnaire items focus on information which the respondents know best, and therefore the researcher has little opportunity to double-check the validity of the answers. Sometimes respondents deviate from the truth intentionally (see further) but it is also common that -as just mentioned - they simply misunderstand or forget something, or do not remember it correctly. Another fairly common situation is when informants do not know the exact response to a question yet answer it without...

Not indeed

The world is full of well-meaning people who believe that everyone who can write plain English and has a modicum of common sense can produce a good questionnaire. This book is not for them. As already mentioned in the Introduction, my interest in questionnaires is pragmatic and practice-driven. I use them all the time and I would like the measures obtained by them to meet high research standards. Having fallen into many of the existing pitfalls several times, I intend for this book to offer...

Numerical rating scales

Teenagers sometimes play a rating game whereby they evaluate the appearance and 'sexiness' of the various girls boys they see passing by in the street on a scale of 1-10. They would be surprised to hear that what they are doing is applying numerical rating scales. These scales involve 'giving so many marks out of so many,' that is, assigning one of several numbers corresponding to a series of ordered categories describing a feature of the target. The popularity of this scaling technique is due...

Opening questions

Just like with any other piece of writing, the initial section of a questionnaire is particularly important in that it sets the tone. This is partly the reason that instructions (cf. Sections 2.2.2 and 3.3.7) play a significant role, and this is also why the first few 'opening' questions should be carefully selected. In order to create a pleasant first impression, the starter questions need to be interesting, relatively simple yet at the same time focused on some important and salient aspect,...

Potential threat

With regard to items in which an honest answer can pose some real threat to the respondent (e.g., questions about illegal activities, or asking students to evaluate their language teacher), the main task is to convince the respondents that their answers will remain confidential. Obviously, offering complete anonymity in such cases might be helpful, but this may not be feasible in certain complex research projects where we need to match the data with information obtained from other sources (cf....

Questionnaire items

After the instructions comes the central part of the questionnaire, the actual items. They will be discussed in detail in Sections 2.3 - 2.7. Questionnaire items rarely take the form of actual questions that end with a question mark. The item type found in Sample 2.3, for example, is far more common than that in Sample 2.2 (which is a real question). The items need to be veiy clearly separated from the instructions. This is where different typefaces and font styles come in handy. Unless the...

Quota sampling

In quota sampling the researcher defines certain distinct subgroups (e.g., boys and girls, or age cohorts) and determines the proportion of the population that belongs to each of these subgroups (e.g., when targeting language teachers, determining that the female-male ratio among them is 70 -30 in a particular setting). The actual sample, then, is selected in a way as to reflect these proportions (i.e. 70 of the sample will be women). In fact, the common intention to select a roughly equal...

Random sampling

The key component of scientific sampling procedures is random sampling. This involves selecting members of the population to be included in your sample on a completely random basis, a bit like drawing numbers from a hat (e.g., by numbering each member and then asking the computer to generate random numbers). The assumption underlying this procedure is that it minimizes the effects of any extraneous or subjective variables that might affect the outcome of the survey study. Combining random...

Recoding values

It has been recommended earlier (in Section 2.6.2) that in order to avoid a response set where the respondents mark only one side of a rating scale, it is worth including in the questionnaire both positively and negatively worded items this may also reduce the harmful effects of the acquiescence bias. However, if we have such negatively worded items, we must not forget to reverse the scoring for these be fore including them in multi-item scales. This may sound like an obvious and trivial...

Reducing the number of variables in the questionnaire

Once we have completed data cleaning and data manipulation, we are ready to embark on the examination of the obtained data. The first step in analyzing questionnaire data is always to reduce to manageable proportions the number of variables measured by the question naire so that the mass of details does not prevent us from seeing the forest through the trees. The rationale for this is that - in accordance with the arguments in Section 2.3.2 - a well-designed questionnaire contains multiple...

Regrettably

People tend not to take questionnaires seriously their answers may be frankly frivolous. In view of these handicaps, the researcher's task to motivate the respondents to give truthful and thoughtful answers to all the relevant items on the questionnaire might seem daunting if not impossible. The good news, however, is that people in general like to express their opinions and do not mind answering questions as long as they think that the survey is related to a worthy cause and that their opinion...

Sample General instruction

We would like to ask you to help us by answering the following questions concerning foreign language learning. This survey is conducted by the Language Research Group of the University of X to better understand This is not a test so there are no right or wrong answers and you don't even have to write your name on it. We are interested in your personal opinion. Please give your answers sincerely as only this will guarantee the success of the investigation. Thank you very much for your help.

Sampling procedures

Broadly speaking, the sample is the group of people whom researchers actually examine and the population is the group of people whom the survey is about. For example, the population in a study might be EFL learners in Taiwanese secondary schools and the actual sample might involve three Taiwanese secondary classes. That is, the target population of a study consists of all the people to whom the survey's findings are to be applied or generalized. Why don't we include every member of the...

Specific instructions

Specific instructions explain and demonstrate how respondents should go about answering the questions. It is obvious that this is a crucial part. Each new task-type requires instructions, and in order to separate these instructions from the rest of the text, they should be graphically highlighted, for example by printing them in bold (just like the general instruction). A very important role of the instructions is to explain how various rating scales (cf. Section 2.4.1) work and what the...

Standardizing the data

When we use pooled results from various subsamples, one way of controlling for the heterogeneous nature of the subgroups is to use standardized scores. The standardization of raw scores involves the conversion of the distribution within a sample in a way that the mean will be 0 and the standard deviation 1. Thus, standard scores express how much each raw value is different from the subgroup mean, and by equalizing the means, scores obtained from different subsamples (e.g., different classes in...

Tables

Tables are used to summarize data about the respondents and their responses, and to present results of statistical analyses (see Sample 4.2 on page 113, and Sample Table 1 on page 129). They are typically made up of rows and columns of numbers, each marked with headings and subheadings. They can provide a more accurate and richer description than figures but they are less digestible because they lack the advantage of a visual impact. Tables are, therefore, more appropriate for articles in...

Technical information to accompany survey results

Novice researchers often make the mistake of concentrating only on the presentation of their actual findings in their survey reports. While this approach may appear logical, it fails to take into account the fact that in order to be able to interpret (and believe) the claims made, readers will have to be convinced that the methodology used to produce the particular findings was appropriate. This does not mean that we can only report results if our study did not have any methodological...

The problem of respondent selfselection

To conclude the discussion of the various sampling issues for research purposes in general, we need to highlight a potential pitfall that might put the validity of the survey at risk the problem of participant self-selection. This refers to cases when for various reasons the actual composition of the sample is not only the function of some systematic selection process but also of factors related to the respondents' own willingness to participate. Problems can arise, for example, when...

The style and layout of the questionnaire

As argued earlier, respondents are normally willing to spend time and effort on a questionnaire if they believe that they are contributing to a serious investigation. One factor that plays an important role in convincing them about this is the professional quality of the questionnaire. The tone and content of the printed instructions, the layout and typesetting of the items, and small details such as thanking the participants for their cooperation, can all contribute to the formation of a...

Title

Just like any other piece of writing, a questionnaire should have a title to identify the domain of the investigation, to provide the respondent with initial orientation, and to activate various content schemata. Because uninformative titles fail to achieve these objectives, Aiken (1997) suggests that we should try and avoid title words like questionnaire or survey. For better identification, the title might be accompanied by the date of the survey administration and the name of the...

Truefalse items

In some scales the designers only set two response options true versus false (or 'yes' or 'no'), resulting in what is usually referred to as a 'true-false item' While generally it is true that the more options an item contains, the more accurate evaluation it yields, there might be cases when only such a polarized, yes-no decision can be considered reliable. For example, little children are sometimes seen as incapable of providing more elaborate ratings, and some personality test items also...

Use simple and natural language

As a rule, in questionnaire items we should always choose the simplest way to say something. Items need to be kept clear and direct, without any acronyms, abbreviations, colloquialisms, proverbs, jargon, or technical terms. We should try to speak the 'common language' and find synonyms for the polysyllabic and Latinate constructions that come easily to the tongue of the college educated (Converse & Presser, 1986, p. 10). Oppenheim (1992) argues that the most important rule in writing rating...

Well said

When an individual uses descriptive statistics, he talks about the data he has but with inferential statistics, he talks about data he does not have. (Popham & Sirotnik, 1973, p. 40) Statistical significance denotes whether a particular result is powerful enough to indicate a more generalizable phenomenon. If a result is non-significant, this means that we cannot be certain that it did not occur in the particular sample only because of chance (e.g., because of the unique composition of the...

Tests are not questionnaires

Written, self-completed (or self-report) questionnaires are very similar to written tests, yet there is a basic difference between them. A 'test' takes a sample of the respondent's behavior knowledge and, on the basis of this sample, inferences are made about the degree of the development of the individual's more general underlying competence abilities skills (e.g., overall L2 proficiency). Thus, a test measures how well someone can do something. In contrast, questionnaires do not have good or...

Summarizing And Reporting Questionnaire Data

Survey data can be used for a great variety of purposes and each of these might require somewhat different types of summaries and reports of the findings. It is obvious, for instance, that a PhD dissertation will have to meet criteria that are very different from the presentation requirements of a summary of student achievement at a school staff meeting. Rather than attempting to provide templates for all the diverse applications (such templates are readily available in various writers'...

Openended Questions

Open-ended questions include items where the actual question is not followed by response options for the respondent to choose from but rather by some blank space (e.g., dotted lines) for the respondent to fill. As we have seen in the previous chapter (in Section 1.3), questionnaires are not particularly suited for truly qualitative, exploratory research. Accordingly, they tend to have few open-ended questions and even the ones included are relatively short, with their 'openness' somehow...

Measuring and ensuring internal consistency reliability

Internal consistency reliability is measured by the Cronbach Alpha coefficient (named after its introducer, L. J. Cronbach). This is a figure ranging between zero and +1, and if it proves to be very low, either the particular scale is too short or the items have very little in common. Internal consistency estimates for well-developed attitude scales containing as few as 10 items ought to approach 0.80. Because of the complexity of the second language acquisition process, L2 researchers...

Respondent literacy problems

Questionnaire research makes the inherent assumption that the respondents can read and write well. Even in the industrialized world this is not necessarily the case with regard to the whole population Statistics of about 5 -7 are regularly quoted when estimating the proportion of people who have difficulty reading, and the number of those who are uncomfortable with writing is even bigger. It is therefore understandable that for respondents with literacy problems, filling in a questionnaire can...

Ellard and Rogers p Ten Commandments of Question Writing

Thou shalt not create double-barreled items. II. Thou shalt not use 'no' and 'not' or words beginning with 'un.' III. Thou shalt match the vocabulary used in items to the vocabulary of those who will respond to them. IV. Thou shalt not use complex grammatical forms. V. Thou shalt have 40 to 60 true- or agree-keyed items. VI. Thou shalt not use redundant or irrelevant items. VII. Thou shalt not permit any loaded questions to appear in your questionnaire. VIII. Thou shalt not mix response...

Avoid doublebarreled questions

Double-barreled questions are those that ask two (or more) questions in one while expecting a single answer. For example, the question How are your parents asks about one's mother and father, and cannot be answered simply if one of them is well and the other unwell. Indeed, questions dealing with pluralisms (children, students) often yield double-barreled questions, but compound questions also often fall into this category (e.g., Do you always write your homework and do it thoroughly ). With...

Unreliable and unmotivated respondents

Most people are not very thorough in a research sense, and this is all the more true about dealing with questionnaires - an activity which typically they do not enjoy or benefit from in any way. Thus, the re- suits may vary greatly from one individual to another, depending on the time and care they choose or are able to give (Hopkins, Stanley, & Hopkins, 1990). Respondents are also prone to leave out some questions, either by mistake or because they did not like them, and Low (1999) presents...

Promising feedback on the results

Christopher Ryan (personal communication) has always maintained that survey researchers can do great damage if they pursue what he called a 'slash and burn' strategy. By this he meant that surveyors typically exploit their participants without offering anything in return - as soon as the data have been gathered, they disappear. On the other hand, if someone puts reasonable effort into answering the questions, this involvement will create a natural curiosity about the project and its outcome. It...

Strategies for getting around anonymity

We saw in Section 2.1.3 that - from the researcher's point of view -respondent anonymity is often undesirable in survey research because without proper identification we cannot match survey data with other sources of information obtained about the same participants (e.g., course marks or other questionnaires). The other side of the coin, however, is that with certain sensitive questions anonymity may be desirable from the respondents' point of view because they may feel safer this way to...

Content Analysis Of Openended Questions

Although it was argued in Sections 1.3 and 2.5 that wide-open, essaylike questions do not work well in questionnaires and therefore should be avoided, questions that are slightly 'less open' can have some merits and are well worth experimenting with as long as this does not exist at the expense of the closed questions (in terms of response time or willingness). Because open-ended questions do not have precoded response options, their processing is less straightforward than that of closed items....

What do questionnaires measure

Broadly speaking, questionnaires can yield three types of data about the respondent factual, behavioral, and attitudinal. 1. Factual questions (also called 'classification' questions or 'subject descriptors') are used to find out about who the respondents are. They typically cover demographic characteristics (e.g., age, gender, and race), residential location, marital and socioeconomic status, level of education, religion, occupation, as well as any other background information that may be...

Main Types Of Questionnaire Administration

In social research the most common form of administering questionnaires is by mail. Educational research is different in this respect because administration by hand is just as significant (if not more) as postal surveys. Within non-postal surveys, we can distinguish two distinct subtypes, one-to-one administration and group administration. Because the administration method has a significant bearing on the format and to some extent also on the content of the questionnaire, we need to examine...

Introduction

One of the most common methods of data collection in second language (L2) research is to use questionnaires of various kinds. The popularity of questionnaires is due to the fact that they are easy to construct, extremely versatile, and uniquely capable of gathering a large amount of information quickly in a form that is readily process-able. Indeed, the frequency of use of self-completed questionnaires as a research tool in the L2 field is surpassed only by that of language proficiency tests....

Questionnaires in Second Language Research

Asking questions is one of the most natural ways of gathering information and, indeed, as soon as babies have mastered the basics of their mother tongue they launch into a continuous flow of questions, and keep going throughout the rest of their lives. Some people such as reporters actually make a living of this activity and survey polling organizations can base highly successful businesses on it. Because the essence of scientific research is trying to find answers to questions in a systematic...

Using multiitem scales

Multi-item scales are the key components to scientific questionnaire design, yet this concept is surprisingly little known in the L2 profession. The core of the issue is that when it comes to assessing attitudes, beliefs, opinions, interests, values, aspirations, expectations, and other personal variables, the actual wording of the questions assumes an unexpected importance minor differences in how the question is formulated and framed can produce radically different levels of agreement or...

Processing Questionnaire Data

Having designed a questionnaire and administered it to an appropriate sample is half the battle. Now comes the final phase of our research, the processing of the data. The starting point of this phase is the very salient presence of stacks of completed questionnaires taking up what little empty space there is in our office. Accordingly, our initial priority is to get rid of these stacks and transform the information that is hidden in these piles of questionnaires into a more useful form that we...

Sensitive topics

'Sensitive' topics are not confined to explicitly illegal or embarrassing subjects but also include basic demographic items such as age or marital status. Indeed, various facts of life can carry such a prominent social and emotional loading that questions targeting them often fall prey to the respondents' 'social desirability' bias (cf. Section 1.2.2). Depending on our core values, we are likely to overreport on what we conceive as a positive aspect and underreport on a negative one....

Disadvantages

Although the previous description of the virtues of questionnaires might suggest that they are perfect research instruments, this is not quite so. Questionnaires have some serious limitations, and some of these have led certain researchers to claim that questionnaire data are not reliable or valid. I do not agree with this claim in general, but there is no doubt that it is very easy to produce unreliable and invalid data by means of ill-constructed questionnaires. In fact, as Gillham (2000, p....

Simplicity and superficiality of answers

Because respondents are left to their own devices when filling in self-completed questionnaires, the questions need to be sufficiently simple and straightforward to be understood by everybody. Thus, this method is unsuitable for probing deeply into an issue (Moser & Kalton, 1971) and it results in rather superficial data. The necessary simplicity of the questions is further augmented by the fact that the amount of time respondents are usually willing to spend working on a questionnaire is...

On length and crowdedness

Perhaps the most common mistake of the beginner in questionnaire construction is to crowd questions together in the hope of making the questionnaire look short. While length is important, the respondent's perception of the difficulty of the task is even more important on self-administered questionnaires. A less crowded questionnaire with substantial white space looks easier and generally results in higher cooperation and fewer errors. (Sudman & Bradburn, 1983, p. 244) Orderly layout. Even if...

Questionnaires In Quantitative And Qualitative Research

The typical questionnaire is a highly structured data collection instrument, with most items either asking about very specific pieces of information (e.g., one's address or food preference) or giving various response options for the respondent to choose from, for example by ticking a box. This makes questionnaire data particularly suited for quantitative, statistical analysis. After all, the essential characteristic of quantitative research is that it employs categories, viewpoints, and models...

Complementing Questionnaire Data With Other Information

Having discussed how to construct and administer questionnaires, and then how to analyze and report the responses we have obtained, the final section of this book addresses ways of proceeding toward a Sample Table 1. Descriptive statistics of the content of this book Sample Table 1. Descriptive statistics of the content of this book fuller understanding of the content area targeted by our survey. As discussed in Chapter 1, although questionnaires offer a versatile and highly effective means of...

The coding frame and the codebook

Having marked each questionnaire with an identification number, we are ready to embark on the coding of the items. Except for extensive texts obtained by open-ended questions (which require special content analysis - cf. Section 4.4), the coding process for each item involves converting the answer into a numerical score. Because numbers are meaningless themselves and are also easy to mix up, a major element of the coding phase is to compile (a) a coding frame that specifies the meaning of the...

Conclusion and Checklist

The previous four chapters have provided a summary of questionnaire theory. Hopefully, they have also made a strong case for basing questionnaire design and processing on scientific principles rather than merely on the researcher's common sense. As emphasized in the Introduction, this book has been intended to serve practical purposes and therefore in this concluding section I will draw up a checklist of what I consider the most important points and recommendations for every phase of the...

Avoid ambiguous or loaded words and sentences

It goes without saying that any elements that might make the language of the items unclear or ambiguous need to be avoided. The most notorious of such elements are Nonspecific adjectives or adverbs e.g., good, easy, many, sometimes, often . Items containing universals such as 'all,' 'none,' 'never.' Modifying words such as 'only,' 'just,' 'merely' - these should be used with moderation. Words having more than one meaning. Loaded words e.g., 'democratic,' 'modern,' 'natural,' 'free,' etc. ,...

Inferential statistics

Descriptive statistics are useful, for example, to describe the achievement of a particular class of learners. What happens, however, if we notice that, say, the L2 learning achievement of boys and girls shows a remarkable difference in our sample, with girls outperforming boys which is often the case Can we draw the inference that girls are better language learners No. Based on descriptive statistics all we can say is that in this class girls did better than boys. In order to venture any...

Group administration

In L2 research, group administration is the most common method of having questionnaires completed. One reason for this is that the typical targets of the surveys are language learners studying within institutional contexts, and it is often possible to arrange to administer the instrument to them while they are assembled together, for example, as part of a lesson or slotted between certain other organized activities. The other reason for the popularity of this administration format is that it...

Initial piloting of the item pool

The first time in the questionnaire construction process that some external feedback is indispensable is when we have prepared an initial item pool cf. Section 2.6.1 , that is, a large list of possible items, and we are ready to reduce the number of questions to the intended final number. The initial piloting of the item pool usually consists of the following steps Select three or four people who are motivated to spend some time to help you and whose opinion you value. Some of them should not...

Onetoone administration

One-to-one administration refers to a situation when someone delivers the questionnaire by hand to the designated person and arranges the completed form to be picked up later e.g., handing out questionnaires to colleagues at work . This is a much more personal form of administration than mail surveys and therefore the chances for the questionnaires to be returned are significantly better. The personal contact also allows the questionnaire administrator to create rapport with the respondent, to...

Rank order items

It is a common human mental activity to rank order people, objects, or even abstract concepts according to some criterion, and rank order items in questionnaires capitalize on our familiarity with this process. As the name suggests, these items contain some sort of a list and respondents are asked to order the items by assigning a number to them according to their preferences. Wilson and McClean 1994 warn us that it may be very demanding to arrange items in order of importance whenever there...

Sensitive topics and anonymity

It was mentioned in Section 1.2.2 that respondents are sometimes reluctant to give honest answers to sensitive questions. Questionnaire items differ greatly in terms of how threatening imposing sensitive embarrassing they feel. It requires little justification that we need to approach the issue of constructing and administering the questionnaire in a very different way if it concerns, for example, the evalua tion of the L2 teacher or the school rather than one's interest in travelling abroad....

Clear and orderly structure

The most important aspect of sequencing questions is to ensure that the respondents' overall impression is that the structure is well-organized and orderly. If the ordering of questions is unpredictable or seemingly haphazard, it will frustrate respondents and make the study appear ill-considered and amateurish Newell, 1993 . Neither the content nor the style of the questionnaire should jump around Aiken, 1997 - the items should seem as a series of logically organized sequences. To achieve...

Convenience or opportunity sampling

The most common sample type in L2 research is a convenience or opportunity sample, where an important criterion of sample selection is the convenience for the researcher Members of the target population will be selected for the purpose of the study if they meet certain practical criteria, such as geographical proximity, availability at a certain time, or easy accessibility. If we decide, for example, to study a class group because we have good contacts with the particular school, that would be...

The coding frame

The coding frame is a classification scheme that offers a numerical score for every possible answer to an item see Sample 4.2 on page 100 . The minimum number of categories is two, as with yes no questions or gender data 'Yes' and 'male' are usually coded '1,' whereas 'No' and 'female' are coded '2.' For some open-ended questions e.g., What foreign languages have you learned in the past the coding frame can have many more categories - in fact, as many as the number of different answers in all...

Factual or personal or classification questions at the end

As Oppenheim 1992 concludes, novice researchers typically start to design a questionnaire by putting a rather forbidding set of questions at the top of a blank sheet of paper, asking for name, address, marital status, number of children, religion, and so on. These personal classification questions tend to be very off-putting Having been through the various introductory phases, respondents are now ready to look at some interesting questions dealing with the topic of the study. Instead, they are...

Constructing the Questionnaire

Section 1.2.2 contained a long list of potential problems with self-completed questionnaires. My goal was not to dissuade people from using such instruments but rather to raise awareness of these possible shortcomings. It is true that respondents are often unmotivated, slapdash, hasty, and insincere, yet it is also an established fact that careful and creative questionnaire construction can result in an instrument that motivates people to give relatively truthful and thoughtful answers, which...

Layout

Sanchez 1992 points out that the design of the questionnaire layout is frequently overlooked as an important aspect of the development of the instrument. This is a mistake Because in surveys employing self-completed questionnaires the main interface between the researcher and the respondent is the hard copy of the questionnaire the format and graphic layout carry a special significance and have an important impact on the responses. Over the past 15 years I have increasingly come to the belief...

References

Rating scales and checklists Evaluating behavior, personality, and attitudes. New York John Wiley. Aiken, L. 1997 . Questionnaires and inventories Surveying opinions and assessing personality. New York John Wiley. Anderson, L. W. 1985 . Attitudes and their measurement. In T. Hus n amp T. N. Postlethwaite Eds. , The international encyclopedia of education Vol. 1, pp. 352-58 . Oxford Pergamon. Bardovi-Harlig, K. 1999 . Researching method. In L. F. Bouton Ed. , Pragmatics and...

General Features

Between the initial idea of preparing a questionnaire for the purpose of our research and actually getting down to writing the first draft, a number of important decisions need to be taken regarding the general features of the would-be instrument. First of all, we need to specify the maximum length of time that the completion of the questionnaire could take then we need to consider general format characteristics', and finally we need to think about the issue of anonymity, particularly if we are...

The behavior of the survey administrator

After all the preliminary considerations, we have finally arrived at the actual day of the survey. The survey administrator is facing the participants obviously, this section does not apply to postal surveys and is ready to launch into his her pep talk. However, in line with the saying, 'Actions speak louder than words,' we need to be aware that our behavior is also conveying important messages to the respondents. The administrators of the questionnaire are, in many ways, identified with the...

Piloting The Questionnaire And Conducting Item Analysis

Because in questionnaires so much depends on the actual wording of the items even minor differences can change the response pattern an integral part of questionnaire construction is 'field testing,' that is, piloting the questionnaire at various stages of its development on a sample of people who are similar to the target sample the instrument has been designed for. These trial runs allow the researcher to collect feedback about how the instrument works and whether it performs the job it has...

Social desirability or prestige bias

The final big problem with regard to questionnaires is that people do not always provide true answers about themselves that is, the results represent what the respondents report to feel or believe, rather than what they actually feel or believe. There are several possible reasons for this, and the most salient one is what is usually termed the social desirability or prestige bias. Questionnaire items are often 'transparent,' that is, respondents can have a fairly good guess about what the...

Openended questions at the end

As discussed in Section 2.5, if we include real open-ended questions that require substantial and creative writing, it is preferable to place them near the end rather than at the beginning of the questionnaire. In this way, their potential negative consequences e.g., the required work can put some people off others might get bogged down and spend most of the available time and mental energy agonizing over what they should write will not affect the previous items. In addition, some people find...