Piloting The Questionnaire And Conducting Item Analysis

Because in questionnaires so much depends on the actual wording of the items (even minor differences can change the response pattern) an integral part of questionnaire construction is 'field testing,' that is, piloting the questionnaire at various stages of its development on a sample of people who are similar to the target sample the instrument has been designed for. These trial runs allow the researcher to collect feedback about how the instrument works and whether it performs the job it has been designed for. Based on this information, we can make alterations and fine-tune the final version of the questionnaire.

"if you do not have the resources to pilot-test your questionnaire, don't do the study."

The pilot test can highlight questions:

• whose wording may be ambiguous;

• which are too difficult for the respondent to reply to;

• which may, or should be, eliminated because, contrary to the initial expectations, they do not provide any unique information or because they turn out to measure something irrelevant;

• which - in the case of open-ended questions - are problematic to code into a small set of meaningful categories.

Piloting can also indicate problems or potential pitfalls concerning:

• the administration of the questionnaire;

• the scoring and processing of the answers.

Valuable feedback can also be gained about:

• the overall appearance of the questionnaire;

• the clarity of the instructions;

• the appropriateness of the cover letter (if there is one);

• the length of time necessary to complete the instrument.

Finally, this is also the phase when omissions in the coverage of content can be identified.

The importance of the piloting is in sharp contrast with the reality that so many researchers completely omit the pilot stage from their research design. Although this is understandable from a personal point of view because researchers at this stage are eager to get down to the survey and see the results, from a measurement perspective this practice is untenable. Regardless of how experienced the questionnaire designer is, any attempt to shortcut the piloting stage will seriously jeopardize the psychometric quality of the questionnaire (Moser & Kalton, 1971). Furthermore, my experience is that by patiently going through the careful editing procedures we can avoid a great deal of frustration and possible extra work later on.

Sometimes the omission of the pilot stage is not due to the lack of will/interest but rather to insufficient time. To do it well, piloting takes up a substantial period, which has often not been included in the timing of the research design. As we will see below, piloting is a stepwise process that, when properly done, can take several weeks to complete. This is usually much more than was originally intended for this phase of the research.

Was this article helpful?

0 0
Online Survey Champion

Online Survey Champion

There are people all over the world trying to find ways to make money online. From stay at home moms looking to make a few extra dollars to college students and entrepreneurs, the allure of making your own hours and working from home or from the local coffee shop is very appealing.

Get My Free Ebook


Responses

  • Niko Ketterer
    How to conduct item analysis on questionnaires?
    8 years ago

Post a comment