There are a number of problems in reviewing the literature that apply to all of the above strategies. First of all, there is what is sometimes called the 'file-drawer' problem. This relates to the fact that it is easier to publish studies that have statistically significant findings than it is to publish ones that do not, and so the latter get filed away. Torgerson (2006) calls this 'the Achilles heel' of systematic reviews, but it applies to all attempts to review the literature in any field.
Next, there is the problem of interpreting the findings of the published studies and seeing if these findings are relevant to your review. Research papers summarise a great deal of time and effort in a few pages. Reviewers summarise these papers in a few lines. Different reviewers emphasise different aspects of the same studies, and thus their accounts vary. Hartley (2000, pp. 166-7), for example, cites four different accounts of one particular study on homework. Readers reading only one of these may be mislead.
Relatedly, it may be more difficult to summarise adequately the results of qualitative studies. Dixon-Woods et al. (2006) discuss this at length in the context of summarising evidence-based studies and come to the conclusion that this really is a tricky problem.
Finally, there are some other assumptions made in literature reviews that do not withstand close scrutiny. These are:
1 that different dependent variables (manipulated by different investigators in different studies but designed to test the same hypotheses) are of equal validity or importance;
2 that the results obtained in one culture (e.g. American) are directly relevant to another one (e.g. British) and can thus be pooled together;
3 that the results obtained in one period (e.g. the 1960s) are the same as those that would be obtained today;
4 that the results obtained from limited samples (e.g. schoolchildren) apply to wider populations (e.g. adults); and
5 that the results obtained in simplified experiments apply to the much more complex 'real world'.
When writing a literature review, one solution to some of these problems is to examine in more detail the original papers and, in particular, the original materials used in the papers being reviewed. There are few examples of reviewers using such strategies - although it is clearly advisable to do so when writing the literature review in theses. Hartley et al. (1980) provided three such illustrations. One, by Macdonald-Ross (1977), concluded that Vernon's (1946) results on the effectiveness of diagrams arose largely as a consequence of her using poorly designed diagrams. Similarly, Elashoff and Snow (1971) were able to write a devastating critique of Pygmalion in the Classroom after examining the tests and procedures used by Rosenthal and Jacobson (1968). And finally, Klare (1976) read thirty-six studies on the effects of readability upon the comprehension of text. Nine of these were published papers, and twenty-seven were unpublished theses. Klare found that 100 per cent of the published studies contained statistically significant findings, compared with sixty per cent of the dissertations. This, of course, altered the nature of his review, and his conclusions.
Was this article helpful?