Strategies For Presenting Results In Reviews

There are at least six ways of presenting summaries of the results of research reviews, which can be placed along a continuum of statistical precision.

1 The narrative review: This is the kind of review that is typically used in this book. Writers research around a particular topic and then write a review of the field, giving their own 'take' on it, selecting evidence from whatever seems appropriate to them. This type of review is most common in text-books and popular journals. I once provided a case-history account of how to write such a review that was motivated by the need to rebut a claim by the UK government that primary school children benefited from doing homework (Hartley, 2000). The government had used spurious claims in order to specify how many hours each week children in primary school should spend on homework.

2 The narrative review with scoreboard: Here, writers strengthen the arguments of their reviews by supporting the claims made with tabular 'scoreboards'. Table 3.3.1 shows an example (with fictitious data).

3 A scoreboard plus details: Table 3.3.2 shows an example (with limited data) of how more detail can be provided in a scoreboard. The advantages of listing individual studies in different categories are that it enables the reader to trace the studies should they wish and, if they are familiar with the field, to see if any have been omitted.

Table 3.3.1 A 'scoreboard' giving the number of studies that show homework has an effect at different ages*

No. of studies showing that homework has or does not have an effect

Yes No

Primary school studies 1 6

Secondary school studies 10 3

* Fictitious data.

Table 3.3.2 An extract from a more detailed (unpublished) 'scoreboard'*

Studies showing that homework has an effect Yes No

Primary school Alton-Lee and Nuthall (1990) Cooper et al. (1998)

Secondary school Cooper et al. (1998) Faulkner and Blyth (1998)

Holmes and Croll (1989) Mau (1997)

Keith and Benson (1992) Wharton (1997)

Rutter et al. (1979) Tymms and FitzGibbon (1992) Zellman and Waterman (1999)

* With many references left out to save space.

4 A 'scoreboard' showing critical features: A common method of summarising results, particularly used in theses and dissertations, is to provide a table listing the key features of the studies being discussed. Table 3.3.3 provides a simplified and fictitious example. Such tables take a good deal of time to construct, but they can be enormously helpful for readers. The information provided in such tables also means that key information (e.g. the numbers and the ages of the participants, and the place of study) is not omitted, as often occurs in narrative reviews. Indeed, a series of such tables can be presented, each dealing with one particular feature in turn.

5 Meta-analytic 'scoreboards': Meta-analysis involves pooling the results that can be found from all the known studies on a given topic. Sometimes this number of studies is very high (e.g. studies of the effects of television), and sometimes it is quite small (e.g. studies of the effects of homework). The aim, however, is to arrive at an overall summary of the results for the topic in question.

To conduct a meta-analysis, all of the studies known to the researcher (or team of researchers) are accumulated, and the results are averaged according to certain rules. This usually involves, first of all, discarding a number of studies that do not include sufficient data, or the right kind of data (see below). Then, for each one of the remaining studies, the mean score of the control group is subtracted from the mean score of the experimental group, and the result is divided by the standard deviation of the control group (or both groups combined). Finally, the results obtained in step two are averaged over all the studies. The ensuing result is expressed in terms of an 'effect size' that indicates the importance of a particular variable. Table 3.3.4 provides an example from the field of homework. Effect sizes are typically interpreted as follows: 0.0 = no effect; 0.2 = small effect; 0.5 = medium effect; 0.8 = large effect. Thus, in Table 3.3.4, the effects of homework get larger as the children get older.

Table 3.3.3 A 'scoreboard' with critical features*

Study

Age group

Number of pupils

Subject matter

Length of study

Abba (1988)

5-7 yrs

20 per year

Arithmetic

1 week

Becca (1997)

7-8 yrs

2,0000

Varied

3 months

Cedda (2001)

6 yrs

10

Reading

7 weeks

Deffa (1999)

11-12 yrs

25 per year

English

8 weeks

Maths

Egga (1996)

12-14 yrs

13 per year

Science

1 week

Fehha (2005)

15-16 yrs

21 per year

English

8 weeks

Maths

Science

* Fictitious data.

Table 3.3.4 Effect sizes for studies of the effectiveness of homework

Homework versus

Homework versus

Time spent on

no homework

supervised study

homework

Primary school

0.15

0.8

0.04

11-14 years

0.31

0.24

0.14

15-17 years

0.64

0.33

0.53

Adapted with permission from Cooper and Valentine (2001). © Taylor & Francis, www.informaworld. com.

Adapted with permission from Cooper and Valentine (2001). © Taylor & Francis, www.informaworld. com.

Some people think that such meta-analytic reviews are superior to narrative reviews, but others provide criticisms (see Fink, 2005). To carry out a meta-analysis you need to know the sample sizes and the means and standard deviations of the experimental and control groups in every study included. This stricture, of course, excludes qualitative studies, and these studies can make important contributions. Student performance in homework is undoubtedly related to what they and their parents think about it. There is also some debate over whether or not some studies should be excluded from the averaging procedure — say on the grounds of limited sample sizes — but with meta-analytic studies it is usual to include all of the studies that one can. Some studies, however, do compare the results obtained with different procedures. Anderson's (2004) meta-analytic review of the effects of violent video games, for example, contrasted the results obtained when all of the studies known to the author were included with those obtained from a smaller sample of better studies. In this case the better studies yielded higher effect sizes.

6 Evidence-based 'scoreboards': With the 'evidence-based' approach, more studies are excluded on particular methodological grounds when making the overall summary of the results. In medical research, for example, it is usual to exclude comparison studies where the participants have not been allocated at random into experimental and control conditions. However, it is difficult to do this in all areas of study, and randomised controlled trials (RCTs) are rare in social science research. Torgerson et al. (2003), for example, were only able to find twelve RCTs in 4,555 reported investigations into improving adult literacy and/or numeracy, and, I know of no RCTs on the topic of homework.

The criteria for including studies in evidence-based studies have thus got wider for disciplines in the social sciences compared with medicine, but there are still many strictures concerning what should and should not be included in reviews of this kind (see Andrews, 2005). The importance of the evidence-based approach becomes more obvious when the overall picture obtained from RCTs is different from that obtained from studies using other, less stringent methods. Guyatt et al. (2000), for example, found that the pooled results from ten studies using RCTs in the field of sex education for adolescents showed no significant effects for the treatments overall, whereas the pooled results from seventeen non-RCT studies showed the treatments to be effective . . .

Profit From Fiction

Profit From Fiction

Theres a lot of talk about just how lucrative the fiction genre is and how countless new writers are able to cash in with a combination of selfpublishing and hitting the right market.

Get My Free Ebook


Post a comment