Search J Rheum

Advanced Search

Home

Current Issue

Archives

Guidelines for Authors

Classified Ads

Links

Search PubMed

Subscriptions

Subscriber Registration

Guidelines for Website Users

JRheum Update Service

Contact Info


Download PDF

View Table of Contents

Scope for Improvement in the Quality of Reporting of Systematic Reviews. From the Cochrane Musculoskeletal Group

BEVERLEY SHEA, LEX M. BOUTER, JEREMY M. GRIMSHAW, DANIEL FRANCIS, ZULMA ORTIZ, GEORGE A. WELLS, PETER S. TUGWELL, and MAARTEN BOERS

ABSTRACT.

Objective.
To assess the quality of reporting in Cochrane musculoskeletal systematic reviews (excluding back and injury reviews).

Methods. This study assessed all the Cochrane Musculoskeletal Group's systematic reviews from Issue 4, 2002, of the Cochrane Library Database of Systematic Reviews. Two reviewers independently extracted data and assessed quality. Two assessment tools were used, including an 18 item checklist and flow chart developed by the Quality of Reporting of Meta-analysis (QUOROM) consensus group, and a 10 item scale, the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ). One question on the latter scale (item 10) scores overall quality on a 7 point scale, with high scores indicating superior quality. Data were analyzed using univariate approaches.

Results. The 57 systematic reviews assessed were found to have good overall quality, with scores on individual items revealing only minor flaws. Documenting the flow of included and excluded studies and summarizing the results are 2 areas needing improvement in reporting. According to the Oxman-Guyatt scale the overall scientific quality of the Cochrane musculoskeletal reviews was good [mean 5.02 (95% CI 3.71–6.32)].

Conclusion. Our study found that the reporting quality of Cochrane musculoskeletal systematic reviews was generally good, although there was room for improvement. For example, it might be feasible to develop specific guidelines for reporting protocols. Certainly more work is needed in reporting search results, documentation of the flow of studies, identification of the type of studies, and summarization of the key findings. (J Rheumatol 2006;33:9-15; First Release: Nov 1, 2005)

Key Indexing Terms:

MUSCULOSKELETAL
COCHRANE
SYSTEMATIC REVIEWS
QUALITY ASSESSMENT
REPORTING


From EMGO Institute, Departments of Clinical Epidemiology and Biostatistics, VU University Medical Center, Amsterdam, The Netherlands; and Community Information Epidemiological Technologies (CIETcanada), Ottawa, Ontario, Canada.

B. Shea, MSc, PhD Candidate, VU University Medical Center, Community Information and Epidemiological Technologies; L.M. Bouter, PhD, Director, EMGO Institute, VU University Medical Center; J.M. Grimshaw, MD, PhD, Director, Ottawa Health Research Institute Clinical Epidemiology Program, Ottawa Hospital, Ottawa, Ontario, Canada; D. Francis, BSc, Research Associate, University of Brisbane, Brisbane, Australia; Z. Ortiz, MD, Teaching and Research, Centre for Epidemiological Research, National Academy of Medicine, Buenos Aires, Argentina; G.A. Wells, PhD, Professor, Department of Epidemiology and Community Medicine, University of Ottawa; P.S. Tugwell, MSc, MD, Professor, Department of Medicine, University of Ottawa; M. Boers, MD, PhD, Professor of Clinical Epidemiology, Department of Clinical Epidemiology and Biostatistics, VU University Medical Center.

Address reprint requests to B. Shea, CIETcanada, 1 Stewart Street, Room 319, Ottawa, Ontario K1N 6N5 Canada. E-mail: bshea@ciet.org

Accepted for publication August 24, 2005.


It has been estimated that a physician would need to read 17–20 journal articles a day to keep abreast of all research relevant to a particular area of clinical practice1. Because this is clearly impractical, interest has been growing in the use of pre-appraised and synthesized evidence resources such as systematic reviews, metaanalyses, and evidence-based clinical practice guidelines as aids to clinical decision-making.

Recognition of the need for systematic reviews of healthcare studies continues to grow and is indicated by the number of articles and empirical studies dealing with methods used in reviews, the number of systematic reviews published in healthcare journals, and the rapid growth of the Cochrane Collaboration2-4. The Cochrane Library is a compilation of systematic reviews designed to provide high quality scientific evidence on the effectiveness of various healthcare interventions5. Cochrane reviews serve an invaluable function by summarizing healthcare literature and supporting decisions for more effective clinical practices. The Cochrane Musculoskeletal Group (CMSG) is one of more than 50 entities within the Cochrane Collaboration. Its members are dedicated to preparing and maintaining systematic reviews of musculoskeletal conditions. Many interventions for gout, lupus erythematosus, osteoarthritis, osteoporosis, rheumatoid arthritis, soft tissue conditions, spondyloarthropathy, systemic sclerosis, and vasculitis have been reviewed. Separate review groups study reviews of back and musculoskeletal injuries. We assessed the quality of reviews conducted by the CMSG.

Growing recognition of the key role of reviews in synthesizing and disseminating research results has prompted careful scrutiny of the validity of reviews. In the 1970s and early 1980s, psychologists and social scientists drew attention to the systematic steps needed to minimize bias and random errors in literature reviews6-9. In the late 1980s attention began to focus on the poor scientific quality of healthcare review articles10,11.

An appreciation of the quality of a systematic review is essential to assessment of whether its recommendation of the use or avoidance of an intervention should be followed12,13. Two major areas are assessed in determining the quality of a systematic review. The first is its methodological quality, which is an assessment of how well the systematic review was conducted (literature search, pooling of data, etc.). The second is its quality of reporting, which is an assessment of how well its systematic reviewers have reported their methodology and findings. Separate tools were used to assess each quality area to obtain a comprehensive quality assessment. Assessors were sensitive to the fact that although methodological quality and reporting quality are intrinsically linked, a review may be strong in either area and weak in the other. It was also recognized that poor reporting makes it difficult to assess the methodological quality of a review.

Our objective was to review both the methodological and reporting quality of all published Cochrane systematic musculoskeletal reviews. This review should serve as a baseline, enabling the CMSG to measure improvement in both the methodological and the reporting quality of its reviews over time.

MATERIALS AND METHODS

Choosing the assessment instruments. A review of published scales and checklists was performed to inventory the instruments available for assessing the quality of systematic reviews. Each item of a scale is scored numerically, and individual numerical scores are combined to generate an overall quality score. To be considered a scale, an instrument should be able to measure across a continuum. On the other hand, a checklist provides an estimate of the overall quality of a review by using itemized criteria to assess individual aspects of reviews and facilitate their qualitative comparisons. Individual checklist items do not have numerical scores attached to them.

A literature search was conducted using Medline from January 1966 to February 1999 to identify all published quality assessment instruments. Three independent searches were completed using the following keywords: metaanalysis, review literature, systematic or quantitative or methodologic review, overview, review, information synthesis, integrative research review, guideline, checklist, tool, scoring, scale, clinimetric, quality, critical reading, methodology, and Medline. The "related articles" function was also used14.

Twenty-four assessment instruments were found, including 21 checklists and 3 scales. The 2 instruments selected were a methodological assessment tool that had been rigorously developed by Oxman and Guyatt, known as the Overview Quality Assessment Questionnaire (OQAQ)15,16 quality of reporting tool (Quality of Reporting of Meta-analysis (QUOROM)) developed by consensus17.

The Overview Quality Assessment Questionnaire. The specified purpose of the OQAQ15 is to evaluate the scientific quality (i.e., adherence to scientific principles) of systematic reviews published in medical literature. It is not designed to measure literary quality, importance, relevance, originality, or any other esthetic or philosophical attribute of reviews.

The scale is divided into 9 areas (Table 1). Question 10, the final question of the scale, requires each assessor to rate the overall scientific quality of each report. Possible scores on question 10 range from 1 to 7. This global question is answered based on how well the review scored on the first 9 questions16.

The Quality of Reporting of Meta-analysis (QUOROM) Checklist. The only tool found that was designed to assess the reporting quality of systematic reviews was the Quality of Reporting of Meta-analysis17 statement consisting of a checklist and flow diagram. This checklist of standards for the reporting of metaanalyses describes the preferred way to present the Abstract, Introduction, Methods, Results, and Discussion sections of a report of metaanalysis. The checklist includes 18 items (Table 2). It requires authors of reviews to include a flow diagram that provides information about the number of studies identified, included, and excluded, and the reasons for excluding them.

The OQAQ and the QUOROM statement were found to be the instruments of choice for assessing the quality of systematic reviews.

Literature. The study examined all published CMSG systematic reviews from the Cochrane Database of Systematic Reviews of the Cochrane Library, Issue 4, 20025.

Pilot testing. Two assessors (DF, BS) conducted a pilot study to test the 2 quality assessment tools. As recommended by Glass, et al7, agreement among assessors was maximized through consensus training involving discussion among reviewers. All inconsistencies identified were discussed and resolved during weekly meetings, and assessments were revised after consensus was reached. The pilot exercise was conducted to achieve a high level of interrater agreement. It was decided that an interclass correlation coefficient (ICC) of 0.60 would be the lowest acceptable level of interrater reliability. An ICC was calculated for a subset of questions on the OQAQ, and the ICC results > 0.60 were eventually obtained. However, initially there was less agreement between reviewers on the questions on the OQAQ. Therefore, each item was reviewed and discussed in detail, with agreement being reached on its scoring in cases where there had been disagreement. This pilot was repeated until an ICC > 0.60 was achieved. Final ICC of 0.81 (95% CI 0.74–0.89), 0.83 (0.77–0.90), and 0.88 (0.85–0.93) were obtained for the questions on the OQAQ. Minor refinements were incorporated into the interpretation of the scale, permitting a greater degree of precision when performing assessments. A kappa test was also performed on a subset of the CMSG reviews. The kappa was 0.321 (95% CI 0.136–0.498) prior to discussion and consensus. Additional pilot testing improved the level of agreement slightly 0.420 (95% CI 0.322–0.514).

Data analysis. Data were extracted using prepared forms that included all the items of both quality instruments. All included reviews were assessed using this structured format. Frequencies were provided on all items of both instruments using SPSS 12.0. Percentages were recorded, along with an overall mean score and confidence intervals.

Main study. The quality of all 57 included systematic reviews18-74 was assessed by 2 independent reviewers using the 2 selected tools, with a third reviewer available when needed to reach a consensus (GW). One reviewer (DF) had been involved with the Cochrane Collaboration for two and a half years, and continues to work as a reviewer with the Acute Respiratory Infections Review Group in Brisbane, Australia. The second reviewer (BS) had been involved with the CMSG over the previous 10 years and was a coauthor of some of the reviews.

RESULTS

Table 1. Scores for Cochrane musculoskeletal group (CMSG) systematic reviews from the Overview Quality Assessment Questionnaire (OQAQ).

Table 2. Scores for Cochrane musculoskeletal group (CMSG) systematic reviews from Quality of Reporting of Meta-analysis (QUOROM).

Using the OQAQ, the mean overall scientific quality of the 57 Cochrane Musculoskeletal Reviews evaluated was mean 5.02 (95% CI 3.71–6.32). The scores for each item were of similar quality (Table 1). Of the 10 items making up the scale, the Cochrane musculoskeletal systematic reviews scored poorly on items 2 and 4: 63% of CMSG systematic reviews reported whether the search strategy for the evidence was reasonably comprehensive (item 2) and 53% reported whether study selection bias was avoided (item 4). All CMSG systematic reviews reported the criteria used for deciding which studies to include in the overview (item 3) and combined the findings of the relevant studies appropriately relative to the primary question addressed (item 8). The methods used for combining studies were reported in 95% of the reviews (item 7), while 97% of CMSG reviews reported the criteria used for assessing the validity of included studies (item 5) and drew conclusions that were supported by the data and/or analysis reported in the overview (item 9). Eighty-eight percent of the reviews reported the search methods used to find evidence (item 1). However, only 72% reported whether the validity of all the studies referred to in the text was assessed using appropriate criteria (either in selecting studies for inclusion or in analyzing the studies that are cited) (item 6).

Scores on individual QUOROM items ranged from 5.0% (item 1) to 100% (items 7, 10, 11, 16) (Table 2). Only 5% of CMSG reviews identified the review as a metaanalysis or systematic review of randomized trials in their title (item 1). Almost all CMSG reviews had an abstract with a structured format (item 2) and included objectives (item 3) and data sources (item 4) in the abstract. Items on which they were less likely to report adequately were results (item 6) and selection criteria (item 5) (i.e. population, intervention, outcome, study design, methods for validity assessment, data abstraction, study characteristics, and quantitative data synthesis).

Almost 90% of the reviews described their method of data abstraction (item 12), while no review provided a flow chart for the included and excluded studies (item 15).

More than half the CMSG reviews received a rating of "adequate" on 50% or more of the 10 OQAQ quality items. On the overall quality item (range 0–7) (item 10), the CMSG reviews scored relatively well, with only minor flaws identified [mean 5.02 (95% CI 3.71–6.32)] (Table 1). Of the 18 QUOROM items, the CMSG systematic reviews scored more than 50% on all but 2 of the items (items 1 and 15).

One item (18) was noted as being more difficult to assess and certainly needs further exploration [i.e., the summarization of the key findings, discussion of clinical inferences based on internal and external validity, interpretation of the results in light of the totality of available evidence, description of potential biases in the review process (e.g., publication bias), and suggestion of a future research agenda presented].

Documenting the flow of included and excluded studies and summarizing the results are 2 areas needing improvement in reporting.

DISCUSSION

Assessments made using both quality instruments indicated that the quality of Cochrane musculoskeletal systematic reviews was good, although minor flaws were observed. This is important to users of CMSG reviews, as it provides assurance that their results are relatively reliable. Although their methodological quality and quality of reporting were found to be fair to good, there is room for improvement. For example, it might be feasible to develop specific guidelines for reporting protocols and improve on reporting search results, and documentation of the flow of studies.

The quality of systematic reviews requires examination in order to substantiate the claim that they are the best evidence available to clinicians, health policymakers, and consumers. The use of assessment tools to structure peer review systems can encourage quality improvement in systematic reviews. The Cochrane Collaboration has begun to achieve this objective through continual peer review of protocols, reviews, and updated reviews from the analytical process through to the report. The use of evidence-based criteria such as the QUOROM statement can contribute to the improvement of reporting quality over time by establishing consistent guidelines for the conduct of systematic reviews. At least 2 studies have addressed improvement or lack of improvement over time. A review of 86 English-language metaanalyses assessed every report on 14 items from 6 content areas believed to be critical in the conduct and reporting of metaanalyses. These items included study design, combinability, control of bias, statistical analysis, sensitivity analysis, and problems of applicability. They found that only 24 of the 86 (28%) metaanalyses addressed all 6 content areas75. This survey was updated in 1992 with little change in the results76. A similar study by the authors of this paper showed that the quality of systematic reviews does improve over time, but that the differences on specific items remain variable77. A comparison of Cochrane versus paper-based reviews revealed similar results78.

Inadequate reporting79,80-82 is a significant impediment to the assessment of the quality of systematic reviews! Essential criteria may be met in a given study without being adequately reported in a review. In such a case, a study of high quality may appear to be poor in a review. It may be inaccurate to assume that items not included in a systematic review were missing in the study it reviews, but this is what users of systematic reviews are likely to do. Assessors of a systematic review may invest the time and effort required to obtain additional data directly from the investigators who conducted the systematic review, but ordinary readers cannot reasonably be expected to do so.

The ongoing use of the quality of reporting checklists6,80–82 is to be encouraged, because it will facilitate assessment of study validity by ensuring a high level of congruence between the quality of individual studies and their depiction in systematic reviews. Such an improvement in the quality of reporting of reviewed studies will serve to make systematic reviews more accurate, reliable, persuasive, and useful to those who depend on them.

We tried to address the issue of potential conflict of interest by inviting someone from outside the Musculoskeletal group who would be willing to work with the team on a voluntary basis to carry out this study, a common practice for Cochrane work, especially methods work. One of the main reviewers of the studies is from Brisbane, Australia, and had worked with the Respiratory Airways group for about 3 years. He is very well informed of the format of a Cochrane review. The authors felt that because he was not involved in any of the CMSG reviews he would be considered a nonbiased reviewer. We believe this will negate any potential biases.

Although the 2 instruments used in this study proved useful, the challenges that had to be overcome in applying them clearly demonstrated the need for better measurement instruments. Both instruments proved to be difficult to apply; the main problem encountered was a lack of published guidance on their application. It was only after 3 rounds of pilot testing and resolution of several questions that arose concerning the application of the OQAQ assessment tool that the assessors were ready to apply it. Moreover, while using this scale, reviewers continued to encounter difficulties with its application. Although the QUOROM checklist was rather long and time-consuming to apply, it was accompanied by more detailed directions regarding its use.

One additional item that was not addressed adequately because the measurement tool does not address this well is the inferences based on the results of the systematic reviews. More work is needed in this area.

The statistical analysis revealed that the reliability was poor to fair. There may be several explanations for this. First, the relative magnitude of the kappa value (i.e., the proportion of agreement beyond that expected by chance alone) is difficult to interpret. In the pilot test, the kappa values were categorized and labeled as suggested by Fleiss83, but this classification is purely arbitrary. Kappa coefficient is a popular measure for chance-corrected nominal scale agreement between 2 raters. Future methodological work must include alternative options for calculating agreement among raters, such as exploring Bayesian inferences.

This study found that the overall quality of reports of Cochrane musculoskeletal systematic reviews was generally good, although there was room for improvement. Areas in particular that need special attention include the title and protocol, documentation of the flow of the studies, and inferences made by the conclusions. A recent study reported that the methods for assessment of methodological quality of systematic reviews are still in their infancy and there is substantial room for improvement84.

ACKNOWLEDGMENT

The authors express their appreciation to David Moher for his helpful suggestions and to Ashley Porter and Thelma Hasson for their comments on the manuscript.

REFERENCES

Search PubMed for:

1. Davidoff F, Case K, Fried PW. Evidence-based medicine: why all the fuss? [editorial]. Ann Intern Med 1995;122:727. [MEDLINE]

2. The Cochrane Methodology Register (CMR). The Cochrane Library. Chichester, UK: John Wiley & Sons Ltd; 2004:3.

3. Yusuf S. Obtaining medically meaningful answers from an overview of randomized clinical trials. Stat Med 1987;6:281-94. [MEDLINE]

4. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. CMAJ 1988;138:697-703. [MEDLINE]

5. The Cochrane Library. Chichester, UK: John Wiley & Sons, Ltd.; 2003:4.

6. Light RJ, Pillemer DB. A checklist for evaluating reviews. In: The systematic review. Cambridge, London: Harvard University Press; 1984:160-86.

7. Glass GV, McGraw B, Smith ML. Meta-analysis in social research. Beverly Hills, CA: Sage Publications; 1981.

8. Rosenthal R. Meta-analytic procedures for social research. Rev. ed. Newbury Park, CA: Sage Publications; 1991.

9. Hunter JE, Schmidt FL, Jackson GB. Meta-analysis: Cumulating research findings across studies. In: Cooper HM, Hedges LV, editors. The handbook of research synthesis. New York: Russell Sage Foundation; 1994.

10. Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987;106:485-8. [MEDLINE]

11. National Health Service (NHS) Centre for Reviews and Dissemination (CRD). York, UK: University of York; Y010 5DD.

12. Moher D, Jadad A, Nichol G, Penman M, Tugwell T, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Controlled Clin Trials 1995;16:62-73. [MEDLINE]

13. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998;280:278-80. [MEDLINE]

14. Shea B, Dube C, Moher D. Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. Systematic review in health care meta-analysis in context. London: BMJ Books; 2001:122-39.

15. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991;44:1271-8. [MEDLINE]

16. Oxman AD. Checklists for review articles. BMJ 1994;309:648-51. [MEDLINE]

17. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999;354:1896-900. [MEDLINE]

18. Blumenauer B, Judd M, Wells G, et al. Infliximab for the treatment of rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

19. Bonaiuti D, Shea B, Iovine R, et al. Exercise for preventing and treating osteoporosis in postmenopausal women (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

20. Brosseau L, Casimiro L, Milne S, et al. Deep transverse friction massage for treating tendonitis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

21. Brosseau L, Welch V, Wells G, et al. Low level laser therapy (classes I, II and III) for treating osteoarthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

22. Brosseau L, Welch V, Wells G, et al. Low level laser therapy (classes I, II and III) for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

23. Brosseau L, Casimiro L, Robinson V, et al. Therapeutic ultrasound for treating patellofemoral pain syndrome (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

24. Buchbinder R, Green S, White M, Barnsley L, Smidt N, Assendelft WJJ. Shock wave therapy for lateral elbow pain (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

25. Buchbinder R, Green S, Bell S, Barnsley L, Smidt N, Assendelft WJJ. Surgery for lateral elbow pain (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

26. Busch A, Schachter CL, Peloso PM, Bombardier C. Exercise for treating fibromyalgia syndrome (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

27. Casimiro L, Brosseau L, Milne S, Robinson V, Wells G, Tugwell P. Acupuncture and electroacupuncture for the treatment of RA (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

28. Casimiro L, Brosseau L, Robinson V, et al. Therapeutic ultrasound for the treatment of rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

29. Clark P, Tugwell P, Bennet K, et al. Injectable gold for rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

30. Cranney A, Welch V, Adachi JD, et al. Calcitonin for preventing and treating corticosteroid-induced osteoporosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

31. Cranney A, Welch V, Adachi JD, et al. Etidronate for treating and preventing postmenopausal osteoporosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

32. Criswell LA, Saag KG, Sems KM, et al. Moderate-term, low-dose corticosteroids for rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

33. Dagfinrud H, Hagen K. Physiotherapy interventions for ankylosing spondylitis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

34. Garner S, Fidan D, Frankish R, et al. Celecoxib for rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

35. Garner S, Fidan D, Frankish R, et al. Rofecoxib for rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

36. Gotzsche PC, Johansen HK. Short-term low-dose corticosteroids vs placebo and nonsteroidal anti-inflammatory drugs in rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

37. Green S, Buchbinder R, Barnsley L, et al. Acupuncture for lateral elbow pain (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

38. Green S, Buchbinder R, Glazier R, Forbes A. Interventions for shoulder pain (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

39. Green S, Buchbinder R, Barnsley L, et al. Non-steroidal anti-inflammatory drugs (NSAIDs) for treating lateral elbow pain in adults (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

40. Haguenauer D, Welch V, Shea B, Tugwell P, Wells G. Fluoride for treating postmenopausal osteoporosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

41. Homik J, Cranney A, Shea B, et al. Bisphosphonates for steroid induced osteoporosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

42. Homik J, Suarez-Almazor ME, Shea B, Cranney A, Wells G, Tugwell P. Calcium and vitamin D for corticosteroid-induced osteoporosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

43. Hulme J, Robinson V, de Bie R, Wells G, Judd M, Tugwell P. Electromagnetic fields for the treatment of osteoarthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

44. Jones G, Crotty M, Brooks P. Interventions for treating psoriatic arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

45. Karjalainen K, Malmivaara A, van Tulder M, et al. Multidisciplinary rehabilitation for fibromyalgia and musculoskeletal pain in working adults (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

46. Little CV, Parsons T, Logan S. Herbal therapy for treating osteoarthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

47. Little C, Parsons T. Herbal therapy for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

48. Ortiz Z, Shea B, Suarez-Almazor M, Moher D, Wells G, Tugwell P. Folic acid and folinic acid for reducing side effects in patients receiving methotrexate for rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

49. Osiri M, Welch V, Brosseau L, et al. Transcutaneous electrical nerve stimulation for knee osteoarthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

50. Pelland L, Brosseau L, Casimiro L, Robinson V, Tugwell P, Wells G. Electrical stimulation for the treatment of rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

51. Pope J, Fenlon D, Thompson A, et al. Cyclofenil for Raynaud's phenomenon in progressive systemic sclerosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

52. Pope J, Fenlon D, Thompson A, et al. Iloprost and cisaprost for Raynaud's phenomenon in progressive systemic sclerosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

53. Pope J, Fenlon D, Thompson A, et al. Ketanserin for Raynaud's phenomenon in progressive systemic sclerosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

54. Pope J, Fenlon D, Thompson A, et al. Prazosin for Raynaud's phenomenon in progressive systemic sclerosis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

55. Riemsma RP, Kirwan JR, Taal E, Rasker JJ. Patient education for adults with rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

56. Robinson V, Brosseau L, Casimiro L, et al. Thermotherapy for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

57. Saenz A, Ausejo M, Shea B, Wells G, Welch V, Tugwell P. Pharmacotherapy for Behcet's syndrome (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

58. Struijs PAA, Smidt N, Arola H, van Dijk CN, Buchbinder R, Assendelft WJJ. Orthotic devices for the treatment of tennis elbow (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

59. Suarez-Almazor ME, Belseck E, Shea B, Homik J, Wells G, Tugwell P. Antimalarials for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

60. Suarez-Almazor ME, Spooner C, Belseck E, Shea B. Auranofin versus placebo in rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

61. Suarez-Almazor ME, Spooner C, Belseck E. Azathioprine for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

62. Suarez-Almazor ME, Belseck E, Shea B, Wells G, Tugwell P. Cyclophosphamide for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

63. Suarez-Almazor ME, Belseck E, Shea B, Wells G, Tugwell P. Methotrexate for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

64. Suarez-Almazor ME, Spooner C, Belseck E. Penicillamine for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

65. Suarez-Almazor ME, Belseck E, Shea B, Wells G, Tugwell P. Sulfasalazine for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

66. Takken T, van der Net J, Helders PJM. Methotrexate for treating juvenile idiopathic arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

67. Towheed T, Shea B, Wells G, Hochberg M. Analgesia and non-aspirin, non-steroidal anti-inflammatory drugs for osteoarthritis of the hip (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

68. Towheed TE, Anastassiades TP, Shea B, Houpt J, Welch V, Hochberg MC. Glucosamine therapy for treating osteoarthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

69. Trevisani VFM, Castro AA, Neves Neto JF, Atallah AN. Cyclophosphamide versus methylprednisolone for treating neuropsychiatric involvement in systemic lupus erythematosus (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

70. van den Ende CHM, Vliet Vlieland TPM, Munneke M, Hazes JMW. Dynamic exercise therapy for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

71. Verhagen AP, de Vet HCW, de Bie RA, Kessels AGH, Boers M, Knipschild PG. Balneotherapy for rheumatoid arthritis and osteoarthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

72. Watson MC, Brookes ST, Kirwan JR, Faulkner A. Non-aspirin, non-steroidal anti-inflammatory drugs for treating osteoarthritis of the knee (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

73. Welch V, Brosseau L, Peterson J, Shea B, Tugwell P, Wells G. Therapeutic ultrasound for osteoarthritis of the knee (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

74. Wells G, Haguenauer D, Shea B, Suarez-Almazor ME, Welch VA, Tugwell P. Cyclosporine for treating rheumatoid arthritis (Cochrane Review). In: The Cochrane Library, Issue 4, 2002. Oxford: Update Software; 2002.

75. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N Engl J Med 1987;316:450-5. [MEDLINE]

76. Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mt Sinai J Med 1996;3-4:216-24.

77. Shea B. Assessing the quality of reporting meta-analyses of randomized controlled trials. MSc thesis. Ottawa: University of Ottawa; 1999.

78. Shea B, Moher D, Graham I, Pham B, Tugwell P. A comparison of the quality of reporting of Cochrane reviews and systematic reviews published in paper-based journals. Eval Health Prof 2002;25:116-29. [MEDLINE]

79. The Standards of Reporting Trials Group (SORT). A proposal for structured reporting of randomized controlled trials. JAMA 1994;272:1926-31. [MEDLINE]

80. Schulz KF, Chalmers I, Grimes DA, Altman DG. Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA 1994;272:125-8. [MEDLINE]

81. Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature. Call for comments on a proposal to improve reporting of clinical trials in the biomedical literature. Ann Intern Med 1994;121:894-5. [MEDLINE]

82. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637-9. [MEDLINE]

83. Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York: John Wiley & Sons; 1981.

84. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 2005;330:1053. Epub 2005 Apr 7. [MEDLINE]



Return to January 2006 Table of Contents



© 2006. The Journal of Rheumatology Publishing Company Limited.
All rights reserved.