Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning
Short Title: 
Meta-analysis under the spotlight

OBJECTIVE: Because the results of a meta-analysis are used to formulate the highest level recommendation in clinical practice guidelines, clinicians should be mindful of problems inherent in this technique. Rather than reviewing meta-analysis in abstract, general terms, we believe readers can gain a more concrete understanding of the problems through a detailed examination of one meta-analysis. The meta-analysis on which we focus is that conducted by an American College of Chest Physicians/American Association for Respiratory Care/American College of Critical Care Medicine Task Force on ventilator weaning. DATA SOURCE: Two authors extracted data from all studies included in the Task Force's meta-analysis. DATA SYNTHESIS AND OVERVIEW: The major obstacle to reliable internal validity and, thus, reliable external validity (generalizability) in biological research is systematic error, not random error. If systematic errors are present, averaging (as with a meta-analysis) does not decrease them--instead, it reinforces them, producing artifact. The Task Force's meta-analysis commits several examples of the three main types of systematic error: selection bias (test-referral bias, spectrum bias), misclassification bias (categorizing reintubation as weaning failure, etc.), and confounding (pressure support treated as unassisted breathing). Several additional interpretative errors are present. CONCLUSIONS: An increase in study size, as achieved through the pooling of data in a meta-analysis, is mistakenly thought to increase external validity. On the contrary, combining heterogeneous studies poses considerable risk of systematic error, which impairs internal validity and, thus, external validity. The strength of recommendations in clinical practice guidelines is based on a misperception of the relative importance of systematic vs. random error in science.

Tobin, Martin J.
Jubran, Amal
Item Type: 
Journal Article
Publication Title: 
Critical Care Medicine
Journal Abbreviation: 
Crit. Care Med.
Publication Date: 
Publication Year: 
Library Catalog: 
PMID: 18007269

Turabian/Chicago Citation

Martin J. Tobin and Amal Jubran. 2008-01. "Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning." Critical Care Medicine 36: 1: 7-Jan. 10.1097/01.CCM.0000297883.04634.11.

Wikipedia Citation

<ref> {{Cite journal | doi = 10.1097/01.CCM.0000297883.04634.11 | issn = 1530-0293 | volume = 36 | pages = 7-Jan | last = Tobin | first = Martin J. | coauthors = Jubran, Amal | title = Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning | journal = Critical Care Medicine | date = 2008-01 | pmid = | pmc = }} </ref>