<- file 96metaa.html ->

Meta analysis. Vokey. Lumley.

=====================John Vokey, 30 Sep 1995========sse Message-ID: <009972D3.B90F7E1F.230@hg.uleth.ca> From: vokey@hg.uleth.ca Subject: Meta-analysis, formal and informal I suspect many of us have some sympathy for Professor Ryan's position regarding meta-anlysis and its less "formal" analogues, but: 1. There is no evidence that formal meta-analyses are or have been any better at accomplishing the goals of less formal reviews. But then, it also is not clear that the goals are necessarily equivalent, commensurate, or even well-specified enough to make such judgments. 2. In psychology and, I suspect, most other social, life, and bio-sciences, the results per se of experiments are *NOT* particularly important or useful *except in so far as they are seen to be logical entailments of some theoretical perspective*, and even the experiments that produce them are only connected with the theories at a few points. Consequently, it is not clear in many if not most cases just what the meta-level is that would be served by the simple (indeed, simplistic) meta-analytic arithmetic. 3. Despite the tacit (and sometimes vocal) hopes of some (those of the "Bayesian" ilk, mostly it seems) it is unlikely that induction will (or can) ever be reduced to an algorithmic process -- and if it is, I doubt it will accomplish much in the way of what most of us judge to be "good science". 4. Thus, it is not at all clear that formal meta-analyses *could* save us from the cases like the one Professor Ryan describes -- or that anything *should*. John R. Vokey <vokey@hg.uleth.ca> ===============Thomas Lumley, 11 Apr 1996=========ssc, ... From: Thomas Lumley <thomas@biostat.washington.edu> Newsgroups: sci.stat.consult,bit.listserv.medlib-l,bit.listserv.asis-l Subject: Re: Meta-analysis literature searches Message-ID: <316DA864.41C67EA6@biostat.washington.edu> > ==> Is a published meta-analysis paper to be considered adequate if it > gives an inadequate explanation of the literature search (the > resources, the techniques, strategies, dates & files & details galore) > ...? You would probably find more discussion of these points on the evidence-based-health mailing list (at mailbase in the UK). There are guidelines for "systematic reviews" produced by the Cochrane Collaboration and various other people which do point out how important the initial selection of papers is. In particular, restricting an analysis to just those papers found in a MEDLINE search is not a highly regarded approach: MEDLINE searches, even conducted by experts, miss an enormous number of studies. The Cochrane Collaboration has a Web site at http://hiru.mcmaster.ca/cochrane/default.html In regard to your questions about the need to report search strategies and so on, here is a quote from the Cochrane Collaboration handbook. ------------ As with any research study, a number of methodological decisions must be made when undertaking a systematic review, and there are potential threats to validity associated with each decision. These include: how studies are identified and selected for inclusion, how their quality (validity) is assessed, how much weight each one is given, and how their results are combined and interpreted. If reviewers do not say how these decisions were made, there is no way of knowing how much faith to put in their conclusions. ------------- There is a lot more similar information and recommendations in the handbook, which is available online at the Web site. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Document by Rich Ulrich. E-mail to wpilib+@pitt.edu

FAQ top.

Ulrich home page.

Ulrich FAQ. http://www.pitt.edu/~wpilib/stats99.html