<- file stat .html -> ******************* Designs and analyses ******************

Observational studies. Kademan. O'Rourke.

=====================Ed Kademan, 19 Dec 1995========ssc From: kademan@stat.wisc.edu (Ed Kademan) Subject: self-selection and non-random samples Message-ID: <KADEMAN.95Dec19075216@mars.stat.wisc.edu> About two weeks ago I posted a message to this group asking about self-selection and non-random samples. What follows is my original request and the references that people were kind enough to send in. At this time I can't do a very good job of annotating the bibliography. The sources by Cochran and DB Rubin look fundamental and important. Also, the epidemiology books came highly recommended, and the opening chapter of Rothman's text has a very thoughtful discussion of causality. Most of the references came courtesy of: David Hitchin <ccfg4@central.susx.ac.uk>, Pilon Paul <pilon@server.uwindsor.ca>, A Staines <lrf6as@gps.leeds.ac.uk>, "Keith O'Rourke" <orourke@utstat.utstat.toronto.edu>, Winson Taam <taam@Oakland.edu> >>>>>>>>>> >I am interested in learning about the different ways of dealing with >bias in nonrandom samples. In experimental settings many classical >techniques assume that the researcher assigns subjects to control and >treatment groups at random so that---in the large---the two groups are >identical. In many observational studies however the subjects >themselves decide which group to join, leaving open the possibility >that the propensities that drive them might confound the treatment >effects. > @book{coch ,author = "William Gemmell Cochran" ,title = "Planning and Analysis of Observational Studies" ,year = 1983 ,publisher = "Wiley" ,address = "New York" } @book{bres ,author = "Norman E. Breslow" ,title = "Statistical Methods in Cancer Research" ,year = 1980 ,publisher = "Lyon: International Agency for Research on Cancer" ,address = } @book{miett ,author = "Olli S. Miettinen" ,title = "Theoretical Epidemiology" ,year = 1985 ,publisher = "Wiley" ,address = "New York" } @book{roth ,author = "Kenneth J. Rothman" ,title = "Modern Epidemiology" ,year = 1986 ,publisher = "Little, Brown" ,address = "Boston" } @book{seber ,author = "G. A. F. Seber" ,title = "The Estimation of Animal Abundance and Related Parameters" ,year = 1982 ,publisher = "C. Griffin and Co., Ltd." ,address = "London" } @book{mmt ,author = "Bryan F. J. Manly and Lyman L. McDonald and Dana L. Thomas" ,title = "Resource Selection by Animals : Statistical Design and Analysis for Field Studies" ,year = 1993 ,publisher = "Chapman and Hall" ,address = "London; New York" } @book{train1 ,author = "Kenneth Train" ,title = "Qualitative Choice Analysis" ,year = 1986 ,publisher = "The MIT Press" ,address = "Cambridge, Massachusetts" } @book{wainer ,editor = "Howard Wainer" ,title = "Drawing Inferences from Self-Selected Samples" ,year = 1986 ,publisher = "Springer-Verlag" ,address = "New York, New York" } @book{campbell ,author = "D. T. Campbell and J. C. Stanley" ,title = "Experimental and Quasi-Experimental Designs for Research" ,year = 1966 ,publisher = "Rand McNally" } @book{cook ,author = "T. D. Cook and D. T. Campbell" ,title = "Quasi-Experimentation" ,year = 1979 ,publisher = "Rand McNally" } @book{achen ,author = "C. H. Achen" ,title = "The Statistical Analysis of Quasi-Experiments" ,year = 1986 ,publisher = "University of California Press" } @article{roru ,author = "Paul R. Rosenbaum and Donald B. Rubin" ,title = "The central role of the propensity score in observational studies for causal effects" ,journal = "Biometrika" ,year = 1983 ,volume = 70 ,number = 1 ,pages = "41--55" } @article{heck1 ,author = "James J. Heckman" ,title = "Dummy endogenous variables in a simultaneous equation system" ,journal = "Econometrica" ,year = 1978 ,volume = 46 ,number = 6 ,pages = "931--959" } @article{heck2 ,author = "James J. Heckman" ,title = "Sample selection bias as a specification error" ,journal = "Econometrica" ,year = 1979 ,volume = 47 ,number = 1 ,pages = "153--161" ,month = jan } @article{heck3 ,author = "James J. Heckman" ,title = "Causal inference and nonrandom samples" ,journal = "Journal of Educational Statistics" ,year = 1989 ,volume = 14 ,pages = "159--168" } @article{dubmac ,author = "Jeffrey A. Dubin and Daniel L. McFadden" ,title = "An econometric analysis of residential electric appliance holdings and consumption" ,journal = "Econometrica" ,year = 1984 ,volume = 52 ,number = 2 ,pages = "345--362" ,month = mar } @article{train2 ,author = "Kenneth E. Train" ,title = "Estimation of net savings from energy conservation programs" ,journal = "Energy" ,year = 1994 ,volume = 19 ,number = 4 ,pages = "423--441" ,month = apr } @article{croft ,author = "Brennan P. Croft" ,title = "Interpreting the results of observational research: chance is not such a fine thing" ,journal = "Brit. Med. Journal" ,year = 1994 ,volume = 309 ,pages = "727--730" } @article{holland ,author = "P. Holland" ,title = "Statistics and causal inference" ,journal = "Journal of the American Statistical Association" ,year = 1986 ,volume = 81 ,number = 396 ,pages = "945--970" } @article{rubin1 ,author = "Donald B. Rubin" ,title = "Practical implications of modes of statistical inference for causal effects and the critical role of the assignment mechanism" ,journal = "Biometrics" ,volume = 47 ,year = 1991 ,pages = "1213--1234" } @article{lavori ,author = "P. W. Lovori and R. Dawson and R. Shera" ,title = "A multiple imputation strategy for clinical trials with truncation of patient data" ,journal = "Stat. in Med." ,volume = 14 ,year = 1995 ,pages = "1913--1925" } =================Keith O'Rourke, 15 Mar 1996========ssc,(sas) From: orourke@utstat.toronto.edu (Keith O'Rourke) Message-ID: <DoBAtq.MvB@utstat.toronto.edu> <<observational studies>> You may wish to look at Matthews Altman Cambel Royston. Analysis of serial measurements in medical research. Br. Med J 1990; 300:230-235 Brennan Croft. Interpreting the results of observational research .. Br. Med J 1994; 309:727-30 Rosenbaum Rubin The central role of the propensity score in observational studies for causal effects Biometrika 1983 70 No 1 41-55 The first two can be given out to clients :-) * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Bonferroni, Holm correction

=========================Michael Babyak, 30 May 1996======ssc From: mbabyak@acpub.duke.edu (Michael Babyak) Message-ID: <4okcjp$1sc@newsgate.duke.edu> I think I may have mentioned it before on this list, but Benjamini and Hochberg have recently introduced a technique for controlling the "False Discovery Rate" which is more powerful than the Bonferroni or even the sequential approach proposed earlier by Hochberg or Holm. Most importantly, they remind us that the choice of techniques for controlling the error rate should be based on the substantive consequence(s) of committing those errors. The full cite is: Benjamini, Y. & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J Royal Statistical Society, Series B, 57 (1), 289-300. Hans-Peter Piepho (piepho@WIZ.UNI-KASSEL.DE) wrote: : >Check out the May 1996, Vol. 86, No. 5, American Journal of Public : >Health article by Aickin, et al, entitled, "Adjusting for Multiple : >Testing When Reporting Research Results: The Bonferroni vs. Holm : >Methods", where the authors make an argument for using the Holm method : >instead of the Bonferroni procedure. What do people think? : While Holm and Bonferroni both control family-wise Type I error in the : strong sense (Hochberg and Tamhane 1987), Holm tends to be more powerful. : Also see : SP Wright 1992 Adjusted P-values for simultaneous inference. Biometrics 48, * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

... compute eta-squared?

=====================David Nichols, 30 Nov 1994========ssc From: nichols@spss.com (David Nichols) Subject: Re: Magnitude of effect in rptd measures Message-ID: <D037w5.I27@spss.com> In article <1994Nov29.150157.35671@cobra.uni.edu>, <gilpin@cobra.uni.edu> wrote: >Can anyone tell me formulae for computing eta^2 and omega^2 in a 1-way repeated >measures ANOVA? All the discussions I've found address only independent >designs; are these the same in this regard? In SPSS, eta^2 and omega^2 are defined for the averaged univariate F-test as: eta^2= SSH/(SSH+SSE) omega^2 = (SSH-DFH*MSE)/(SST+MSE) where: SSH is sums of squares for hypothesis SSE is sums of squares for error SST is the total mean corrected sums of squares MSE is mean squared error DFH is hypothesis degrees of freedom They are printed only when you use SEQUENTIAL sums of squares (which gives the same results as UNIQUE given the inherent balance of the design). If UNIQUE sums of squares are requested (this is the default) the partial eta^2 is given, which has the same definition as that given above. Note that in more complicated designs the definition given above is that for a partial eta^2, which is a positively biased but consistent effect size measure. There are two other kinds of tests one might look at: individual univariate tests and the overall multivariate test. The eta^2 values given for the individual univariate tests follow the same definition as above, while the multivariate effect size measure is the same as the Pillai statistic: lambda/(1+lambda), where lambda is the nonzero characteristic root (eigenvalue) of the characteristic equation involving the SSH and SSE matrices (there will be one nonzero root as long as there is at least one contrast SSH that is nonzero; otherwise, the potentially nonzero eigenvalue will be zero). These definitions are again partial eta^2 ones used throughout the procedure, which reduce to "total" or regular eta^2 in a simple design like this. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

... test for rho=1 ?

=====================Rich Ulrich, 10 Dec 1996======ssc From: wpilib+@pitt.edu (Richard F Ulrich) Subject: Re: Is there a test for H0:Pearson-Rho=1? Message-ID: <58juof$enc@usenet.srv.cis.pitt.edu> Renaud Langis (nakhob@mat.ulaval.ca) wrote: : Is there a test for H0:Pearson-Rho=1? : I found tests for Rho=0 and Rho=Rho0 with Rho0<1. Can't gat to find one for : testing Rho=1. -- To borrow an idea from another thread on hypotheses: If your null hypothesis, in terms of marbles in the jar, is that *every* marble is green with white dots, then you must reject the hypothesis if your draw includes any marble that is otherwise. If your idea is about testing for perfect correlation, while taking into account the prospect that measures are imperfect, then you need to test for an imperfect, non-zero correlation, and make strong arguments about the expected reliability of each of your measures. Tests of correlations against a certain value are usually done by using Fisher's z transformation; see texts about comparing correlations. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Why is it called Jackknife?

=====================Rich Ulrich, 15 May 1996=======sse From: wpilib+@pitt.edu (Richard F Ulrich) Subject: Re: Why is it called "jackknife" ? Message-ID: <4ncql7$lp0@usenet.srv.cis.pitt.edu> : > : > Can anyone explain to me where the "jackknife" in jackknife estimates : > comes from ? Jackknife estimates are usually discussed along with : > bootstrap estimates, but while I can understand why you call it : > "Bootstrap", "Jackknife" does not seem so obvious to me. : A *very good* tool for many different things, the *perfect* tool : for almost nothing, kind of like a . . . That is what I remember from the original explanation, too. But jack-knives are the pocket knives with multiple folding blades. -- there was also a metaphorical "unfolding" in the computation of the original jack-knifed estimates. It was NOT the same as simple leave-one-out replication, even though the latter may be sometimes mis-labeled as jack-knifing today. Here is a statement from Efrom's "Jackknife, Bootstrap, and other Resampling Plans" (1982): "This form of cross validation looks like the jackknife in the sense that data points are omitted one at a time. However, there is no obvious statistic theta-hat being jackknifed, and any deeper connection between the two data ideas has been firmly denied in the literature." (page 49). ===================Rich Ulrich, 15 May 1996======sse From: wpilib+@pitt.edu (Richard F Ulrich) Subject: Re: Why is it called "jackknife" ? Message-ID: <4nde3u$qeq@usenet.srv.cis.pitt.edu> Sylvia J. Hysong (shysong@ruf.rice.edu) wrote: (citing my post) : > : > But jack-knives are the pocket knives with multiple folding blades. : > -- there was also a metaphorical "unfolding" in the computation : > of the original jack-knifed estimates. It was NOT the same as : > simple leave-one-out replication, even though the latter (I think) : > is sometimes mis-labeled as jack-knifing today. : > : I'm glad you brought this up, because that was my notion of what : jack-knifing was. Can you please tell me what the difference is between : leave-one-out replication and jack-knifing? It is just a matter of a step of extra computation, which takes away some bias. Right now, I am looking in Sage Monograph #95, Bootstrapping, Mooney CZ and Duval RD, pg. 23-24. It includes as one reference: Miller, 1974, "The jackknife, a review", Biometrika 61:1-15; I think that is the usual, important, modern reference.. Jackknifing can be applied to *blocks* of data to be left out, which people did (more often) back when computing was expensive. Adapting the Mooney formula to the leave-one-out MODEL, then, for each (i) in sample, size N, theta-tilde(i) equals N times (whole sample theta-hat) minus (N-1) times (whole-minus-case-i theta-hat) . -- See how it is FOLDED? ... N() minus (N-1)() ... "The most common use of the jackknife has been in estimating the bias of theta-hat. Theta-tilde is second-order unbiased, and a simple estimate of the first-order bias can be developed by subtraction..." When I read the BMDP (1988) manual, under `stepwise discriminant function' which offers some `jackknifing', I do not see mention of any extra step beyond leave-one-out, and one mention seems to imply it is purely leave-one-out; but what they are doing is the CLASSIFICATION TABLE, so I am not sure whether there is a distinction that makes a difference. The manual gives a 1968 Technometrics (10:1-11) reference that I have not yet looked up; and it does say that the purpose of what they do is to reduce bias. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

... CI for relative risk?

======================Charles C. Berry, 18 Jun 1996=======ssc CI for RR From: cberry@tajo.edu (Charles C. Berry) Subject: Re: Small sample, relative risk Message-ID: <4q76ha$7se@news1.ucsd.edu> E. Caldwell (ellenc@u.washington.edu) wrote: : I'd appreciate help--formula/algorithm/reference--with computing a : confidence interval for a small sample relative risk problem. Some of the approximations reviewed in: Gart, J. J. and Nam, J. (1988) ``Approximate Interval Estimation of the Ratio of Binomial Parameters: A review and Corrections for Skewness,'' {\it Biometrics}, 44, 323-338. perform well enough for practical work in small data sets. If you are interested in the small sample theory, see: Santner, T. J. and Snell, M. K. (1980) ''Small Sample Confidence Intervals for $p_1 - p_2$ and $p_1/p_2$ in 2 $\times$ 2 Contingency Tables,'' {\it Journal of the American Statistical Association}, 75, 386-394. For what it is worth, a method proposed by Howe (and also by Graybill and also by others) can be used to produce quite accurate intervals for log(p_1) - log(p_2) and thence p_1/p_2 by starting with a table of exact binomial intervals. (I've checked a number of setups with quite small n's). See: Howe, W.G. (1974) Approximate confidence intervals for the mean of $X + Y$ where $X$ and $Y$ are two tabled independent random variables. JASA 69:789--794. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

... compute CI on annual rates?

=======================Rich Ulrich, 09 May 1996=======ssc CI on annual rates From: wpilib+@pitt.edu (Richard F Ulrich) Subject: Re: CI's for incidence Message-ID: <4mtdgg$9tt@usenet.srv.cis.pitt.edu> ZHANGYA@QUCDN.QueensU.CA wrote: : In calculating confidence intervals for annual incidence rates, does one : treat the incidence as a proportion even though it is a rate? Hmm... that raises the question, "Who is the ONE?" What annual incidence rates are you referring to? The rates that I most often notice are news items about deaths, illnesses, crimes - and they are NEVER accompanied by any confidence intervals. When I look at them, I usually compute a crude C.I. which assumes (as you mention) that the rate is a proportion - that is: I assume the events are independent, and Poisson, and I look at the square root of the number of incidents. If it has not changed by plus or minus 1, then there is not 2 SDs of change. This often shows me that some politician or journalist's purported increase/decrease is apt to be random fluctuation. For something like burglaries in a community, the rate or number of incidents in a year is a multiple of the number of burglars out there, so the events are not independent... my crude C.I. should estimate the number of burglars, not the number of burglaries (which will thus be more variable than one would expect from Poisson). When medical studies are reporting what you think are incident `rates', you may discover, if you read closely, that careful researchers have counted each individual only ONCE, even if there could be two+ events per person; so the rate is legitimately a proportion. (When there are FEW, it is hard to accommodate those reports of `second recurrence' except by explicitly reporting them as such; and saying they were excluded.) * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

... compare variables across time?

===========================Rich Ulrich, 12 Nov 1996======ssc,sse From: wpilib+@pitt.edu (Richard F Ulrich) Subject: Re: Compare two independent grps with variable varies with TIME ? Message-ID: <56akbi$hu1@usenet.srv.cis.pitt.edu> sunnylo@hknet.com wrote: : I want to compare the differences between a treatment and placebo group : of the increase of blood flow after an intervention. The blood flow : measurement will be taken periodically from -10, 0, 10, 20, 30, 40 and : 50 minutes after intervention. : How could I compare the whole change (varies with time) with another : independent group ? For power in your analysis, you might focus on the period where you expect there to be MOST change (for example, 30? ) and compare the CHANGE, where the baseline is the average of Time= (-10,0). If you expect a LINEAR change by time, then the repeated measures test could be the best (that BMDP provides very conveniently), where you look at the interaction of the group-by-linear for the test of hypothesis. Generically speaking, if you were really at a loss and knew nothing about the subject area, then you would have to report on tests that fit your problem less precisely. In that case, the Repeated Measures ANOVA does test the overall difference of Groups, and Groups-by-time tests the difference in patterns across time. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

... analyze a survey?"

========================Clay Helberg, 12 May 1995======ssc From: helberg@maddog (Clay Helberg) Subject: Re: Help requested with Social Sci.Stats Message-ID: <3ovsfs$vju@news.doit.wisc.edu> K Hardie (kjmh@festival.ed.ac.uk) wrote: : I'm a secondyear phd student in political linguistics. I;ve since : january tried to get help from anyone here that i could think of with my : analysis. I've finished collecting my data which looks into the : attitudes towards the languages of Scotland according to people's ideas : about the constitutional question in Scotland. The 3 groups i'm dealing : with are: : 1. in favour of independence : 2. in favour of devolution : 3. in favour of the union as it is. : My questionnaire is set up in the form of 5 point Likert Scales ranging : from Strongly Disagree to Strongly Agree. : I have done Chisquare, Ctau and Gamma. Out of the 30 questions I only : found 6 to have a correlation and a significant chisquare result with my : 3 groups. Well, there's good news and bad news. The good news is that there is a much more coherent way of handling this sort of data. The first thing you want to do is reduce the number of dependent variables--I assume you aren't *really* interested in the individual responses to all thirty questions, but rather that you hope to get at some underlying factor or factors which those questions tap into. So, you will probably want to do a "factor analysis", which will use the relationships between responses within individuals to discover which questions tend to "stick together"--i.e. which questions tap into the same factor. Technically, there will be as many factors as there are items, but practically, if your questionnaire has any reasonable structure to it at all, there will be a small number (maybe 2-4) of factors which account for nearly all the variance, and you can more or less ignore the others. By studying which questions cluster together to form a "factor", you can usually make a pretty good guess about what that factor might be. Now that you have reduced your dependent variable space to a manageable number of dimensions, you should be able to develop a MANOVA (multivariate analysis of variance) model, which will find differences based on group membership (i.e. "indepenents" vs. "devolutionists" vs. "status-quo-ists") for the factors (which you identified earlier) as a set. I think this will tell you what you want to know. Now, for the bad news.... These techniques are quite complex, and require a good understanding of multivariate statistics and probability. Implementing these strategies will require a large investment in time and energy to acquire the necessary skills. On the other hand, these skills would be considered by many to be indispensable for a Ph.D. in social science. They will serve you well in your research career. Not only will they give you new tools for analyzing data, they will also prove valuable as tools for *conceptualizing* complex phenomena and designing informative research studies. You will probably want to take some advanced statistics classes, and/or hook up with someone who has experience with these techniques. (If you're having trouble finding such a person in your department, try the sociology dept.--these are well-established methods in sociology.) In the meantime, if you want to get your feet wet with multivariate statistics, pick up a copy of Dillon & Goldstein, _Multivariate Analysis: Methods and Applications_. You will probably find the first three chapters to be the most useful, and chapter 11 is where you will find info on MANOVA. Pay special attention to the first chapter, and Appendix I, which introduce concepts and notation you will need as a foundation to learning multivariate statistical techniques. Good luck in your quest! --Clay * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

How good are meta/mega analyses?

======================Rich Ulrich, 09 Aug 1996======ssm From: wpilib+@pitt.edu (Richard F Ulrich) Subject: Re: Help: Mega Analysis Message-ID: <4ug9ll$dn7@usenet.srv.cis.pitt.edu> chris mahmood (hbpsy022@csun.edu) wrote: : I kind of like 'Mega-analysis' better! Yes, meta-analysis is where you : take 20-30 crappy studies and combine them into one big crappy study (My : only experience is in psychology). One of the seminal books on the topic : is by Rosenthal, i think. : -ckm Come now, don't you think you are overstating, just a little, how bad they are? Sometimes they will combine 20-30 crappy studies along with 2 or 3 GOOD studies, thereby burying the good studies in crap -- this seems to be the way it works in research related to psychiatry, which includes some psychology. I think the fellow is named Richard Peto who has been the most reasonable advocate of Meta-studies, and of Mega-studies (i.e., collaborative ventures with 10-30 thousand subjects or more). One thing that that he urges is: the studies to be combined should have simple, unambiguous outcomes, such as "Death within a week", etc., which he as used to describe Emergency Room interventions in the treatment of stroke, etc. * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Document by Rich Ulrich. E-mail to wpilib+@pitt.edu

FAQ top.

Ulrich home page.

Ulrich FAQ. http://www.pitt.edu/~wpilib/stats99.html