- Re: Highly significant development, and AAPS link very useful for evidence as well
Posted by walter
on June 17, 2020, 12:10 pm, in reply to "Re: Highly significant development, and AAPS link very useful for evidence as well "
|
"If the population is not randomly assorted study size is important irrespective of statistical significance. If a study is underpowered it may not be showing the expected result despite significance. It may just be revealing a bias." Are biases more likely in small studies? I'm not sure they are. Risch's point was that when a study is small but records an outcome below the threshold of probability for chance as the explanation, statistical significance is not the issue. For critics to point to the small size is misguided as the statistical significance has already been determined. Reaching statistical significance means that the null hypothesis (ie that there is no true effect, or the findings were due to chance) is rejected according to the mechanics of the test. Other explanations for the 'unlikely' outcome can and should then be looked at, but chance isn't one of them. Just as it wouldn't be for a larger study. Bias sources are valid explanations but they should not be automatically assumed from the study size. In the absence of proposed specific sources of bias the result should be accepted as evidence on the basis of the statistical significance. "Study powering is confusing and I have had to sit down and read about it when writing studies in the past. I'm surprised this hasn't been picked up in peer review." Yep. It seems the statistical power calculation (usually conspicuous by absence) is regarded as an irksome chore which often would reveal a significant limitation of the study to the world. That's because it is! But really, the power to detect a specific effect size should be on display beside a negative result, or listed as a notable absentee. When the power of the test is absent a negative outcome is almost sure to be misinterpreted when the study had low power. Peer reviewers will not usually be statisticians but they could still require to note the fact the statistical power of the test was not on display while the study formally clocked a 'negative' result. In your coins example, the p-value still determines the significance and is a function of the study size. If there is a true effect, the larger study size makes finding it more likely by reducing the p-value obtained. If the p-value indicates statistical significance I don't think the study size reduces it. Other explanations eg sources of bias need then to be considered; it is possible a single source of bias might have caused the outcome, and in a small study the scales may be more readily tipped but as against that the threshold for statistical significance is harder to reach. But critics start with the swipe that the study was small and wrongly deny statistical significance. That's what Risch is saying is wrong. These critics are going against the science they claim to be following. |