The p-value is the type I error probability, to reject a correct H0, under the assumption that H0 is correct.
When the p-value is small enough, less than α, the risk of type I error is low, and we reject the H0.
In the following two-tailed example the p-value is greater than α, hence we can not reject the null assumption.
Please click the "P-Value" radio button.
Every student learns how important it is to get a significant result, but the p-value is only one component in a package.
The "power" and "effect size" are not less important.
All the bellow examples are fake, the made-up data was created only for demonstration. The used significance level is 0.05.
Several researchers looked for high blood pressure treatment.
|Standardized effect size||Small, 0.26|
|Standardized effect size||large, 1.1|
The garlic treatment is significant, but the effect size is very small, it isn't really useful treatment.
The daily exercise treatment is not Significant, but there are several reasons to prefer this treatment.
1. The test power is weak, hence the test may not have enough power to reject an incorrect null assumption. We can't know if the result is significant.
2. The p-value 0.07 is larger than the significance level 0.05, but 0.05 is not a "holy" number. You may choose a different significance level like 0.1 or 0.01 if you decided before the experiment that this is the appropriate risk. Usually, you don't have the freedom to choose the α when all the researchers in your field use the same value.
3. The effect size is large, so if H0 is correct, there is a small probability, 0.07, that the treatment doesn't help, but if it helps, there is a high chance that it is a meaningful reduction.
The correct solution is to repeat the daily exercise research with larger sample size.
|Standardized effect size||Small, 0.26||Large, 1.1|