Test Power

Priori power

The probability that the test will reject an incorrect H0, for a predefined effect size

While the significance level (α) represents the probability of rejecting a correct H0 , it doesn't mean that H0 is correct. It only indicates that the probability of rejecting a correct H0 will be bigger than α.
This a risk we are not willing to take. Usually α is 0.05 meaning there is a 5% chance of the above error (Type I Error).

We plan the study and determine the required priori test power before running the study.
The statistical power represents the probability of the test to reject an incorrect H0, but it depends on the H1 value. In some cases we don't know the H1 value as it just proposes that H0 is incorrect. In such cases we need to define the “expected effect” that the test is expected to discover. The smaller the expected effect, the weaker the statistical power. Let d be “absolute effect” needed to be discovered. .
Commonly, a priori power of 0.8 is being used.

statistical power = 1 - β .

β is the probability of a type II error, meaning the test won't reject an incorrect H0.

Observed power (post-hoc power)

Be careful not to confuse the priori power with the observed power, they are different measurements.
The priori power is calculated before collecting data, while the observed power is calculated after collecting data.
The observed power represents the probability of the test to reject H0 based on the observed effect. It is directly correlated to the significance level (α). When accepting H0, the observed effect will be small and the observed power will be weak, on the contrary, when rejecting H0, the observed effect will be big and the observed power will be strong.

z-test power

In this tool I prefer to treat the expect effect as a proportion from the H0 value

Two tailed

H0: μ = μ0
H1: μ ≠ μ0

The test need to reject H0 when μ = μ1 = μ0 ± d
Since z is symmetrical it get the same result when calculating for μ0 + d or μ0 - d

Calculate reject area

$$p(z < Z_1) = \alpha/2 \:,\: p(z < Z_2) = 1-\alpha/2$$ H0 The test's accepted range for x is: [R1 , R2]. Reject H0 when x< R1 or x > R2

  • $$p(z < \frac { R_1 - \mu_0} {\frac {\sigma }{\sqrt{n}}}) = \alpha/2 \:\rightarrow\: R_1= \mu_0 + Z_{\alpha/2}* \frac {\sigma }{\sqrt{n}}$$
  • $$p(z < \frac { R_2 - \mu_0} {\frac {\sigma }{\sqrt{n}}}) = 1-\alpha/2 \:\rightarrow\: R_2= \mu_0 + Z_{1-\alpha/2}* \frac {\sigma }{\sqrt{n}}$$
  • Now we can calculate the probability to reject H0 when we know the mean is μ1 instead of μ0, $$\overline{X}\:distribution\:N(\mu_1, \frac {\sigma }{\sqrt{n}}) $$
  • $$power = p(\overline{x} < R_1 |\mu_=\mu_1 ) + p(\overline{x} > R_2 |\mu_=\mu_1) $$
  • $$power = p(z < \frac { R_1 - \mu_1} {\frac {\sigma }{\sqrt{n}}}) + 1 - p(z < \frac { R_2 - \mu_1} {\frac {\sigma }{\sqrt{n}}})$$

Normal distribution two tailed Normal distribution two tailed
β - type II error
power

Left tail

Since z is symmetrical, for the same d, left tail test has the same power as right tail test. H0: μ ≥ μ0
H1: μ < μ0

The test need to reject H0 when μ = μ1 = μ0 - d

Calculate reject area

$$p(z < Z_1) = \alpha$$ H0 The test's accepted range for x is: [R1 , ∞ ]. Reject H0 when x < R1

Normal distribution two tailed Normal distribution two tailed
β - type II error
power

Right tail

H0: μ ≤ μ0
H1: μ > μ0

The test need to reject H0 when μ = μ1 = μ0 + d

Calculate reject area

$$p(z < Z_2) = 1-\alpha$$ H0 The test's accepted range for x is: [-∞ , R2]. We reject H0 when x > R2

Normal distribution two tailed Normal distribution two tailed
β - type II error
power

Effect size

A measurement of the size of a statistical phenomenon, for example a mean difference, correlation, etc. there are different ways to measure the effect size.

Effect size (unstandardized)

the pure effect as it, for example if needs to identify change of 1 mm in the size of mechanical part, the effect size is 1mm

Standardizes effect size

Used when there is no clear cut definition regard the required effect, or the scale is arbitrary,or to compare between different researches with different scales
this site uses the Cohen's d as Standardizes effect size

Expected Cohen's d $$\:d=\frac{|\mu_1-\mu_0|}{\sigma}$$ Observed Cohen's d $$\:d=\frac{|\overline{x}-\mu_0|}{\sigma} $$ The Cohen's standardized effect was also named as following:

Effect ratio

Used when there is no clear cut definition regard the required effect, and compare the effect size to the current expected value.
Calculate the effect size as a ratio of the expected value