Statistical tests
Probability and Statistics
21 vs. 20
\(FPR_{21} = 1.3\% + 0.5\% + 0.2\% + 0.1\% \approx 2.1\% < 5\%\)
\(FPR_{20} = 2.8\% + 1.3\% + 0.5\% + 0.2\% + 0.1\% \approx 4.9\% < 5\%\)
Density function
- The statistic \(Q\) has a binomial distribution:
\[ Q \sim \text{Binomial}(30, 0.5) \]
- The probability function of a discrete distribution \(p_{\xi}(x)\) is the probability that a random variable \(\xi\) takes the value \(x\):
\[ p_{\xi}(x) = P(\xi = x) \]
\(P(\xi = 10)\) = 0.028
\(P(\xi = 16)\) = 0.135
\(P(\xi = 15)\) = 0.144
\(P(\xi = 14)\) = 0.135
Probability of the critical area
\[ P(Q \geq x) = \sum_{i = x}^{n} p_{\xi}(i) \]
\(P(Q \geq 20) = \sum_{i = 20}^{30} p_{\xi}(i)\) = 0.049
What if \(\color{#e64173}{Q \geq 19}\)?
\(P(Q \geq 19) = \sum_{i = 19}^{30} p_{\xi}(i)\) = 0.1002
Then the probability of error is even higher than 10%, which unlikely good for us at all.
Cumulative distribution function
The cumulative distribution function \(F_{\xi}(x) = P(\xi \leq x)\) is the probability that a random variable \(\xi\) will take a value no greater than \(x\).
\[ F_{\xi}(x) = \sum_{i = 0}^{x} p_{\xi}(i) \]
For \(x = 19\): \(P(\xi \leq 19) = \sum_{i = 0}^{19} p_{\xi}(i)\) = 0.9506
Since \(P(\xi \leq 19) + P(\xi \geq 21) = 1\), we can calculate the level of significance of our test:
\(1 - P(\xi \leq 19) = P(\xi \geq 20)\) = 0.0494
Quantile: example
For the value \(\xi \sim Bin(30, 0.5)\), let’s calculate the \(0.95\)-quantile. We will solve the problem simply by selection.
\[P(\xi \leq 18) \approx 0.9\]
\[P(\xi \leq 19) \approx 0.951\]
\[P(\xi \leq 20) \approx 0.97\]
Additional example
- Number of deliveries — 50
- Sufficient probability of success — 0.1, i.e. if the courier’s work costs 100₴, then the delivery cost is 1000₴.
If \(Q \geq 10\), then reject \(H_0\).
Sum of probabilities in the critical region: 0.025
\(p\)-value
The \(p\)-value is the probability of obtaining a result that is at least as extreme as our observations, provided that the null hypothesis \(H_0\) is true.
graph TD
A["n = 30 \n H0: μ = 0.5 \n H1: μ > 0.5 \n α = 0.05"] --> B["If Q >= 20: \n Reject H0"]
- \(p > \alpha \equiv\) \(q\) outside the critical region \(\equiv\) does not reject \(H_0\).
- \(p \leq \alpha \equiv\) \(q\) in the critical region \(\equiv\) reject \(H_0\).
Two-sided criterion
Does the color of the car affect compliance with traffic rules?
- \(Q = \xi_1 + \xi_2 + \ldots + \xi_{n}\)
- \(H_0: \mu = 0.5\)
- \(H_1: \mu \neq 0.5\)
- \(\alpha = 0.05\)
If \(q \geq 21\) or \(q \leq 9\), then we reject \(H_0\).
Asymmetric distribution
In order to construct a two-sided criterion, you need to find regions on the left and right whose area is no more than \(\frac{\alpha}{2}\).
Asymmetric distribution: criterion
So, our criterion for testing the hypothesis is
\[H_0: \mu = 0.8\] \[H_1: \mu \neq 0.8\]
is as follows:
\[S = \{Q(\xi) \leq 18\} \cup \{Q(\xi) \geq 29\}\]
Asymmetric distribution: \(p\)-value
\(N = 30, \mu = 0.8\).
If \(Q = 28\), then the \(p\)-value is 0.088.
It can be seen that the \(p\)-value is \(> 0.05\), so at a significance level of \(0.05\), even \(28\) successes are not enough to reject the probability of success of \(80\%\).
How to remember?
Power
Let’s recall the delivery task:
\(H_0: \mu = 0.5 \\H_1: \mu > 0.5 \\Q = \text{number of confirmed orders} \\\alpha = 0.05 \\ S = \{Q \geq 20\}\)
Suppose that \(H_1\) is true: \(\mu = 0.6\).
\[\text{Power} = \mathbb{P}(Q \geq 20|\mu = 0.6)\]
\[\text{Power} = \mathbb{P}(Q \geq 20|\mu = 0.6)\]
False Positive Rate is 4.9%Power is 29.1%You can see that the power is about \(30\%\). This is quite a small value, because if our product is profitable, we will only see it with a probability of \(30\) percent using our test. We can easily miss the effect.
Power vs. Sample Size
It is generally accepted that \(80\%\) of power is considered acceptable for work.
Let’s see how the power changes as the sample size increases, and how many experiments are needed to detect the effect at \(\mu=0.6\) in \(80\%\) of cases.
Power vs. Sample Size (cont.)
What if we want to detect an even smaller effect? For example, if we want to reject the hypothesis at \(\mu = 0.51\). Often, an improvement in the probability of success by \(1\%\) can be significant for a product, so this question is not without meaning.
Power vs. effect size
The graphs show that a larger sample size is required to detect a smaller effect.
Let’s see how the power changes for different parameters \(\mu\) for a fixed \(N = 30\).
In our experiment, we detect an effect well only if the probability of success in the population is at least \(72\%\).
Minimum detectable effect
Minimal Detectable Effect (MDE, Minimal Detectable Effect) — this is the smallest effect that we can detect with an experiment (usually at \(80\%\) power).
In our example, \(\text{MDE} = +0.22\)
Summary
Hypothesis testing algorithm for our task:
- Business problem / hypothesis
- Formulation of the null and alternative hypotheses:
- \(\mathsf{H}_0: \mu = 0.5\)
- \(\mathsf{H}_1: \mu > 0.5\).
- Statistics of the criterion \(Q = \sum_{i=1}^n \text{Bernoulli}(\mu)\), \(q = 19\).
- The distribution of \(Q\) at \(\mathsf{H}_0\) and the critical region (\(p\)-value) at \(\alpha = 0.05\).
- \(\text{MDE} (n, \text{power}, \alpha)\).
- Confidence interval for \(\mu\).
