There are many good articles about Frequentist vs Bayesian. Cassie Kozyrkov’s blog is a good place to start.

Comparison

  • The Frequentist world
    • Parameters are fixed
    • Uncertainty comes from sampling
    • Focus on the problem $p(x | \theta)$
  • The Bayesian world
    • Parameters are random variables
    • Uncertainty comes from both sampling and parameters
    • Focus on the problem $p(\theta | x)$

CI under Frequentist and Bayesian framework

I want to discuss this since I worked closely with medical practitioners in the past and found many of them misinterpreted “confidence intervals”. A Bayesian CI is the “confidence intervals” we commonly referred to, i.e., with 95% probability that the unobserved parameter falls in the CI. However, many practitioners misinterprets the Frequentist CI in the same way. This is incorrect.

A Bayesian CI is called a credible interval. In Bayesian world, parameters are random variables with a distribution. Therefore it is valid to construct a “interval” based on posterior distribution.

Bayesian Credible Interval
Let $\theta$ be the unobserved parameter and $X$ be the observed data. A 95% credible intertal is an interval $[L(X), U(X)]$ such that \(\operatorname{P}[L(X) < \theta < U(X)] = 0.95\)

A Frequentist CI is called a confidence interval. In a Frequentist world, parameters are fixed; there is one and only one true value for the unobserved parameter $\theta$. Therefore, it is not valid to discuss the distribution of the $\theta$ since it is not a random variable.

Frequentist Confidence Interval
Let $\theta$ be the unobserved parameter and $X_1, X_2, …, X_m$ be $m$ random samples, each with sample size $n$ drawn from the population. A 95% confidence interval is $[L(X_i), U(X_i)]$, such that, \(\operatorname{P}(L(X_i) < \theta < U(X_i)) = 0.95\)

Note that here we assume you can repeatedly sample datasets $X_1, …., X_m$ from a population. What the frequentist C.I. speaks about is that all confidence intervals $[L(X_1), U(X_1)], … , [L(X_m), U(X_m)]$ derived from those datasets will contains the unobserved parameter $\theta$. $\theta$ is either in the interval or outside the interval. In the unfortunate case that the confidence interval does not contains $\theta$, the confidence interval can be completely and ridiculously wrong. Due to $\theta$ is not a random variable and does not have a distribution, you have to propose a C.I. The C.I. could be anything (e.g., one standard derivation of data), and it is valid as long as, 95% of the C.I. contains $\theta$.

The difference in C.I. originate from the different view of parameter $\theta$. In the Bayesian world, the credible interval $(L,U)$ is fixed and $\theta$ is random. In the Frequentist world, $\theta$ is fix and endpoints $L,U$ of confidence interval $(L,U)$ are random. If you are interested in this topic, you can read more here.