Bayesian Inference Notes

1. Bayes Theorem

1.1. Bayes theorem

𝑝 (𝜃 | 𝑦) = \frac{𝑝 (𝑦 | 𝜃) 𝑝 (𝜃)}{𝑝 (𝑦)} \propto 𝑝 (𝑦 | 𝜃) 𝑝 (𝜃)

1.2. Prior predictive distribution

1.3. Posterior predictive distribution

2. Fundamental Distributions

Name	PDF/PMF	Mean	Variance	Mode
$Beta (𝑦 \| 𝛼, 𝛽)$	$\frac{Γ (𝛼 + 𝛽)}{Γ (𝛼) Γ (𝛽)} 𝑦^{𝛼 - 1} {(1 - 𝑦)}^{𝛽 - 1}$	$\frac{𝛼}{𝛼 + 𝛽}$	$\frac{𝛼 𝛽}{{(𝛼 + 𝛽)}^{2} (𝛼 + 𝛽 + 1)}$	$\frac{𝛼 - 1}{𝛼 + 𝛽 - 2}$
$Binomial (𝑦 \| 𝑛, 𝑝)$	$(\binom{𝑛}{𝑦}) 𝑝^{𝑦} {(1 - 𝑝)}^{𝑛 - 𝑦}$	$𝑛 𝑝$	$𝑛 𝑝 (1 - 𝑝)$
$Exponential (𝑦 \| 𝜆)$	$𝜆 𝑒^{- 𝜆 𝑦}$	$\frac{1}{𝜆}$	$\frac{\ln 2}{𝜆}$	$0$
$Erlang (𝑦 \| 𝜆, 𝑘)$	$\frac{𝜆^{𝑘} 𝑦^{𝑘 - 1} 𝑒^{- 𝜆 𝑦}}{(𝑘 - 1)!}$	$\frac{𝑘}{𝜆}$	$\frac{𝑘}{𝜆^{2}}$	$\frac{1}{𝜆} (𝑘 - 1)$
$ExGauss (𝑦 \| 𝜇, 𝜎, 𝜆)$	$\frac{𝜆}{2} \exp (\frac{𝜆}{2} (2 𝜇 + 𝜆 𝜎^{2} - 2 𝑦)) erfc (\frac{𝜇 + 𝜆 𝜎^{2} - 𝑦}{\sqrt{2} 𝜎}$
$Gamma (𝑦 \| 𝛼, 𝛽)$	$\frac{𝛽^{𝛼}}{Γ (𝛼)} 𝑦^{𝛼 - 1} 𝑒^{- 𝛽 𝑦}$	$\frac{𝛼}{𝛽}$	$\frac{𝛼}{𝛽^{2}}$	$\frac{𝛼 - 1}{𝛽}$
$InvGamma (𝑦 \| 𝛼, 𝛽)$	$\frac{𝛽^{𝛼}}{Γ (𝛼)} 𝑦^{- 𝛼 - 1} 𝑒^{- 𝛽 / 𝑦}$	$\frac{𝛽}{𝛼 - 1}$	$\frac{𝛽^{2}}{{(𝛼 - 1)}^{2} (𝛼 - 2)}$	$\frac{𝛽}{𝛼 - 1}$
$LogNormal (𝑦 \| 𝛼, 𝛽)$	$\frac{1}{𝑦 𝜎 \sqrt{2 𝜋}} 𝑒^{- \frac{{(\ln 𝑦 - 𝜇)}^{2}}{2 𝜎^{2}}}$	$𝑒^{𝜇 + \frac{𝜎^{2}}{2}}$	$(𝑒^{𝜎^{2}} - 1) 𝑒^{2 𝜇 + 𝜎^{2}}$	$𝑒^{𝜇 - 𝜎^{2}}$
$Possion (𝑦 \| 𝜆)$	$\frac{𝜆^{𝑦} 𝑒^{- 𝜆}}{𝑦!}$	$𝜆$	$𝜆$
$NegBinomial (𝑘 \| 𝑟, 𝑝)$	$(\binom{𝑘 + 𝑟 - 1}{𝑘}) {(1 - 𝑝)}^{𝑘} 𝑝^{𝑟}$	$\frac{𝑟 (1 - 𝑝)}{𝑝}$	$\frac{𝑟 (1 - 𝑝)}{𝑝^{2}}$
$Normal (𝑦 \| 𝜇, 𝜎^{2})$	$\frac{1}{\sqrt{2 𝜋} 𝜎} \exp (- \frac{{(𝑦 - 𝜇)}^{2}}{2 𝜎^{2}})$	$𝜇$	$𝜎^{2}$	$𝜇$
$Student (𝑦 \| 𝜈)$	$\frac{Γ (\frac{𝜈 + 1}{2})}{\sqrt{𝜋 𝜈} Γ (\frac{𝜈}{2})} {(1 + \frac{𝑦^{2}}{𝜈})}^{- \frac{𝜈 + 1}{2}}$	$0$	$\frac{𝜈}{𝜈 - 2}$	$0$
$Uniform (𝑦 \| 𝑎, 𝑏)$	$\frac{1}{𝑏 - 𝑎}$	$\frac{𝑎 + 𝑏}{2}$	$\frac{{(𝑏 - 𝑎)}^{2}}{12}$

Table 1: Single Variate Distributions

3. Functions

3.1. Beta Function

Properties:

$𝐵 (𝑧_{1}, 𝑧_{2}) = 𝐵 (𝑧_{2}, 𝑧_{1})$
$𝐵 (𝑧_{1}, 𝑧_{2}) = \frac{Γ (𝑧_{1}) Γ (𝑧_{2})}{Γ (𝑧_{1} + 𝑧_{2})}$
$𝐵 (𝑚, 𝑛) = \frac{(𝑚 - 1)! (𝑛 - 1)!}{(𝑚 + 𝑛 - 1)!} = \frac{𝑚 + 𝑛}{𝑚 𝑛} / (\binom{𝑚 + 𝑛}{𝑚})$

4. Conjugate Prior

The idea of conjugate prior is that for a give likelihood we choose a prior distribution such that, after observing data and applying Bayes’ theorem, the posterior distribution belongs to the same family as the prior.

That is, if $𝑝 (𝜃)$ and $𝑝 (𝜃 | 𝑦)$ have the same distributional form, then the prior is called a conjugate prior for the likelihood model.

This is useful because it makes Bayesian updating analytically tractable. Instead of performing difficult integration or numerical approximation, we can often derive the posterior parameters in closed form.

5. Conjugate Prior for Exponential Families

Note general exponential family:

𝑝 (𝑦_{𝑖} | 𝜃) = ⁠

⁠

⁠exp(

⁠

⁠−

⁠

⁠)

⁠

Likelihood of a sequence of i.i.d.samples:

𝑝 (𝑦 | 𝜃) = ⁠

⁠

⁠exp(

⁠

⁠−

⁠

⁠)

⁠

So conjugate prior for that likelihood is

Posterior is

6. Proper and Improper Prior Distributions

A prior is called proper if it is a valid probability distribution:

And improper if

If a prior is proper, so must the posterior.
If a prior is improper, the posterior could be proper or improper.

In theory, all priors are acceptable, as long as the posterior is proper.

7. Fisher Information Matrix

8. Jeffreys’ Prior

9. Pivotal Quantities

For the binomial and other single-parameter models, different principles give (slightly) different noninformative prior distributions. But for two cases—location parameters and scale parameters—all principles seem to agree[1].

9.1. Location Parameter

𝑝 (𝜃) \sim 1

9.2. Scale Parameter

𝑝 (𝜃) \sim \frac{1}{𝜃}

10. Predictive Accuracy

People care about the accuracy in two different ways. First to assume that the model is all we known and check posterior predictions. The second is to compare several candidate models. Even if all of the models being considered have mismatches with the data, it can be informative to evaluate their predictive accuracy, compare them, and consider where to go next[2].

11. KL Divergence

12. Linear Algebra

12.1. Convex Combination

A subset $𝐴$ of a vector space $𝑉$ is said to be convex if

for all vectors

, and all scalars $𝜆$ in $[0, 1]$ .

Via induction, this can be seen to be equivalent to the requirement that

for all vectors

, and for all scalars $𝜆_{1}, 𝜆_{2}, \dots, 𝜆_{𝑛} \geq 0$ such that $\sum 𝑘_{𝑖} = 1$ .

Bibliography

[1] A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and others, Bayesian Data Analysis, Third. Boca Raton, Florida: Crc, 2013. [Online]. Available: https://stat.columbia.edu/~gelman/book/
[2] A. Gelman, J. Hwang, and A. Vehtari, “Understanding predictive information criteria for Bayesian models,” Statistics and Computing, vol. 24, no. 6, pp. 997–1016, Nov. 2014, doi: 10.1007/s11222-013-9416-2.