Bayesian Inference Notes

Bayes Theorem

Bayes theorem

Prior predictive distribution

Posterior predictive distribution

Fundamental Distributions

Functions

Beta Function

Properties:

Conjugate Prior

The idea of conjugate prior is that for a give likelihood we choose a prior distribution such that, after observing data and applying Bayes’ theorem, the posterior distribution belongs to the same family as the prior.

That is, if and have the same distributional form, then the prior is called a conjugate prior for the likelihood model.

This is useful because it makes Bayesian updating analytically tractable. Instead of performing difficult integration or numerical approximation, we can often derive the posterior parameters in closed form.

Conjugate Prior for Exponential Families

Note general exponential family:

Likelihood of a sequence of i.i.d.samples:

So conjugate prior for that likelihood is

Posterior is

Proper and Improper Prior Distributions

A prior is called proper if it is a valid probability distribution:

And improper if

• If a prior is proper, so must the posterior.
• If a prior is improper, the posterior could be proper or improper.

In theory, all priors are acceptable, as long as the posterior is proper.

Linear Algebra

Convex Combination

A subset of a vector space is said to be convex if for all vectors , and all scalars in .

Via induction, this can be seen to be equivalent to the requirement that for all vectors , and for all scalars such that .