Math and physics notes

matrix identities

linear regression

The Moorse-Penrose pseudoinverse for orthogonal real matrices is

\[A^+ = (A^T A)^{-1} A^T\]

The normal equation is

\[\widehat\beta = (X^T \Sigma^{-1} X)^{-1} X^T \Sigma^{-1} y\]

The hat matrix is given by

\[X (X^T \Sigma^{-1} X)^{-1} X^T \Sigma^{-1}\]

matrix decompositions

Singular Value decomposition: \(A = U D V^T\) where

LU decomposition: \(A = LU\) where

The Normal distribution

\[\mathcal{N}(\mathbf{x} | \mathbf{\mu}, \Sigma) = \exp\left( - \frac{1}{2} \mathbf{r}^T \Sigma^{-1} \mathbf{r} \right) \, |2 \pi \Sigma|^{-\frac{1}{2}}\]

where \(\mathbf{r} = \mathbf{x} - \mathbf{\mu}\). Note that the \(2 \pi\) is inside the determinant.

linear operations on Gaussian random variables: If \(x \sim \mathcal{N}(\mu_x, \Sigma_X)\), and \(y \sim \mathcal{N}(\mu_y, \Sigma_y)\), then

Product of Gaussian PDFs: \(\mathcal{N}(\mathbf{x} \vert \alpha, \Sigma) \mathcal{N}(\mathbf{x} \vert \beta, \Omega) = \eta \mathcal{N}(\mathbf{x} | \mathbf{m}, C)\), where

Refactoring the product of heirarchical Gaussian PDFs:
\(\mathcal{N}(\mathbf{x} \vert M \theta, C) \mathcal{N}(\theta \vert \mu, \Lambda) = \mathcal{N}(\theta \vert \mathbf{a}, A) \mathcal{N}(\mathrm{x} \vert \mathrm{b}, B)\), where

Sources: Wikipedia, The Matrix Cookbook, Hogg+ 2020 (in prep)