1D Exponential Family

Learn More

Exponential family

A univariate density of the exponential family has the form

$$p(x) = \frac{1}{Z(\underline{\theta})} \exp\Big( \sum_{k} \theta_k \, T_k(x) \Big), \qquad Z(\underline{\theta}) = \int_{a}^{b} \exp\Big( \sum_{k} \theta_k \, T_k(x) \Big) \, \mathrm{d}x ,$$

with the sufficient statistics (features) $T_k(x)$, the natural parameters (weights) $\theta_k$ and the normalization constant (partition function) $Z$. Its logarithm $\log Z$ is the log-partition function. The log-density is a weighted sum of the features, shifted by $-\log Z$.

Special cases

  • Gaussian: $T(x) = (x, x^2)$ with $\theta_1 = \mu/\sigma^2$ and $\theta_2 = -1/(2\sigma^2) < 0$.
  • Laplace: $T(x) = |x|$, $\theta < 0$.
  • Generalized normal: $T(x) = |x|^\beta$, $\theta < 0$.
  • Exponential ($x > 0$): $T(x) = x$, $\theta < 0$.
  • Gamma ($x > 0$): $T(x) = (\log x, x)$ with $\theta_1 = k - 1$ and $\theta_2 = -1/\vartheta$.
  • Gumbel: $T(x) = (x, e^{-x})$ with $\theta_1 = \theta_2 = -1$.
  • Logistic: $T(x) = (x, \log(1+e^x))$ with $\theta_1 = 1$ and $\theta_2 = -2$.
  • Hyperbolic secant: $T(x) = \log\cosh x$, $\theta = -1$.
  • Cauchy: $T(x) = \log(1+x^2)$, $\theta = -1$ (for fixed scale).
  • von Mises-like: $T(x) = (\cos x, \sin x)$ (periodic, on a bounded interval).
  • Uniform: no features.

Normalizability

$Z$ must be finite. On $\mathbb{R}$ this requires the exponent to decrease fast enough towards $\pm\infty$. For example, a polynomial exponent must have an even highest power with a negative weight. This demo always normalizes on the bounded domain $[a, b]$, so every choice gives a valid (truncated) density. A warning is shown if the density does not vanish at the boundary.

Moments and maximum entropy

The density with the largest entropy among all densities with given expectations $\mathrm{E}\{T_k(x)\}$ is the exponential family density with features $T_k$. Therefore, the Gaussian is the maximum entropy density for a given mean and variance. The dashed line shows the Gaussian with the same mean and variance as $p(x)$; its entropy is always at least the entropy of $p(x)$.

The derivatives of the log-partition function give the moments of the sufficient statistics:

$$\frac{\partial \log Z}{\partial \theta_k} = \mathrm{E}\{T_k(x)\}, \qquad \frac{\partial^2 \log Z}{\partial \theta_k \, \partial \theta_l} = \mathrm{Cov}\{T_k(x), T_l(x)\} .$$

Numerics

$p(x)$, $\log Z$, the moments and the entropy are computed with the midpoint rule on 2000 points of the domain.