3.6  Properties of Covariance Matrices

Covariance matrices are always positive semidefinite. To see why, let X be any random vector with covariance matrix Σ, and let b be any constant row vector. Define the random variable

[3.33]

By [3.28], the variance of Y is

[3.34]

The variance of any random variable Y must be nonnegative, so expression [3.34] is nonnegative. Recall from Section 2.7 that a symmetric matrix Σ is positive semidefinite if bΣb′ ≥ 0 for all row vectors b. A covariance matrix is necessarily symmetric, so we conclude that all covariance matrices Σ are positive semidefinite.

We shall call a random vector nonsingular or singular according to whether its covariance matrix is positive definite or singular positive semidefinite.