3.6 Properties of Covariance Matrices
Covariance matrices are always positive semidefinite. To see why, let X be any random vector with covariance matrix Σ, and let b be any constant row vector. Define the random variable
By [3.28], the variance of Y is
The variance of any random variable Y must be nonnegative, so expression [3.34] is nonnegative. Recall from Section 2.7 that a symmetric matrix Σ is positive semidefinite if bΣb′ ≥ 0 for all row vectors b. A covariance matrix is necessarily symmetric, so we conclude that all covariance matrices Σ are positive semidefinite.
We shall call a random vector nonsingular or singular according to whether its covariance matrix is positive definite or singular positive semidefinite.