3.7.2 Definition of Principal Components

3.7.2  Definition of Principal Components

Our example informally introduced principal components. Now let’s formalize them. Consider an n-dimensional random vector Z with mean μZ and nonsingular covariance matrix ΣZ. We construct principal components in such a manner that the first accounts for as much of the variability of Z as possible. The second accounts for as much of the remaining variability of Z as possible, and so on.

Specifically, the first principal component D1 is defined as

[3.64]

where v1 has unit length and is selected to maximize the variance of D1. This is achieved by setting v1 equal to the normalized first eigenvector of ΣZ —the eigenvector with the largest eigenvalue. In this case, the variance of D1 equals that eigenvalue, λ1.

The second principal component D2 is defined as

[3.65]

where v2 is selected from the set of all n-dimensional unit vectors that are orthogonal to v1 in such a manner as to maximize the variance of D2. This is achieved by setting v2 equal to the normalized second eigenvector of ΣZ —the eigenvector with the second largest eigenvalue. The variance of D2 equals that eigenvalue, λ2.

Proceeding in this manner, we define the remaining principal components. There will be m principal components Di, each one corresponding to a normalized eigenvector vi of ΣZ. We can represent

[3.66]

The vector of principal components D has mean μD = 0 and covariance matrix

[3.67]

If ΣZ is nonsingular, the number m of principal components equals the dimensionality n of Z. If ΣZ is singular, some of its eigenvalues will equal 0, and the number m of principal components will be less than the dimensionality n of ΣZ. In this case, [3.66] will have reduced the dimensionality of the singular Z in the same manner as that described in Section 3.6.

Exercises
3.21

Consider a random vector Z with mean and covariance matrix

[3.68]

  1. Calculate the determinant of the corresponding correlation matrix.
  2. Is Z singular, multicollinear or neither of these?
  3. Calculate eigenvalues and eigenvectors of ΣZ.
  4. Represent Z in terms of its principal components as in [3.66].
  5. What is the covariance matrix ΣD of the vector of principal components D?
  6. Construct an approximation  for Z based on the first two principal components of Z.
  7. Construct the covariance matrix of . Compare your result with the covariance matrix of Z.

Solution