###### 3.7.2 Definition of Principal Components

Our example informally introduced principal components. Now let’s formalize them. Consider an *n*-dimensional random vector ** Z** with mean

**μ**and nonsingular covariance matrix

_{Z}**Σ**. We construct principal components in such a manner that the first accounts for as much of the variability of

_{Z}**as possible. The second accounts for as much of the remaining variability of**

*Z***as possible, and so on.**

*Z*Specifically, the first principal component *D*_{1} is defined as

[3.64]

where *v*_{1} has unit length and is selected to maximize the variance of *D*_{1}. This is achieved by setting *v*_{1} equal to the normalized first eigenvector of **Σ _{Z }**—the eigenvector with the largest eigenvalue. In this case, the variance of

*D*

_{1}equals that eigenvalue, λ

_{1}.

The second principal component *D*_{2} is defined as

[3.65]

where *v*_{2} is selected from the set of all *n*-dimensional unit vectors that are orthogonal to *v*_{1} in such a manner as to maximize the variance of *D*_{2}. This is achieved by setting *v*_{2} equal to the normalized second eigenvector of **Σ _{Z }**—the eigenvector with the second largest eigenvalue. The variance of

*D*

_{2}equals that eigenvalue, λ

_{2}.

Proceeding in this manner, we define the remaining principal components. There will be *m* principal components *D _{i}*, each one corresponding to a normalized eigenvector

*v**of*

_{i}**Σ**.

_{Z}*We can represent*

[3.66]

The vector of principal components ** D** has mean

**μ**=

_{D}**0**and covariance matrix

[3.67]

If **Σ _{Z}** is nonsingular, the number

*m*of principal components equals the dimensionality

*n*of

**. If**

*Z***Σ**is singular, some of its eigenvalues will equal 0, and the number

_{Z}*m*of principal components will be less than the dimensionality

*n*of

**Σ**. In this case, [3.66] will have reduced the dimensionality of the singular

_{Z}**in the same manner as that described in Section 3.6.**

*Z*###### Exercises

Consider a random vector ** Z** with mean and covariance matrix

[3.68]

- Calculate the determinant of the corresponding correlation matrix.
- Is
singular, multicollinear or neither of these?*Z* - Calculate eigenvalues and eigenvectors of
**Σ**._{Z} - Represent
in terms of its principal components as in [3.66].*Z* - What is the covariance matrix
**Σ**of the vector of principal components_{D}?*D* - Construct an approximation for
based on the first two principal components of*Z***Z**. - Construct the covariance matrix of . Compare your result with the covariance matrix of
.*Z*