2.3 Gradient and Gradient-Hessian Approximations
Polynomials are frequently used to locally approximate functions. There are various ways this may be done. We consider here several forms of differential approximation.
2.3.1 Univariate Approximations
Consider a function f : → that is differentiable in an open interval about some point x . The linear polynomial
provides a good approximation for f, at least in a small interval about x. This is because:
- p1 equals f at x, and
- p1 has the same first derivative as f at x.
If f is twice differentiable in an open interval about x, we can improve the approximation with a quadratic polynomial
Consider the function
which has first and second derivatives
on . Let’s construct a linear polynomial approximation for f about the point x = 0. Applying [2.16], we obtain
This is graphed in Exhibit 2.2.
We can improve the approximation, at least for values of x close to 0, with a quadratic polynomial. Applying [2.17] at x = 0, we obtain
This is graphed in Exhibit 2.3.
2.3.2 Multivariate Approximations
Polynomial approximations [2.16] and [2.17] generalize to multiple dimensions. For f : n → , gradients replace first derivatives and Hessians replace second derivatives, so linear polynomial [2.16] and quadratic polynomial [2.17] become
We call these gradient approximations and gradient-Hessian approximations, respectively.
Consider the function
which has gradient and Hessian
Let’s construct a gradient-Hessian approximation about the point (0, 1). Applying [2.26], we obtain
2.3.3 Taylor Polynomials
The linear and quadratic polynomial approximations discussed in this section are examples of a more general concept called Taylor polynomials. Consider a function f : → whose first m derivatives exist in an open interval about a point x . The polynomial
is called the mth-order Taylor polynomial of f about the point x. It provides a good approximation for f, at least in a small interval about x. If all derivatives exist for f in an open interval about x, we may consider the limiting polynomial as m approaches infinity. This is called the Taylor series expansion of f about the point x. In some cases—but not all!—a function equals its Taylor series expansions on . For example, functions ex and sin(x) both equal their Taylor series expansions about the point x = 0:
Taylor polynomials and Taylor series generalize to higher dimensions.
Construct the Taylor series expansion for the function log(1 + x) about the point x = 0.