2.3 Gradient and Gradient-Hessian Approximations
Polynomials are frequently used to locally approximate functions. There are various ways this may be done. We consider here several forms of differential approximation.
2.3.1 Univariate Approximations
Consider a function f : →
that is differentiable in an open interval about some point x[0]
. The linear polynomial
[2.16]

provides a good approximation for f, at least in a small interval about x[0]. This is because:
- p1 equals f at x[0], and
- p1 has the same first derivative as f at x[0].
If f is twice differentiable in an open interval about x[0], we can improve the approximation with a quadratic polynomial
[2.17]

Consider the function
[2.18]

which has first and second derivatives
[2.19]

[2.20]

on . Let’s construct a linear polynomial approximation for f about the point x[0] = 0. Applying [2.16], we obtain
[2.21]

[2.22]

This is graphed in Exhibit 2.2.

We can improve the approximation, at least for values of x close to 0, with a quadratic polynomial. Applying [2.17] at x[0] = 0, we obtain
[2.23]

[2.24]

This is graphed in Exhibit 2.3.

2.3.2 Multivariate Approximations
Polynomial approximations [2.16] and [2.17] generalize to multiple dimensions. For f : n →
, gradients replace first derivatives and Hessians replace second derivatives, so linear polynomial [2.16] and quadratic polynomial [2.17] become
[2.25]

[2.26]

We call these gradient approximations and gradient-Hessian approximations, respectively.
Consider the function
[2.27]

which has gradient and Hessian
[2.28]

[2.29]

Let’s construct a gradient-Hessian approximation about the point (0, 1). Applying [2.26], we obtain
[2.30]

[2.31]

2.3.3 Taylor Polynomials
The linear and quadratic polynomial approximations discussed in this section are examples of a more general concept called Taylor polynomials. Consider a function f : →
whose first m derivatives exist in an open interval about a point x[0]
. The polynomial
[2.32]

is called the mth-order Taylor polynomial of f about the point x[0]. It provides a good approximation for f, at least in a small interval about x[0]. If all derivatives exist for f in an open interval about x[0], we may consider the limiting polynomial as m approaches infinity. This is called the Taylor series expansion of f about the point x[0]. In some cases—but not all!—a function equals its Taylor series expansions on . For example, functions ex and sin(x) both equal their Taylor series expansions about the point x[0] = 0:
[2.33]

[2.34]

Taylor polynomials and Taylor series generalize to higher dimensions.
Exercises
Apply [2.17] to construct a quadratic polynomial approximation for the function f(x) = xex about the point x[0] = 0.
Solution
Apply [2.26] to construct a quadratic polynomial approximation for the function f(x1, x2) = x1exp(x2) about the point x[0] = (0, 0).
Solution
Construct the Taylor series expansion for the function log(1 + x) about the point x[0] = 0.
Solution