2.3 Gradient and Gradient-Hessian Approximations

2.3  Gradient and Gradient-Hessian Approximations

Polynomials are frequently used to locally approximate functions. There are various ways this may be done. We consider here several forms of differential approximation.

2.3.1 Univariate Approximations

Consider a function f :  →  that is differentiable in an open interval about some point x[0] . The linear polynomial

[2.16]

provides a good approximation for f, at least in a small interval about x[0]. This is because:

  • p1 equals f at x[0], and
  • p1 has the same first derivative as f at x[0].

If f is twice differentiable in an open interval about x[0], we can improve the approximation with a quadratic polynomial

[2.17]

Consider the function

[2.18]

which has first and second derivatives

[2.19]

[2.20]

on . Let’s construct a linear polynomial approximation for f about the point x[0] = 0. Applying [2.16], we obtain

[2.21]

[2.22]

This is graphed in Exhibit 2.2.

Exhibit 2.2: Comparison of function f and linear polynomial p1.

We can improve the approximation, at least for values of x close to 0, with a quadratic polynomial. Applying [2.17] at x[0] = 0, we obtain

[2.23]

[2.24]

This is graphed in Exhibit 2.3.

Exhibit 2.3: Comparison of function f and linear polynomial p2.
2.3.2 Multivariate Approximations

Polynomial approximations [2.16] and [2.17] generalize to multiple dimensions. For f : n → , gradients replace first derivatives and Hessians replace second derivatives, so linear polynomial [2.16] and quadratic polynomial [2.17] become

[2.25]

[2.26]

We call these gradient approximations and gradient-Hessian approximations, respectively.

Consider the function

[2.27]

which has gradient and Hessian

[2.28]

[2.29]

Let’s construct a gradient-Hessian approximation about the point (0, 1). Applying [2.26], we obtain

[2.30]

[2.31]

2.3.3 Taylor Polynomials

The linear and quadratic polynomial approximations discussed in this section are examples of a more general concept called Taylor polynomials. Consider a function f :  → whose first m derivatives exist in an open interval about a point x[0] . The polynomial

[2.32]

is called the mth-order Taylor polynomial of f about the point x[0]. It provides a good approximation for f, at least in a small interval about x[0]. If all derivatives exist for f in an open interval about x[0], we may consider the limiting polynomial as m approaches infinity. This is called the Taylor series expansion of f about the point x[0]. In some cases—but not all!—a function equals its Taylor series expansions on . For example, functions ex and sin(x) both equal their Taylor series expansions about the point x[0] = 0:

[2.33]

[2.34]

Taylor polynomials and Taylor series generalize to higher dimensions.

Exercises
2.1

Apply [2.17] to construct a quadratic polynomial approximation for the function f(x) = xex about the point x[0] = 0.
Solution

2.2

Apply [2.26] to construct a quadratic polynomial approximation for the function f(x1, x2) = x1exp(x2) about the point x[0] = (0, 0).
Solution

2.3

Construct the Taylor series expansion for the function log(1 + x) about the point x[0] = 0.
Solution