8.1 Finite-Difference Derivative Approximations

The approach in this section is to make small, finite perturbations in the values of \(x\) and examine the resulting differences in the function values.

Approximating the Gradient

A popular formula for approximating the partial derivative \(\partial f / \partial x\) at a given point \(x\) is the forward-difference, or one-sided-difference, approximation, defined as

\[\frac{\partial f}{\partial x_i}(x) \approx \frac{f(x + \epsilon e_i) - f(x)}{\epsilon}\]

This process requires \((n+1)\) evaluations of \(f\) at \(x, x + \epsilon e_i\).

When \(f\) is twice continuously differentiable, we have

\[f(x + p) = f(x) + \nabla f(x)^\top p + \frac{1}{2} p^\top \nabla^2 f(x + tp)p\]

for some \(t \in (0, 1)\). Let \(L\) be a bound on the size of \(\lVert \nabla^2 f(\cdot) \rVert\). It follows that

\[\lVert f(x + p) - f(x) - \nabla f(x)^\top p \rVert \leq (L / 2)\lVert p \rVert^2\]

Let \(p\) be \(\epsilon e_i\). We have that

\[\nabla f(x)^\top p = \nabla f(x)^\top e_i = \partial f / \partial x_i\]

and

\[\frac{\partial f}{\partial x_i}(x) = \frac{f(x + \epsilon e_i) - f(x)}{\epsilon} + \delta_\epsilon, \;\;\; \text{where } \lvert \delta_\epsilon \rvert \leq (L / 2)\epsilon\]