MTH 451/551
Lecture 2

1 Orders of Convergence

[Textbook reference: §1.2]

In the previous lecture, we saw two types of numerical methods. We saw the midpoint method, a type of quadrature that approximates integrals using a function evaluation at the midpoint of the interval, and finite difference quotients, which approximate derivatives at a point using multiple function evaluations. In both of these examples, we were able to compute the error of the approximation. Recall that the (signed) errors of these approximations are

(Midpoint Rule) ${\displaystyle {\int }_a^b}f(x)𝑑x-f(c)(b-a)$ $={\displaystyle \frac{f^{\prime \prime }(z)}{24}}{h}^3,$

(Forward Difference) $f^{\prime }(c)-{\displaystyle \frac{f(c+h)-f(c)}{h}}$ $=-{\displaystyle \frac{f^{\prime \prime }(y_{+})}{2}}h,$

(Backward Difference) $f^{\prime }(c)-{\displaystyle \frac{f(c)-f(c-h)}{h}}$ $={\displaystyle \frac{f^{\prime \prime }(y_{-})}{2}}h,$

(Centered Difference) $f^{\prime }(c)-{\displaystyle \frac{f(c+h)-f(c-h)}{2h}}$ $=-{\displaystyle \frac{h^2}{6}}\cdot {\displaystyle \frac{f^{\prime \prime \prime }(z_{+})+f^{\prime \prime \prime }(z_{-})}{2}},$

where $c=(a+b)/2$, $z\in (a,b)$, and $h=b-a$ for the midpoint rule, and $y_{+},z_{+}\in (c,c+h)$ and $y_{-},z_{-}\in (c-h,c)$ for the difference quotients.

In each of these four cases, the error scales in terms of $h$, which we can think of as a parameter. As $h\to 0$, then the errors also go to zero, meaning that the approximations become more and more accurate. We mentioned before that the centered difference approximation is more accurate (at least for small $h$) than either the forward or backward difference approximations. This is because if we halve $h$, then the centered error is (roughly) divided by four, but the forward and backward errors are divided by two. So the centered error shrinks much faster.

We want to study this type of convergence behavior. Consider some quantity $L$ that we wish to approximate (for example, the integral or derivative of $f(x)$), and, for given $h>0$, an approximation $L_h$. If the approximation converges, then $L_h\to L$ as $h\to 0$. Sometimes we may have approximations $L_n$ such that $L_n\to L$ as $n\to \mathrm{\infty }$, where $n$ is usually an integer parameter; we will see examples later (e.g. Gaussian quadrature). We want to quantify and classify how fast $L_h$ converges to $L$ as $h\to 0$.

Definition 1 (Order of convergence).

Let $\{L_h\}$ be a family of approximations of $L$, with $L_h\to L$ as $h\to 0$. Given $p>0$, not necessarily an integer, we say that the convergence is of order $\mathrm{p}$ when there are constants $C,h_0>0$ such that

(1)

If $\{L_n\}$ is a sequence of approximations to $L$, with $L_n\to L$ as $n\to \mathrm{\infty }$, then we say that the convergence is of order $\mathrm{p}$ if there exist constants $C,N>0$ such that

$$|L-L_h|\le C{n}^{-p}\text{whenever }n>N.$$ (2)

It will be very useful when discussing orders of convergence to use big-$𝒪$ notation. This notation is encountered frequently in computer science, when discussing the concept of computational complexity. In our context, the idea is the same, but the setting is slightly different. We write $a_h=𝒪(g(h))$ as $h\to 0$ to express that there exist constants $C,h_0>0$ such that $|a_h|\le Cg(h)$ for all . When it is clear from context, we will omit the phrase “as $h\to 0$”. So,

$$|L-L_h|=𝒪(h^p)\text{means the same thing as}{L}_h\text{ converges to }L\text{ with order }p\text{.}$$

Note that if $a_h=𝒪(h^p)$ then $a_h=𝒪(h^q)$ for all . We prefer to write $𝒪(h^p)$ for the largest $p$ for which it holds.

For a sequence $a_n$, we write $a_n=𝒪(g(n))$ as $n\to \mathrm{\infty }$ to mean that there exist constants $C,N>0$ such that $|a_n|\le Cg(n)$ for all $n\ge N$. So, in the case of $L_n\to L$ as $n\to \mathrm{\infty }$ at order $p$, we write $|L-L_n|=𝒪(n^{-p})$.

Note that functions $g(h)$ or $g(n)$ other than $h^p$ or $n^{-p}$ are commonly encountered. For example, we might see $a_n=𝒪(n\log n)$ or $a_n=𝒪(e^{-n})$.

1.1 Estimating Orders of Convergence

As an example, let $f(x)=\arctan (x)$, and let $c=\sqrt{3}$. We can compute that $f^{\prime }(c)=1/4$. How well do the forward, backward, and centered difference quotients approximate $f^{\prime }(c)$? Let $F(h)=(f(c+h)-f(c))/h$, $B(h)=(f(c)-f(c-h))/h$, and $C(h)=(f(c+h)-f(c-h))/(2h)$. We know from the expressions for the errors that

$|f^{\prime }(c)-F(h)|$ $=𝒪(h),$

$|f^{\prime }(c)-B(h)|$ $=𝒪(h),$

$|f^{\prime }(c)-C(h)|$ $=𝒪(h^2).$

Can we verify these results numerically? We compute the difference quotients for $h=2^{-n}$, $n=0,1,2,\mathrm{\dots }$ (i.e. successive halvings of $h$). Let $E(h)$ denote the error associated with one of the approximations. Then, $E(h)=𝒪(h^p)$ for $p=1$ (forward, backward) or $p=2$ (centered). If we assume that $E(h)=C{h}^p$ for some constant $C>0$, then

$$\frac{E(h)}{E(h/2)}=\frac{C{h}^p}{C{(h/2)}^p}=\frac{h^p}{h^p/2^p}=2^p.$$

So, for the forward and backward difference quotients, we would expect $E(h)/E(h/2)\approx 2$, and for the centered difference, we would expect $E(h)/E(h/2)\approx 4$.

Slightly more generally, again under the assumption that $E(h)=C{h}^p$, if we have two values $h_1$ and $h_2$, then

$$\frac{E(h_1)}{E(h_2)}=\frac{C{h}_1^p}{C{h}_2^p}={\left(\frac{h_1}{h_2}\right)}^p.$$

If we just have the value of the right-hand side of the above expression, we can recover the value of $p$ by computing

$$p={\log }_{h_1/h_2}\left({\left(\frac{h_1}{h_2}\right)}^p\right)=\frac{\log (E(h_1)/E(h_2))}{\log (h_1/h_2)}.$$

We usually use $\log$ interchangeably with $\ln$ (natural logarithm, base $e$), but the above expression is independent of the base of logarithm chosen. In the case described earlier, since we considered successive halvings of $h$, we could estimate the order of convergence by computing ${\log }_2(E(h)/E(h/2))$. We compute these values numerically, and show the results in the table below.

		Forward		Backward		Centered
$n$	$h$	Error	Rate	Error	Rate	Error	Rate
$0$	$2^0$	$7.728\times {10}^{-2}$	—	$1.653\times {10}^{-1}$	—	$4.400\times {10}^{-2}$	—
$1$	$2^{-1}$	$4.521\times {10}^{-2}$	$0.77$	$6.642\times {10}^{-2}$	$1.32$	$1.060\times {10}^{-2}$	$2.05$
$2$	$2^{-2}$	$2.466\times {10}^{-2}$	$0.87$	$2.989\times {10}^{-2}$	$1.15$	$2.616\times {10}^{-3}$	$2.02$
$3$	$2^{-3}$	$1.291\times {10}^{-2}$	$0.93$	$1.421\times {10}^{-2}$	$1.07$	$6.518\times {10}^{-4}$	$2.00$
$4$	$2^{-4}$	$6.606\times {10}^{-3}$	$0.97$	$6.932\times {10}^{-3}$	$1.04$	$1.628\times {10}^{-4}$	$2.00$
$5$	$2^{-5}$	$3.343\times {10}^{-3}$	$0.98$	$3.424\times {10}^{-3}$	$1.02$	$4.069\times {10}^{-5}$	$2.00$
$6$	$2^{-6}$	$1.681\times {10}^{-3}$	$0.99$	$1.702\times {10}^{-3}$	$1.01$	$1.017\times {10}^{-5}$	$2.00$
$7$	$2^{-7}$	$8.432\times {10}^{-4}$	$1.00$	$8.483\times {10}^{-4}$	$1.00$	$2.543\times {10}^{-6}$	$2.00$
$8$	$2^{-8}$	$4.222\times {10}^{-4}$	$1.00$	$4.235\times {10}^{-4}$	$1.00$	$6.358\times {10}^{-7}$	$2.00$
$9$	$2^{-9}$	$2.113\times {10}^{-4}$	$1.00$	$2.116\times {10}^{-4}$	$1.00$	$1.589\times {10}^{-7}$	$2.00$
$10$	$2^{-10}$	$1.057\times {10}^{-4}$	$1.00$	$1.058\times {10}^{-4}$	$1.00$	$3.974\times {10}^{-8}$	$2.00$

Note that if $E(h)=C{h}^p$, then $\log E(h)=\log (C{h}^p)=\log C+p\log h$. Let $\tilde{E}=\log E$, $\tilde{C}=\log C$, and $\tilde{h}=\log h$. Then, the above tells us that $\tilde{E}=\tilde{C}+p\tilde{h}$, in other words, there is a linear relationship between the logarithm of the error and the logarithm of $h$, and the slope is the order of convergence. This means that if we plot the error as a function of $h$ on a log-log plot, then we expect to observe straight lines with slopes of 1 (forward and backward difference) and 2 (centered difference). This is shown in the plot below.

1.2 Richardson Extrapolation

Richardson extrapolation is a common approach for accelerating convergence when some theoretical knowledge of the structure of the error $L-L_h$ is known. For example, $L-L_h$ might be expressed in terms of an expansion in powers of $h$. We motivate the discussion of Richardson extrapolation by revisiting the centered difference approximation $L_h=\varphi (h)=\frac{f(c+h)-f(c-h)}{2h}$ of $L=f^{\prime }(c)$. For simplicity, we assume that $f$ is analytic near $c$ and that $h>0$ is sufficiently small, so we have

$$f^{\prime }(c)=\varphi (h)-\frac{f^{(3)}(c)}{3!}{h}^2-\frac{f^{(5)}(c)}{5!}{h}^4-\frac{f^{(7)}(c)}{7!}{h}^6-\mathrm{\cdots }.$$ (3)

Replacing $h$ by $h/2$ in this formula, we obtain

$$f^{\prime }(c)=\varphi (h/2)-\frac{f^{(3)}(c)}{3!}\frac{1}{2^2}{h}^2-\frac{f^{(5)}(c)}{5!}\frac{1}{2^4}{h}^4-\frac{f^{(7)}(c)}{7!}\frac{1}{2^6}{h}^6-\mathrm{\cdots }.$$ (4)

Both $\varphi (h)$ and $\varphi (h/2)$ are $𝒪(h^2)$ approximations of $f^{\prime }(c)$, but a simple combination of the two eliminates the $h^2$ term in the error expansions, yielding an $𝒪(h^4)$ approximation. Subtracting $2^2\varphi (h/2)$ from $\varphi (h)$ (i.e. subtracting $2^2$ times (4) from (3)) cancels the $h^2$ term, and we obtain

$$\varphi (h)-2^2\varphi (h/2)=-(2^2-1)f^{\prime }(c)+\frac{3}{4}\frac{f^{(5)}(c)}{5!}{h}^4+\frac{15}{16}\frac{f^{(7)}(c)}{7!}{h}^6+\mathrm{\cdots },$$

and so

$$f^{\prime }(c)=\frac{2^2\varphi (h/2)-\varphi (h)}{2^2-1}+\frac{1}{4}\frac{f^{(5)}(c)}{5!}{h}^4+\frac{5}{16}\frac{f^{(7)}(c)}{7!}{h}^6+\mathrm{\cdots }.$$

We have taken a linear combination of two instances of the same approximation method for $f^{\prime }(c)$ to obtain a new approximation of higher order. This process is called Richardson extrapolation.

Now we will do the same process, but with slightly different notation in order to illustrate the general principle. Write ${\varphi }_0(h)=\varphi (h)$, so

$L$ $={\varphi }_0(h)+{\displaystyle \sum _{k=1}^{\mathrm{\infty }}}{a}_{k,0}{h}^{2k}$

$={\varphi }_0(h/2)+{\displaystyle \sum _{k=1}^{\mathrm{\infty }}}{a}_{k,0}{(h/2)}^{2k},$

where $a_{k,0}=-\frac{f^{(2k+1)}(c)}{(2k+1)!}$. We want to cancel the $k=1$ term in the error, which for ${\varphi }_0(h)$ is $a_{1,0}{h}^2$ and for ${\varphi }_0(h/2)$ is $a_{1,0}{\left(\frac{1}{2}\right)}^2{h}^2$, and so we take $2^2$ times ${\varphi }_0(h/2)$ minus ${\varphi }_0(h)$, and then normalize, giving

$$L=\frac{2^2{\varphi }_0(h/2)-{\varphi }_0(h)}{2^2-1}+\sum _{k=2}^{\mathrm{\infty }}\frac{2^{2-2k}-1}{2^2-1}{a}_{k,0}{h}^{2k}={\varphi }_1(h)+\sum _{k=2}^{\mathrm{\infty }}{a}_{k,1}{h}^{2k},$$

where ${\varphi }_1(h)=(2^2{\varphi }_0(h/2)-{\varphi }_0(h))/(2^2-1)$ and $a_{k,1}=\frac{2^{2-2k}-1}{2^2-1}{a}_{k,0}$. Note that the ${\varphi }_1(h)$ approximation has higher order of convergence, since the error term starts at $k=2$. The error scales as $h^4$, so the convergence is fourth order. The important thing to notice here is that this process can now be repeated to cancel the leading order term in the error for ${\varphi }_1(h)$. Richardson extrapolation can be applied to $L={\varphi }_1(h)+{\sum }_{k=2}^{\mathrm{\infty }}{a}_{k,1}{h}^{2k}$ where now we need to cancel the $k=2$ term in the remainder. By the same procedure, we need to take $2^4{\varphi }_1(h/2)-{\varphi }_1(h)$ and normalize by $2^4-1$ in the denominator, giving

$$L=\frac{2^4{\varphi }_1(h/2)-{\varphi }_1(h)}{2^4-1}+\sum _{k=3}^{\mathrm{\infty }}\frac{2^{4-2k}-1}{2^4-1}{a}_{k,1}{h}^{2k}={\varphi }_2(h)+\sum _{k=3}^{\mathrm{\infty }}{a}_{k,2}{h}^{2k},$$

where ${\varphi }_2(h)=(2^4{\varphi }_1(h/2)-{\varphi }_1(h))/(2^4-1)$ and $a_{k,2}=\frac{2^{4-2k}-1}{2^4-1}{a}_{k,1}$. Now, the error term starts at $k=3$, so the error for ${\varphi }_2(h)$ scales as $h^6$ and the convergence is sixth order. This process could be repeated again, canceling the $k=3$ term in the error associated with ${\varphi }_2$, and so on. Notice that we did not need to know the coefficients $a_{k,n}$ of the error term. We just needed to know the general form of the error. This procedure is formalized in the following theorem.

Theorem 1 (Richardson extrapolation, even powers).

Suppose that, for all $h>0$ sufficiently small, the function $\varphi (h)$ approximates the number $L$ with error expansion

$$L=\varphi (h)+\sum _{k=1}^{\mathrm{\infty }}{a}_k{h}^{2k}$$ (5)

for some coefficients $a_k$. Taking ${\varphi }_0=\varphi$, we define the $n$th Richardson extrapolation, ${\varphi }_n$, recursively:

$${\varphi }_n(h)=\frac{2^{2n}{\varphi }_{n-1}(h/2)-{\varphi }_{n-1}(h)}{2^{2n}-1}={\varphi }_{n-1}(h/2)+\frac{{\varphi }_{n-1}(h/2)-{\varphi }_{n-1}(h)}{2^{2n}-1}\text{for }n\ge 1.$$ (6)

If we set $a_{k,0}=a_k$ for $k\ge 1$, then

$$L={\varphi }_n(h)+\sum _{k=n+1}^{\mathrm{\infty }}{a}_{k,n}{h}^{2k}\text{for }n\ge 0,$$ (7)

where $a_{k,n}=\frac{2^{2n-2k}-1}{2^{2n}-1}{a}_{k,n-1}$ for $k\ge n+1$.

Proof.

We prove this by induction on $n$. The base case $n=0$ is just the statement (5), so it holds by assumption. Fix $n\ge 1$ and assume that

$$L={\varphi }_{n-1}(h)+\sum _{k=n}^{\mathrm{\infty }}{a}_{k,n-1}{h}^{2k}.$$

We have

$2^{2n}L$ $=2^{2n}{\varphi }_{n-1}(h/2)+a_{n,n-1}{h}^{2n}+{\displaystyle \sum _{k=n+1}^{\mathrm{\infty }}}{2}^{2n-2k}{a}_{k,n-1}{h}^{2k},$

$L$ $={\varphi }_{n-1}(h)+a_{n,n-1}{h}^{2n}+{\displaystyle \sum _{k=n+1}^{\mathrm{\infty }}}{a}_{k,n-1}{h}^{2k}.$

Subtracting the second from the first, and then dividing by $2^{2n}-1$, we obtain

$$L=\frac{2^{2n}{\varphi }_{n-1}(h/2)-{\varphi }_{n-1}(h)}{2^{2n}-1}+\sum _{k=n+1}^{\mathrm{\infty }}\frac{2^{2n-2k}-1}{2^{2n}-1}{a}_{k,n-1}{h}^{2k}.$$

But this is precisely (7) because of how we recursively defined ${\varphi }_n$ and $a_{k,n}$. ∎

Example 1 (Richardson extrapolation table).

In the context of Theorem 1, we fix (sufficiently small) $h>0$ and define the numbers $R(m,n)={\varphi }_n(2^{n-m}h)$ for $0\le n\le m$. Fixing $N>1$, we may fill out a Richardson extrapolation table,

It holds that

$$R(m,n)=\frac{2^{2n}R(m,n-1)-R(m-1,n-1)}{2^{2n}-1},$$ (8)

so only the initial column $R(m,0)={\varphi }_0(2^{-m}h)=\varphi (2^{-m}h)$ needs to be computed using $\varphi$; all other columns may be computed by the given recursion. For each fixed $n$, the entries $R(m,n)$ are all $𝒪(h^{2n+2})$ approximations of $L$.

Let $L=1/3=f^{\prime }(c)$, where $f(x)=\arctan x$, $c=\sqrt{2}$, and $\varphi (h)=(f(c+h)-f(c-h))/(2h)$. We fill out the first column of the table with $\varphi (1),\varphi (1/2),\mathrm{\dots },\varphi (1/32)$ and use the recursion (8) to complete the Richardson table. The absolute errors $E(m,n)=|L-R(m,n)|$, $0\le n\le m\le 5$, are given below.

	$n=0$	$n=1$	$n=2$	$n=3$	$n=4$	$n=5$
$m=0$	$5.937\times {10}^{-2}$
$m=1$	$1.544\times {10}^{-2}$	$7.950\times {10}^{-4}$
$m=2$	$3.861\times {10}^{-3}$	$1.504\times {10}^{-6}$	$5.139\times {10}^{-5}$
$m=3$	$9.647\times {10}^{-4}$	$5.868\times {10}^{-7}$	$7.262\times {10}^{-7}$	$7.802\times {10}^{-8}$
$m=4$	$2.411\times {10}^{-4}$	$4.688\times {10}^{-8}$	$1.088\times {10}^{-8}$	$4.743\times {10}^{-10}$	$1.702\times {10}^{-10}$
$m=5$	$6.028\times {10}^{-5}$	$3.087\times {10}^{-9}$	$1.681\times {10}^{-10}$	$1.965\times {10}^{-12}$	$1.130\times {10}^{-13}$	$5.322\times {10}^{-14}$

The ratios of successive errors in the initial (zeroth) column quickly approach $2^2=4$, as expected, which corresponds to quadratic convergence, $𝒪(h^2)$. In the next column, the ratios of successive errors are getting closer to the expected value of $2^4=16$ that corresponds to quartic convergence, $𝒪(h^4)$. More generally, we have $E(m,n)=𝒪(h^{2(n+1)})$, so we expect $E(m-1,n)/E(m,n)\to 2^{2(n+1)}$ as $m\to \mathrm{\infty }$ for each fixed $n$. $\mathrm{◆}$

1.3 Sequential Orders of Convergence

[Textbook reference: §1.3]

Along with the definitions of orders of convergence considered previously, there are a set of different concepts that are also used to classify and quantify how quickly a sequence $\{x_n\}$ approaches its limit $x$. We will postpone most of this discussion until after seeing some examples of root-finding methods. However, we will just mention presently that terms such as “linear convergence”, “quadratic convergence,” and so on, have meanings that are different from the “order of convergence” defined previously, even though some authors use terms like linear and quadratic convergence to refer to the previous definition. This can be confusing since similar or identical terms can be used to refer to very different things. We will try to avoid this confusion by being consistent with our terminology, but you may sometimes need to pay attention to the context to understand and correctly interpret which concept is being referred to.