MTH 451/551
Lecture 4

1 Solving Equations and Root Finding

[Textbook reference: §2.1]

We are interested in finding the solution to the system $f(x)=0$ for some function $f(x)$, i.e. the roots of $f$. Depending on the specific form of the function $f(x)$, we may, through some complicated algebraic manipulations, be able to find a closed-form expression for the roots. However, this is a laborious process, and we are not always guaranteed to succeed. For example, as mentioned at the beginning of the course, there is no general closed-form expression (written in terms of radicals) for the roots of polynomials of degree 5 or higher. Therefore, it is of great utility to develop algorithms that approximate the roots, using only evaluations of $f$ (and perhaps its derivatives), without symbolic or algebraic manipulations.

1.1 Root Bracketing Methods

Two of the oldest and most basic methods for root finding are the bisection and regula falsi methods, which are both root bracketing methods. This means that if there is a root $r$ of $f$ in the interval $(a,b)$ (we say that $a$ and $b$ bracket the root, since they lie on either side of it), then we generate a nested sequence of intervals $(a_n,b_n)$, each of which contain the root $r$. Within each interval, we will obtain an approximate root $x_n$ that converges to $r$ as $n\to \mathrm{\infty }$.

1.1.1 The Bisection Method

[Note: we differ slightly from the textbook here by using closed intervals and non-strict inequalities.]

We consider a continuous function $f\in C[a,b]$, such that $f(a)f(b)\le 0$. This means that either $f(a)$ and $f(b)$ have opposite signs, or at least one of them is zero. By the intermediate value theorem, this implies that there exists a root of $f$ in the interval $[a,b]$. The bisection method generates a sequence $x_n$ that converges to a root $r\in [a,b]$.

1. Initialize $[a_0,b_0]=[a,b]$, $x_0=(a_0+b_0)/2$.

2. Iterate .

To turn this into a practical algorithm, we need to set some termination criteria. The sequence $x_n\to r$, and we need to decide when to stop iterating, and accept the current value of $x_n$ as a good enough approximation to $r$ (very rarely we may find $x_n$ such that $f(x_n)=0$ exactly, but this does not happen usually). Typically, we set both a tolerance and maximum number of iterations. In the version of the algorithm presented below, we provide two tolerances and a maximum number of iterations. We terminate if either $|f(x_n)|\le \mathrm{ftol}$ or if $|x_n-r|\le \mathrm{xtol}$. Even though we don’t know the exact value of $r$ (if we did, we wouldn’t need an algorithm to approximate it), we can bound $|x_n-r|$ since $r$ is guaranteed to lie in every interval we generate.

Pseudocode version of the bisection algorithm: return an approximation $x_n$ of a root $r\in [a,b]$ of $f(x)$, given that $f(a)f(b)\le 0$.

1:function Bisection($f,a,b,\mathrm{ftol},\mathrm{xtol},\mathrm{maxit}$)

2:     $x\leftarrow (a+b)/2$

3:     for $n=1,2,\mathrm{\dots },\mathrm{maxit}$ do

4:         if $f(a)f(x)\le 0$ then

5:              $b\leftarrow x$

6:         else

7:              $a\leftarrow x$

8:         end if

9:         $x\leftarrow (a+b)/2$

10:         return $x$ if $|f(x)|\le \mathrm{ftol}$ or $(b-a)/2\le \mathrm{xtol}$

11:     end for

12:     Bisection did not converge in maximum number of iterations.

13:end function

The following theorem guarantees that the bisection method always converges to a root, and gives an a priori bound on the error at each iteration.

Theorem 1 (Convergence of the bisection method; Textbook Theorem 2.2).

Suppose that $f\in C[a,b]$ with $f(a)f(b)\le 0$. If $\{a_n,x_n,b_n\}$ is generated by the bisection method, then $x_n\to r\in [a,b]$, where $f(r)=0$. Furthermore, we have the error bound

$$|x_n-r|\le 2^{-(n+1)}(b-a)\text{for all }n\text{.}$$

Proof.

First, we show by induction that $f(a_n)f(b_n)\le 0$ for all $n$. This holds for $n=0$ by assumption. Suppose it holds for some $n$. If $f(a_n)f(x_n)\le 0$, then $[a_{n+1},b_{n+1}]=[a_n,x_n]$, and so $f(a_{n+1})f(b_{n+1})=f(a_n)f(x_n)\le 0$. Otherwise, $f(a_n)f(x_n)>0$, so $f(a_n)$ and $f(x_n)$ are nonzero and have the same sign. Since $f(a_n)f(b_n)\le 0$, it follows that $f(x_n)f(b_n)\le 0$, and so $f(a_{n+1})f(b_{n+1})=f(x_n)f(b_n)\le 0$.

By construction, $a_n\le x_n\le b_n$, and the intervals are nested, so

$$a\le a_n\le a_{n+1}\le b_{n+1}\le b_n\le b$$

for all $n$. The sequence $\{a_n\}$ is nondecreasing and bounded above by $b$, and the sequence $\{b_n\}$ is nonincreasing and bounded below by $a$. By the Monotone Convergence Theorem, there exist $\overline{a},\overline{b}\in [a,b]$ such that $a_n\to \overline{a}$ and $b_n\to \overline{b}$. Furthermore, since each step halves the length of the interval, $b_{n+1}-a_{n+1}=(b_n-a_n)/2$, we see that

$$b_n-a_n=2^{-n}(b-a)$$

for all $n$. Taking the limit as $n\to \mathrm{\infty }$, we find $\overline{b}-\overline{a}=0$, and so $\overline{a}=\overline{b}$. We take $r$ to be this common value.

Since $f$ is continuous at $r$, we have $f(a_n)f(b_n)\to {[f(r)]}^2$. Since $f(a_n)f(b_n)\le 0$ for all $n$, the limit must also satisfy ${[f(r)]}^2\le 0$. But clearly ${[f(r)]}^2\ge 0$, so we must have $f(r)=0$.

Finally, since $x_n$ is the midpoint of $[a_n,b_n]$ and $r\in [a_n,b_n]$, the distance from $x_n$ to $r$ is at most half the length of the interval,

$$|x_n-r|\le \frac{b_n-a_n}{2}=2^{-(n+1)}(b-a).$$

The right-hand side tends to zero as $n\to \mathrm{\infty }$, and so $x_n\to r$. ∎

1.1.2 Regula Falsi

The regula falsi method (latin for false position) is a simple variation on the bisection method. Like bisection, it is a root bracketing method. The “false positions” in this case are the two endpoints, which are generally not roots themselves, but are used to help find the roots. Bisection proceeded by always choosing the midpoint of the interval, leading to a halving of the error bound at every step. This had no dependence on the function $f$ whatsoever. Regula falsi, on the other hand, chooses $x_n$ to be the zero of the secant line connecting $(a_n,f(a_n))$ and $(b_n,f(b_n))$. This uses more information about the function $f$, and can often converge much faster than bisection, but lacks the same guaranteed error reduction rate.

The setup is the same as in bisection, except here we require that strictly in order for the secant line to have nonzero slope. The method is summarized as follows:

1. Initialize $[a_0,b_0]=[a,b]$, $x_0={\displaystyle \frac{a_{0}f(b_0)-b_{0}f(a_0)}{f(b_0)-f(a_0)}}$.

2. Iterate  $x_{n+1}={\displaystyle \frac{a_{n+1}f(b_{n+1})-b_{n+1}f(a_{n+1})}{f(b_{n+1})-f(a_{n+1})}}$.

The formula for $x_n$ is the zero of the secant line through $(a_n,f(a_n))$ and $(b_n,f(b_n))$. It can equivalently be written as

$$x_n=a_n-\frac{b_n-a_n}{f(b_n)-f(a_n)}f(a_n)=b_n-\frac{b_n-a_n}{f(b_n)-f(a_n)}f(b_n).$$

The termination criteria are the same as for bisection, with one important difference. Since $r$ and $x_n$ both lie in $[a_n,b_n]$, we can still bound the error by $|x_n-r|\le \max (x_n-a_n,b_n-x_n)$. However, unlike in bisection, the length of the interval $[a_n,b_n]$ does not necessarily go to zero. Often, one of the endpoints will stay fixed for all iterations. In that case, this error bound will never drop below $\mathrm{xtol}$, and the iteration will be terminated by $\mathrm{ftol}$ (or $\mathrm{maxit}$) instead.

Pseudocode version of the regula falsi algorithm: return an approximation $x_n$ of a root $r\in (a,b)$ of $f(x)$, given that .

1:function RegulaFalsi($f,a,b,\mathrm{ftol},\mathrm{xtol},\mathrm{maxit}$)

2:     $x\leftarrow \left(af(b)-bf(a)\right)/\left(f(b)-f(a)\right)$

3:     for $n=1,2,\mathrm{\dots },\mathrm{maxit}$ do

4:         if  then

5:              $b\leftarrow x$

6:         else

7:              $a\leftarrow x$

8:         end if

9:         $x\leftarrow \left(af(b)-bf(a)\right)/\left(f(b)-f(a)\right)$

10:         return $x$ if $|f(x)|\le \mathrm{ftol}$ or $\max (x-a,b-x)\le \mathrm{xtol}$

11:     end for

12:     Regula falsi did not converge in maximum number of iterations.

13:end function

The following theorem guarantees that the regula falsi method always converges to a root. In contrast to the bisection method, there is no a priori bound on the error.

Theorem 2 (Convergence of the regula falsi method; Textbook Theorem 2.7).

Suppose that $f\in C[a,b]$ with . If $\{a_n,x_n,b_n\}$ is generated by the regula falsi method, then $x_n\to r\in (a,b)$, where $f(r)=0$.

Proof.

We only consider the non-trivial case, where $f(x_n)\ne 0$ for all $n$. We lose no generality by assuming that and $f(b)>0$, which guarantees that and $f(b_n)>0$ for all $n$. In particular, $x_n$ is well-defined, and since the secant line changes sign between $a_n$ and $b_n$, we have .

As in the proof for bisection, $\{a_n\}$ is nondecreasing and $\{b_n\}$ is nonincreasing, so $a_n\to a^{\ast }$ and $b_n\to b^{\ast }$ for some $a^{\ast }\le b^{\ast }$. By continuity, $f(a^{\ast })\le 0\le f(b^{\ast })$. If $a^{\ast }=b^{\ast }$, then we take $r$ to be this common value, and $f(r)=0$ and $x_n\to r$ by the same argument as for bisection.

Suppose instead that . Since $x_n$ is either $a_{n+1}\le a^{\ast }$ or $b_{n+1}\ge b^{\ast }$, no $x_n$ lies in the interval $(a^{\ast },b^{\ast })$. If $f(a^{\ast })\ne f(b^{\ast })$, then by continuity,

$$x_n\to x^{\ast }=\frac{a^{\ast }f(b^{\ast })-b^{\ast }f(a^{\ast })}{f(b^{\ast })-f(a^{\ast })}.$$

If $f(a^{\ast })=0$, then $x^{\ast }=a^{\ast }$, and we take $r=a^{\ast }$. Similarly, if $f(b^{\ast })=0$, then $x^{\ast }=b^{\ast }$, and we take $r=b^{\ast }$. If both and $f(b^{\ast })>0$, then $x^{\ast }$ lies strictly between $a^{\ast }$ and $b^{\ast }$, which is impossible, since no $x_n$ lies in $(a^{\ast },b^{\ast })$. The only remaining possibility is $f(a^{\ast })=f(b^{\ast })=0$ with . This is clearly impossible if $f$ has only one root in $(a,b)$; the general case is more involved (see textbook §2.1, Exercise 13). ∎