Extrema

In this part of the course we work on the following skills:

• Locating and classifying the extrema of scalar fields.
• Applying Lagrange's multipliers method to optimize quantities with respect to constraints.

See also the graded exercises and additional exercises associated to this part of the course.

In the previous chapter we introduced various notions of differentials for higher dimensional functions (scalar fields, vector fields, paths, etc.). This part of the course is devoted to searching for extrema (minima / maxima) in various different scenarios. This extends what we already know for functions in $\mathbb{R}$ and we will find that in higher dimensions more possibilities and subtleties exist.

Extrema (minima / maxima / saddle)

Let $S\subset {\mathbb{R}}^n$ be open, $f:S\to \mathbb{R}$ be a scalar field and $\mathbf{a}\in S$.

Definition

If $f(\mathbf{a})\le f(\mathbf{x})$ (resp. $f(\mathbf{a})\ge f(\mathbf{x})$) for all $\mathbf{x}\in S$, then $f(\mathbf{a})$ is said to be the absolute minimum (resp. maximum) of $f$.

Definition

If $f(\mathbf{a})\le f(\mathbf{x})$ (resp. $f(\mathbf{a})\ge f(\mathbf{x})$) for all $\mathbf{x}\in B(\mathbf{a},r)$ for some $r>0$, then $f(\mathbf{a})$ is said to be a relative minimum (resp. maximum) of $f$.

Collectively we call the these points the extrema of the scalar field. In the case of a scalar field defined on ${\mathbb{R}}^2$ we can visualize the scalar field as a 3D plot like the figure. Here we see the extrema as the "flat" places. We sometimes use global as a synonym of absolute and local as a synonym of relative.

$f(x,y)=x{e}^{-(x^2{y}^2)}+\frac{1}{4}{e}^{y^{\frac{3}{10}}}$

To proceed it is convenient to connect the extrema with the behaviour of the gradient of the scalar field.

Theorem

If $f:S\to \mathbb{R}$ is differentiable and has a relative minimum or maximum at $\mathbf{a}$, then $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$.

Proof

Suppose $f$ has a relative minimum at $\mathbf{a}$ (or consider $-f$). For any unit vector $\mathbf{v}$ let $g(u)=f(\mathbf{a}+u\mathbf{v})$. We know that $g:\mathbb{R}\to \mathbb{R}$ has a relative minimum at $u=0$ so $u^{\prime }(0)=0$. This means that the directional derivative $D_{\mathbf{v}}f(\mathbf{a})=0$ for every $\mathbf{v}$. Consequently this means that $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$.

$\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$ doesn't imply a minimum or maximum at $\mathbf{a}$, even in $\mathbb{R}$, as seen with the function $f(x)=x^3$. In higher dimensions even more is possible.

Observe that, here and in the subsequent text, we can always consider the case of $f:\mathbb{R}\to \mathbb{R}$, i.e., the case of ${\mathbb{R}}^n$ where $n=1$. Everything still holds and reduces to the arguments and formulae previously developed for functions of one variable.

Definition (stationary point)

If $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$ then $\mathbf{a}$ is called a stationary point.

If $f(x,y)=x^2+y^2$ then $\mathrm{\nabla }f(x,y)=\left(\begin{array}{c}2x\\ 2y\end{array}\right)$ and $\mathrm{\nabla }f(0,0)=\left(\begin{array}{c}0\\ 0\end{array}\right)$. The point $(0,0)$ is an absolute minimum for $f$.

As we see in the inflection example, the converse of the above theorem fails in the sense that a stationary point might not be a minimum or a maximum. The motivates the following.

Definition (saddle point)

If $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$ and $\mathbf{a}$ is neither a minimum nor a maximum then $\mathbf{a}$ is said to be a saddle point.

The quintessential saddle has the shape seen in the graph. However it might be similar to an inflection in 1D or more complicated using the possibilities available in higher dimension.

If $f(x,y)=x^2-y^2$ then $\mathrm{\nabla }f(x,y)=\left(\begin{array}{c}2x\\ -2y\end{array}\right)$ and $\mathrm{\nabla }f(0,0)=\mathbf{0}$. The point $(0,0)$ is a saddle point for $f$.

Hessian matrix

To proceed it is useful to develop the idea of a second order Taylor expansion in this higher dimensional setting. In particular this will allow us to identify the local behaviour close to stationary points. The main object for doing this is the Hessian matrix. Let $f:{\mathbb{R}}^2\to \mathbb{R}$ be twice differentiable and use the notation $f(x,y)$. The Hessian matrix at $\mathbf{a}\in {\mathbb{R}}^2$ is defined as

$$\mathbf{H}f(\mathbf{a})=\left(\begin{array}{cc}\frac{{\partial }^{2}f}{\partial x^2}(\mathbf{a})& \frac{{\partial }^{2}f}{\partial x\partial y}(\mathbf{a})\\ \frac{{\partial }^{2}f}{\partial y\partial x}(\mathbf{a})& \frac{{\partial }^{2}f}{\partial y^2}(\mathbf{a})\end{array}\right).$$

Observe that the Hessian matrix $\mathbf{H}f(\mathbf{a})$ is a symmetric matrix since we know that

$$\frac{{\partial }^{2}f}{\partial x\partial y}(\mathbf{a})=\frac{{\partial }^{2}f}{\partial y\partial x}(\mathbf{a})$$

for twice differentiable functions.

The Hessian matrix is defined analogously in any dimension.

Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be twice differentiable. The Hessian matrix at $\mathbf{a}\in {\mathbb{R}}^n$ is defined as

$$\mathbf{H}f(\mathbf{a})=\left(\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}(\mathbf{a})& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}(\mathbf{a})& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}(\mathbf{a})\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}(\mathbf{a})& \frac{{\partial }^{2}f}{\partial x_2^2}(\mathbf{a})& \cdots & \frac{{\partial }^{2}f}{\partial x_2\partial x_n}(\mathbf{a})\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}(\mathbf{a})& \frac{{\partial }^{2}f}{\partial x_n\partial x_2}(\mathbf{a})& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}(\mathbf{a})\end{array}\right).$$

Observe that the Hessian matrix is a real symmetric matrix in any dimension. If $f:\mathbb{R}\to \mathbb{R}$ then $\mathbf{H}f(a)$ is a $1\times 1$ matrix and coincides with the second derivative of $f$. In this sense what we know about extrema in $\mathbb{R}$ is just a special case of everything we do here.

As an example, let $f(x,y)=x^2-y^2$ (figure). The gradient and the Hessian are respectively

$$\begin{array}{rl}\mathrm{\nabla }f(x,y)& =\left(\begin{array}{c}\frac{\partial f}{\partial x}(x,y)\\ \frac{\partial f}{\partial y}(x,y)\end{array}\right)=\left(\begin{array}{c}2x\\ -2y\end{array}\right),\\ \mathbf{H}f(x,y)& =\left(\begin{array}{cc}\frac{{\partial }^{2}f}{\partial x^2}(x,y)& \frac{{\partial }^{2}f}{\partial x\partial y}(x,y)\\ \frac{{\partial }^{2}f}{\partial y\partial x}(x,y)& \frac{{\partial }^{2}f}{\partial y^2}(x,y)\end{array}\right)=\left(\begin{array}{cc}2& 0\\ 0& -2\end{array}\right).\end{array}$$

The point $(0,0)$ is a stationary point since $\mathrm{\nabla }f(0,0)=\left(\begin{array}{c}0\\ 0\end{array}\right)$. In this example $\mathbf{H}f$ does not depend on $(x,y)$ but in general we can expect dependence and so it gives a different matrix at different points $(x,y)$.

Theorem

If $\mathbf{v}=(v_1,\dots ,v_n)$ then,

$$\mathbf{v}\mathbf{H}f(\mathbf{a}){\mathbf{v}}^{\mathbf{T}}=\sum _{j,k=0}^n\frac{{\partial }^{2}f}{\partial x_j\partial x_k}(\mathbf{a})v_j{v}_k\in \mathbb{R}.$$

Proof

Multiplying the matrices we calculate that

$$\begin{array}{rl}\mathbf{v}\mathbf{H}f(\mathbf{a}){\mathbf{v}}^{\mathbf{T}}& =\left(\begin{array}{ccc}{v}_1& \cdots & v_n\end{array}\right)\left(\begin{array}{ccc}{\partial }_1{\partial }_{1}f(\mathbf{a})& \cdots & {\partial }_1{\partial }_{n}f(\mathbf{a})\\ ⋮& \ddots & ⋮\\ {\partial }_n{\partial }_{1}f(\mathbf{a})& \cdots & {\partial }_n{\partial }_{n}f(\mathbf{a})\end{array}\right)\left(\begin{array}{c}{v}_1\\ ⋮\\ v_n\end{array}\right)\\ & =\sum _{j,k=0}^n{\partial }_j{\partial }_{k}f(\mathbf{a})v_j{v}_k\end{array}$$

as required.

Second order Taylor formula for scalar fields

First let's recall the first order Taylor approximation we saw before. If $f$ is differentiable at $\mathbf{a}$ then

$$f(\mathbf{x})\approx f(\mathbf{a})+\mathrm{\nabla }f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a}).$$

If $\mathbf{a}$ is a stationary point then this only tells us that $f(\mathbf{x})\approx f(\mathbf{a})$ so a natural next question is to search for slightly more detailed information.

Theorem (second order Taylor for scalar fields)

Let $f$ be a scalar field twice differentiable on $B(\mathbf{a},r)$. Then, for $\mathbf{x}$ close to $\mathbf{a}$,

$$f(\mathbf{x})\approx f(\mathbf{a})+\mathrm{\nabla }f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a})+\frac{1}{2}(\mathbf{x}-\mathbf{a})\mathbf{H}f(\mathbf{a})(\mathbf{x}-\mathbf{a}{)}^{\mathbf{T}}$$

in the sense that the error is $o({\Vert (\mathbf{x}-\mathbf{a})\Vert }^2)$.

Proof

Let $\mathbf{v}=\mathbf{x}-\mathbf{a}$ and let $g(u)=f(\mathbf{a}+u\mathbf{v})$. The Taylor expansion of $g$ tells us that $g(1)=g(0)+g^{\prime }(0)+\frac{1}{2}{g}^{″}(c)$ for some $c\in (0,1)$. Since $g(u)=f(a_1+u{v}_1,\dots ,a_n+u{v}_n)$, by the chain rule,

$$\begin{array}{rl}{g}^{\prime }(u)& =\sum _{j=1}^n{\partial }_{j}f(a_1+u{v}_1,\dots ,a_n+u{v}_n)v_j=\mathrm{\nabla }f(\mathbf{a}+u\mathbf{v})\cdot \mathbf{v},\\ g^{″}(u)& =\sum _{j,k=1}^n{\partial }_j{\partial }_{k}f(a_1+u{v}_1,\dots ,a_n+u{v}_n)v_j{v}_k\\ & ={\mathbf{v}}^{\mathbf{T}}\mathbf{H}f(\mathbf{a}+u\mathbf{v})\mathbf{v}.\end{array}$$

Consequently $f(\mathbf{a}+\mathbf{v})=f(\mathbf{a})+\mathrm{\nabla }f(\mathbf{a})\cdot \mathbf{v}+\frac{1}{2}{\mathbf{v}}^{\mathbf{T}}\mathbf{H}f(\mathbf{a}+c\mathbf{v})\mathbf{v}$. We define the "error" in the approximation as $ϵ(\mathbf{v})=\frac{1}{2}{\mathbf{v}}^{\mathbf{T}}(\mathbf{H}f(\mathbf{a}+c\mathbf{v})-\mathbf{H}f(\mathbf{a}))\mathbf{v}$ and estimate that

$$|ϵ(\mathbf{v})|\le \sum _{j,k=0}^n{v}_j{v}_k({\partial }_j{\partial }_{k}f(\mathbf{a}+c\mathbf{v})-{\partial }_j{\partial }_{k}f(\mathbf{a})).$$

Since $\left|v_j{v}_k\right|\le {\Vert \mathbf{v}\Vert }^2$ we observe that $\frac{|ϵ(\mathbf{v})|}{{\Vert \mathbf{v}\Vert }^2}\to 0$ as $\Vert \mathbf{v}\Vert \to 0$ as required.

Classifying stationary points

In order to classify the stationary points we will take advantage of the Hessian matrix and therefore we need to first understand the follow fact about real symmetric matrices.

Theorem

Let $A$ be a real symmetric matrix and let $Q(\mathbf{v})={\mathbf{v}}^{\mathbf{T}}A\mathbf{v}$. Then,

• $Q(\mathbf{v})>0$ for all $\mathbf{v}\ne \mathbf{0}$ $⟺$ all eigenvalues of $A$ are positive,
• $Q(\mathbf{v})<0$ for all $\mathbf{v}\ne \mathbf{0}$ $⟺$ all eigenvalues of $A$ are negative.

Proof

Since $A$ is symmetric it can be diagonalised by matrix $B$ which is orthogonal ($B^{\mathbf{T}}=B^{-1}$) and the diagonal matrix $D=B^{\mathbf{T}}AB$ has the eigenvalues of $A$ as the diagonal. This means that $Q(\mathbf{v})={\mathbf{v}}^{\mathbf{T}}{B}^{\mathbf{T}}BA{B}^{\mathbf{T}}B\mathbf{v}={\mathbf{w}}^{\mathbf{T}}D\mathbf{w}$ where $\mathbf{w}=B\mathbf{v}$. Consequently $Q(\mathbf{v})=\sum _j{\lambda }_j{w}_j^2$. Observe that, if all ${\lambda }_j>0$ then $\sum _j{\lambda }_j{w}_j^2>0$.

In order to prove the other direction in the "if and only if" statement, observe that $Q(B{\mathbf{u}}_k)={\lambda }_k$. This means that, if $Q(\mathbf{v})>0$ for all $\mathbf{v}\ne \mathbf{0}$ then ${\lambda }_k>0$ for all $k$.

Theorem (classifying stationary points)

Let $f$ be a scalar field twice differentiable on $B(\mathbf{a},r)$. Suppose $\mathrm{\nabla }f(\mathbf{a})=\mathbf{0}$ and consider the eigenvalues of $\mathbf{H}f(\mathbf{a})$. Then,

• All eigenvalues are positive $⟹$ relative minimum at $\mathbf{a}$,
• All eigenvalues are negative $⟹$ relative maximum at $\mathbf{a}$,
• Some positive, some negative $⟹$ $\mathbf{a}$ is a saddle point.

Proof

Let $Q(\mathbf{v})={\mathbf{v}}^{\mathbf{T}}\mathbf{H}f(\mathbf{a})\mathbf{v}$, $\mathbf{w}=B\mathbf{v}$ and let $\mathrm{\Lambda }:=\underset{j}{min}{\lambda }_j$. Observe that $\Vert \mathbf{w}\Vert =\Vert \mathbf{v}\Vert$ and that $Q(\mathbf{v})=\sum _j{\lambda }_j{w}_j^2\ge \mathrm{\Lambda }\sum _j{w}_j^2=\mathrm{\Lambda }{\Vert \mathbf{v}\Vert }^2$. We have them 2^nd-order Taylor

$$\begin{array}{rl}f(\mathbf{a}+\mathbf{v})-f(\mathbf{a})& =\frac{1}{2}{\mathbf{v}}^{\mathbf{T}}\mathbf{H}f(\mathbf{a})\mathbf{v}+ϵ(\mathbf{v})\\ & \ge (\frac{\mathrm{\Lambda }}{2}-\frac{ϵ(\mathbf{v})}{{\Vert \mathbf{v}\Vert }^2}){\Vert \mathbf{v}\Vert }^2.\end{array}$$

Since $\frac{|ϵ(\mathbf{v})|}{{\Vert \mathbf{v}\Vert }^2}\to 0$ as $\Vert \mathbf{v}\Vert \to 0$, $\frac{|ϵ(\mathbf{v})|}{{\Vert \mathbf{v}\Vert }^2}<\frac{\mathrm{\Lambda }}{2}$ when $\Vert \mathbf{v}\Vert$ is small. The argument is analogous for the second part. For final part consider ${\mathbf{v}}_j$ which is the eigenvector for ${\lambda }_j$ and apply the argument of the first or second part.

Attaining extreme values

Here we explore the extreme value theorem for continuous scalar fields. The argument will be in two parts: Firstly we show that continuity implies boundedness; Secondly we show that boundedness implies that the maximum and minimum are attained. We use the following notation for interval / rectangle / cuboid / tesseract, etc. If $\mathbf{a}=(a_1,\dots ,a_n)$ and $\mathbf{b}=(b_1,\dots ,b_n)$ then we consider the $n$-dimensional closed Cartesian product

$$[\mathbf{a},\mathbf{b}]=[a_1,b_1]\times \cdots \times [a_n,b_n].$$

We call this set a rectangle (independent of the dimension). As a first step it is convenient to know that all sequences in our setting have convergent subsequences.

Theorem

If ${\{{\mathbf{x}}_n\}}_n$ is a sequence in $[\mathbf{a},\mathbf{b}]$ there exists a convergent subsequence ${\{{\mathbf{x}}_{n_j}\}}_j$.

Proof

In order to prove the theorem we construct the subsequence. Firstly we divide $[\mathbf{a},\mathbf{b}]$ into sub-rectangles of size half the original. We then choose a sub-rectangle which contains infinite elements of the sequence and choose the first of these elements to be part of the sub-sequence. We repeat this process by again dividing the sub-rectangle we chose by half and choosing the next element of the subsequence. We repeat to give the full subsequence.

Theorem

Suppose that $f$ is a scalar field continuous at every point in the closed rectangle $[\mathbf{a},\mathbf{b}]$. Then $f$ is bounded on $[\mathbf{a},\mathbf{b}]$ in the sense that there exists $C>0$ such that $|f(\mathbf{x})|\le C$ for all