Finding Extrema of Multivariable Functions

1. Remember for a curve {y=f(x)}, we have maxima and minima occur whenever
(1)\displaystyle  f'(x_{0}) = 0
What’s the multivariable analog to this notion?

2. Definition. If {\vec{\nabla}f(\vec{x}_{0})=\vec{0}}, we say {\vec{x}_{0}} is a “Critical Point” of {f}.

Example 1. Consider {f(x,y) = x^{2}+y^{2} - 2x - 8y}. What are its critical points?

Solution: We find its gradient first
(2)\displaystyle  \vec{\nabla}f = \langle 2x - 2, 2y - 8\rangle
Next we need to set each component to vanish. This implies
(3)\displaystyle  \vec{x}_{0} = \langle 1, 4\rangle
is the only critical point.

3. Problem: How do we determine if a critical point describes a maxima or minima?

Lets consider the critical point {\vec{x}_{0}} for {f}. We Taylor expand {f} to second order about {\vec{x}_{0}} writing
(4)\displaystyle  f(\vec{x}_{0}+\vec{h}) \approx f(\vec{x}_{0}) + \vec{h}\cdot\underbrace{\vec{\nabla}f(\vec{x}_{0})}_{=0} + \frac{1}{2} \vec{h}\cdot\mathrm{Hess}(f)\vec{h}
where we use the matrix
(5)\displaystyle  \mathrm{Hess}(f) = \begin{bmatrix} \partial_{1}^{2} f & \dots & \partial_{1}\partial_{n}f \\ \vdots & \ddots & \vdots \\ \partial_{n}\partial_{1} f & \dots & \partial_{n}^{2}f \end{bmatrix}
Each row is precisely {\vec{\nabla}\partial_{j}f}, and each column likewise is {\partial_{i}\vec{\nabla}f}; the intuition is {\mathrm{Hess}(f) \approx \vec{\nabla}^{2}f}.

Now, since we are Taylor expanding about a critical point, our approximation becomes
(6)\displaystyle  f(\vec{x}_{0}+\vec{h}) \approx f(\vec{x}_{0}) + \frac{1}{2} \vec{h}\cdot\mathrm{Hess}(f)\vec{h}.
We can consider the behaviour of {f} near {\vec{x}_{0}} by studying the properties of {\mathrm{Hess}(f)}. Specifically, the signs of the eigenvalues tells us whether the critical point is a local maxima (all eigenvalues are positive) or a minima (all are negative) or some saddle point (mixture having both positive and negative eigenvalues). If there exists at least one eigenvalue that vanishes, this test is inconclusive.

4. Parting Thoughts: What if we want to optimize a function constrained to live on a surface?


About Alex Nelson

I like math. I like programming. Most of all, I love puzzles.
This entry was posted in Calculus, Gradient, Optimization, Vector Calculus and tagged . Bookmark the permalink.

One Response to Finding Extrema of Multivariable Functions

  1. Pingback: Lagrange Multipliers | My Math Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s