12
CHAPTER 2. OPTIMIZATION OVER AN OPEN SET
Table 2.1 Does there exist
At how many points an optimal deci-
in Ω is 2.2.2
Further Case
sion for 2.2.1? satisfied?
Consequences 1
Yes Exactly one point,
x
∗
is the say
x
∗
unique optimal 2
Yes More than one point
3 No
None 4
No Exactly one point
5 No
More than one point
2.3.2 Consequences.
Let us evaluate the usefulness of 2.11 and its special case 2.18. Equation 2.11 gives us n
equations which must be satisfied at any optimal decision x
∗
= x
∗ 1
, . . . , x
∗ n
′
. These are
∂f ∂x
1
x
∗
= 0,
∂f ∂x
2
x
∗
= 0, . . . ,
∂f ∂x
n
x
∗
= 0 2.19
Thus, every optimal decision must be a solution of these n simultaneous equations of n variables, so
that the search for an optimal decision from Ω is reduced to searching among the solutions of 2.19.
In practice this may be a very difficult problem since these may be nonlinear equations and it may be necessary to use a digital computer. However, in these Notes we shall not be overly concerned
with numerical solution techniques but see 2.4.6 below.
The theorem may also have conceptual significance. We return to the example and recall the N = R − C. Suppose that R and C are differentiable, in which case 2.18 implies that at every
optimal decision α
∗
, p
∗ ∂R
∂α
α
∗
, p
∗
=
∂C ∂α
α
∗
, p
∗
,
∂R ∂p
α
∗
, p
∗
=
∂C ∂p
α
∗
, p
∗
, or, in the language of economic analysis, marginal revenue = marginal cost. We have obtained an
important economic insight.
2.4 Remarks and Extensions
2.4.1 A warning.
Equation 2.11 is only a necessary condition for x
∗
to be optimal. There may exist decisions ˜
x ∈ Ω such that
f
x
˜ x = 0 but ˜
x is not optimal. More generally, any one of the five cases in Table 2.1 may occur. The diagrams in Figure 2.1 illustrate these cases. In each case
Ω = −1, 1. Note that in the last three figures there is no optimal decision since the limit points -1 and +1 are
not in the set of permissible decisions Ω = −1, 1. In summary, the theorem does not give us any
clues concerning the existence of an optimal decision, and it does not give us sufficient conditions either.
2.4.
REMARKS AND EXTENSIONS 13
Case 1 Case 2
Case 3
Case 5 Case 4
-1 1
-1 1
-1 1
1 -1
-1 1
Figure 2.2: Illustration of 4.1.
2.4.2 Existence.
If the set of permissible decisions Ω is a closed and bounded subset of R
n
, and if f is continuous,
then it follows by the Weierstrass Theorem that there exists an optimal decision. But if Ω is closed
we cannot assert that the derivative of f vanishes at the optimum. Indeed, in the third figure above,
if Ω = [−1, 1], then +1 is the optimal decision but the derivative is positive at that point.
2.4.3 Local optimum.
We say that x
∗
∈ Ω is a locally optimal decision if there exists ε 0 such that fx
∗
≥ f x whenever
x ∈ Ω and |x
∗
− x| ≤ ε. It is easy to see that the theorem holds i.e., 2.11 for local optima also.
2.4.4 Second-order conditions.
Suppose f is twice-differentiable and let x
∗
∈ Ω be optimal or even locally optimal. Then f
x
x
∗
= 0, and by Taylor’s theorem
f x
∗
+ δh = f x
∗
+
1 2
δ
2
h
′
f
xx
x
∗
h + oδ
2
, 2.20
where
oδ
2
δ
2
→ 0 as δ → 0. Now for δ 0 sufficiently small f x
∗
+ δh ≤ f x
∗
, so that dividing by
δ
2
0 yields 0 ≥
1 2
h
′
f
xx
x
∗
h +
oδ
2
δ
2
and letting δ approach zero we conclude that h
′
f
xx
x
∗
h ≤ 0 for all h ∈ R
n
. This means that f
xx
x
∗
is a negative semi-definite matrix. Thus, if we have a twice differentiable objective function, we get an additional necessary condition.
2.4.5 Sufficiency for local optimal.
Suppose at x
∗
∈ Ω, f
x
x
∗
= 0 and f
xx
is strictly negative definite. But then from the expansion 2.20 we can conclude that
x
∗
is a local optimum.
14
CHAPTER 2. OPTIMIZATION OVER AN OPEN SET 2.4.6
A numerical procedure.
At any point ˜
x ∈ Ω the gradient ▽
x
f ˜ x is a direction along which f x increases, i.e., f ˜
x + ε ▽
x
f ˜ x f ˜
x for all ε 0 sufficiently small. This observation suggests the following scheme for finding a point
x
∗
∈ Ω which satisfies 2.11. We can formalize the scheme as an algorithm. Step 1.
Pick x
∈ Ω. Set i = 0. Go to Step 2. Step 2.
Calculate ▽
x
f x
i
. If ▽
x
f x
i
= 0, stop. Otherwise let
x
i+1
= x
i
+ d
i
▽
x
f x
i
and go to Step 3.
Step 3. Set
i = i + 1 and return to Step 2. The step size
d
i
can be selected in many ways. For instance, one choice is to take d
i
to be an optimal decision for the following problem:
Max {f x
i
+ d ▽
x
f x
i
|d 0, x
i
+ d ▽
x
f x
i
∈ Ω}. This requires a one-dimensional search. Another choice is to let
d
i
= d
i−1
if f x
i
+ d
i−1
▽
x
f x
i
f x
i
; otherwise let d
i
= 1k d
i−1
where k is the smallest positive integer such that
f x
i
+ 1k d
i−1
▽
x
f x
i
f x
i
. To start the process we let d
−1
0 be arbitrary.
Exercise:
Let f be continuous differentiable. Let {d
i
} be produced by either of these choices and let
{x
i
} be the resulting sequence. Then 1.
f x
i+1
f x
i
if x
i+1
6= x
i
, i 2. if
x
∗
∈ Ω is a limit point of the sequence {x
i
}, f
x
x
∗
= 0. For other numerical procedures the reader is referred to Zangwill [1969] or Polak [1971].
Chapter 3
OPTIMIZATION OVER SETS DEFINED BY EQUALITY
CONSTRAINTS
We first study a simple example and examine the properties of an optimal decision. This will generalize to a canonical problem, and the properties of its optimal decisions are stated in the form
of a theorem. Additional properties are summarized in Section 3 and a numerical scheme is applied to determine the optimal design of resistive networks.
3.1 Example
