The condition of linear independence. An alternative condition. Second-order conditions.

3.3. REMARKS AND EXTENSIONS 21

3.3 Remarks and Extensions

3.3.1 The condition of linear independence.

The necessary condition 3.13 need not hold if the derivatives f ix x ∗ , 1 ≤ i ≤ m, are not linearly independent. This can be checked in the following example Minimize subject to sin x 2 1 + x 2 2 π 2 x 2 1 + x 2 2 = 1. 3.29

3.3.2 An alternative condition.

Keeping the notation of Theorem 3.2.1, define the Lagrangian function L : R n+m → R by L : x, λ 7→ f x − P m i=1 λ i f i x. The following is a reformulation of 3.12, and its proof is left as an exercise. Let x ∗ be optimal for 3.12, and suppose that f ix x ∗ , 1 ≤ i ≤ m, are linearly independent. Then there exists λ ∗ ∈ R m such that x ∗ , λ ∗ is a stationary point of L, i.e., L x x ∗ , λ ∗ = 0 and L λ x ∗ , λ ∗ = 0.

3.3.3 Second-order conditions.

Since we can convert the problem 3.12 into a problem of maximizing ˆ f over an open set, all the comments of Section 2.4 will apply to the function ˆ f . However, it is useful to translate these remarks in terms of the original function f and f . This is possible because the function g is uniquely specified by 3.16 in a neighborhood of x ∗ . Furthermore, if f is twice differentiable, so is g see Fleming [1965]. It follows that if the functions f i , 0 ≤ i ≤ m, are twice continuously differentiable, then so is ˆ f , and a necessary condition for x ∗ to be optimal for 3.12 and 3.13 and the condition that the n − m × n − m matrix ˆ f 0uu u ∗ is negative semi-definite. Furthermore, if this matrix is negative definite then x ∗ is a local optimum. the following exercise expresses f ˆ f 0uu u ∗ in terms of derivatives of the functions f i . Exercise: Show that ˆ f 0uu u ∗ = [g ′ u .. . I] L ww L uw L wu L uu   g u . . . I   w ∗ , u ∗ where g u u ∗ = −[f w x ∗ ] −1 f u x ∗ , Lx = f x − m X i=1 λ ∗ i f i x. 22 CHAPTER 3. OPTIMIZATION WITH EQUALITY CONSTRAINTS 3.3.4 A numerical procedure. We assume that the derivatives f ix x, 1 ≤ i ≤ m, are linearly independent for all x. Then the following algorithm is a straightforward adaptation of the procedure in Section 2.4.6. Step 1. Find x arbitrary so that f i x = α i , 1 ≤ i ≤ m. Set k = 0 and go to Step 2. Step 2. Find a partition x = w, u 2 of the variables such that f w x k is nonsingular. Calculate λ k by λ k ′ = f 0w f −1 wxk , and ▽ ˆ f k u k = −f ′ u x k λ k + f ′ 0u x k . If ▽ ˆ f k u k = 0, stop. Otherwise go to Step 3. Step 3. Set ˜ u k = u k + d k ▽ ˆ f k u k . Find ˜ w k such that f i ˜ w k , ˜ u k = 0, 1 ≤ i ≤ m. Set x k+1 = ˜ w k , ˜ u k , set k = k + 1, and return to Step 2. Remarks. As before, the step sizes d k 0 can be selected various ways. The practical applicability of the algorithm depends upon two crucial factors: the ease with which we can find a partition x = w, u so that f w x k is nonsingular, thus enabling us to calculate λ k ; and the ease with which we can find ˜ w k so that f ˜ w k , ˜ u k = α. In the next section we apply this algorithm to a practical problem where these two steps can be carried out without too much difficulty.

3.3.5 Design of resistive networks.