Penalty method

Penalty methods are a certain class of algorithms for solving constrained optimization problems.

A penalty method replaces a constrained optimization problem by a series of unconstrained problems whose solutions ideally converge to the solution of the original constrained problem. The unconstrained problems are formed by adding a term, called a penalty function, to the objective function that consists of a penalty parameter multiplied by a measure of violation of the constraints. The measure of violation is nonzero when the constraints are violated and is zero in the region where constraints are not violated.

Description
Let us say we are solving the following constrained problem:
 * $$ \min_x f(\mathbf x) $$

subject to
 * $$ c_i(\mathbf x) \le 0 ~\forall i \in I. $$

This problem can be solved as a series of unconstrained minimization problems
 * $$ \min f_p (\mathbf x) := f (\mathbf x) + p ~ \sum_{i\in I} ~ g(c_i(\mathbf x)) $$

where
 * $$ g(c_i(\mathbf x))=\max(0,c_i(\mathbf x ))^2. $$

In the above equations, $$ g(c_i(\mathbf x))$$ is the exterior penalty function while $$p$$ is the penalty coefficient. When the penalty coefficient is 0, fp=f. In each iteration of the method, we increase the penalty coefficient $$p$$ (e.g. by a factor of 10), solve the unconstrained problem and use the solution as the initial guess for the next iteration. Solutions of the successive unconstrained problems will asymptotically converge to the solution of the original constrained problem.

Convergence
We first consider the set of global optimizers of the original problem, X*. Assume that the objective f has bounded level sets, and that the original problem is feasible. Then:


 * For every penalty coefficient p, the set of global optimizers of the penalized problem, Xp*, is non-empty.
 * For every ε>0, there exists a penalty coefficient p such that the set Xp* is contained in an ε-neighborhood of the set X*.

This theorem is helpful mostly when fp is convex, since in this case, we can find the global optimizers of fp.

A second theorem considers local optimizers. Let x* be a non-degenerate local optimizer of the original problem ("nondegenerate" means that the gradients of the active constraints are linearly independent and the second-order sufficient optimality condition is satisfied). Then, there exists a neighborhood V* of x*, and some p0>0, such that for all p>p0, the penalized objective fp has exactly one critical point in V* (denoted by x*(p)), and x*(p) approaches x* as p→∞. Also, the objective value f(x*(p)) is weakly-increasing with p.

Practical applications
Image compression optimization algorithms can make use of penalty functions for selecting how best to compress zones of colour to single representative values.

The advantage of the penalty method is that, once we have a penalized objective with no constraints, we can use any unconstrained optimization method to solve it. The disadvantage is that, as the penalty coefficient p grows, the unconstrained problem becomes ill-conditioned - the coefficients are very large, and this may cause numeric errors and slow convergence of the unconstrained minimization.