Stanford AA222 I Engineering Design Optimization | Spring 2025 | Disciplined Convex Programming

Discipline Convex Programming and Multi-Disciplinary Optimization: Summary

Key Concepts:

• Discipline Convex Programming (DCP)
• Convex Sets and Functions
• Concave Sets and Functions
• Product-Free Expressions
• Sign Rules for Product-Free Expressions
• Composition Rules (Univariate and Multivariate)
• Atom Library
• Verification (Automatic)
• Canonicalization
• Linearization
• Graph Expansion
• Partitioned Canonical Form
• Interior Point Methods
• Barrier Function
• Newton's Method

1. Introduction to Discipline Convex Programming (DCP)

• DCP is a subclass of convex programs that can be automatically verified, converted to a canonical form, and easily solved.
• It allows for reliable solutions to a wide range of problems, especially those requiring solutions at high frequencies (e.g., 100 Hz) with guaranteed optimality.
• DCP leverages the properties of convex programs, such as local minima being global minima and strong duality.
• Example: Minimizing an objective function subject to a constraint circle using DCP solvers (e.g., CVX, CVXPY) yields the optimal solution efficiently (e.g., in 0.1 seconds).

2. Convex Sets and Functions: Review

• Convex Set: A set where every line segment between any two points within the set lies entirely within the set. Examples: points, lines, half-spaces, balls, intersections of convex sets (orthants, quadrants, polyhedra).
• Convex Function: A function where every line segment between any two points on the function lies at or above the function. Examples: quadratics (with positive semidefinite Q), pointwise maximums, norms.
• Concave Function: A function where every line segment between any two points on the function lies at or below the function.
• Examples of convex functions (single variable): affine, powers (p < 0 or p > 1), exponentials, ReLU.
• Examples of concave functions (single variable): powers (0 < p < 1), logarithm.
• Examples of convex functions (multiple variables): affine, quadratic (with positive semidefinite Q), norms, sum of squares, pointwise maximums.
• Examples of concave functions (multiple variables): geometric mean.

3. Establishing Convexity: DCP Rules

• Verifying convexity manually (sampling points) or checking the Hessian (positive semidefinite) can be difficult.
• DCP provides rules to build convex functions from simpler ones, ensuring automatic verification.
• Two key rules:
  • Sign rules for product-free expressions.
  • Composition rules.

4. Sign Rules for Product-Free Expressions

• Rule 1: All expressions must be product-free, meaning they should be affine in problem variables plus linear combinations of other functions. No products between different functions are allowed.
• Rule 2: An expression is convex if:
  • Each function g_i is convex and its coefficient c_i is non-negative.
  • Each function g_i is concave and its coefficient c_i is non-positive.
  • Each function g_i is affine.
• Negating these rules yields rules for concavity.
• Products are disallowed because knowing the convexity/concavity of individual expressions doesn't guarantee anything about their product (e.g., x1 * x2 is non-convex even if x1 and x2 are affine).

5. Composition Rules

• Univariate Functions (f(g(x))):
  • The composition is convex if:
    • f is convex and f is non-decreasing over the range of g, and g is convex.
    • f is convex and f is non-increasing over the range of g, and g is concave.
  • Similar rules exist for concavity, with the added constraint that the range of g must be within the domain of f.
• Multivariate Functions (f(g(x))):
  • The composition is convex if f is affine or convex, and for each variable, one of the following holds:
    • f is non-decreasing in that expression, and that expression is convex.
    • f is non-increasing in that expression, and that expression is concave.
    • That expression is affine.
  • Similar rules exist for concavity.

6. Example: Applying DCP Rules

• Example: Checking the convexity of x1 + 3x2 + max(exp(1 - log(x2)) + x1^2, x1 - x2).
• The expression is broken down into a tree of expressions.
• Each sub-expression is analyzed using the sign and composition rules.
• For example:
  • x1 - x2 is affine.
  • 1 - log(x2) is convex (negative of a concave function).
  • x1^2 is convex.
  • exp(1 - log(x2)) is convex (convex, non-decreasing function of a convex function).
  • max(exp(1 - log(x2)) + x1^2, x1 - x2) is convex (max of convex and affine functions).
• Therefore, the entire expression is convex and DCP compliant.
• Example: sqrt(1 + x^2) is not directly resolvable by DCP, but the equivalent norm(1, x) is. This highlights that different specifications of the same function can have different DCP compliance.

7. Automatic Verification

• DCP solvers automatically verify convexity by:
  • Forming an expression tree using an atom library.
  • Recursively traversing the tree and checking sign and composition rules.
• Atom Library: A collection of base symbols (e.g., x, log(x), max, norm) with tracked features like curvature, monotonicity, and range. Domain experts can extend the atom library.
• Verification ensures that all expressions are product-free, formed by registered atoms, and satisfy sign, composition, and top-level rules.
• Top-level rules ensure the objective is convex and constraints are in the correct form (affine = affine, convex <= concave).

8. Canonicalization

• Canonicalization converts DCP problems into a canonical form suitable for solvers:
  • Minimize a linear objective.
  • Subject to affine equality constraints.
  • And convex set constraints.
• The process often results in a partitioned canonical form, grouping variables and constraints for solver efficiency.
• Canonicalization relies on:
  • Linearization.
  • Graph expansion.

9. Linearization

• Linearization converts product-free expressions to affine expressions with additional inequality constraints.
• For each function f, a new variable v_j is introduced, and a constraint is added:
  • If f is convex: f <= v_j.
  • If f is concave: f >= v_j.
• Affine inequality constraints are given slack variables s, where s >= 0.
• Example: Linearizing max(exp(1 - log(x2)) + x1^2, x1 - x2) involves introducing new variables for log(x2), 1 - log(x2), exp(1 - log(x2)), x1^2, and the max function itself, along with corresponding constraints.

10. Graph Expansion

• Graph expansion replaces a convex function with a minimization over its epigraph (the area above the function).
• The epigraph of a convex function is a convex set.
• This can transform non-linear constraints into linear constraints.
• Example: Replacing abs(x) <= 5 with y <= 5, y >= x, y >= -x, and minimizing over y.

11. Solution Methods

• If the problem becomes linear or quadratic after canonicalization, LP or QP solvers can be used.
• Interior point methods are commonly used.
• Inequality constraints are converted into a barrier function in the objective.
• Newton's method can efficiently solve the resulting optimization problem.
• The inversion of the Hessian of the barrier function is often sparse due to the partitioned canonical form, leading to efficient computation.

12. Conclusion

• DCP is a powerful tool for convex optimization, enabling automatic verification and efficient solving.
• Convexity requires product-free expressions and adherence to sign, composition, and top-level rules.
• Canonicalization transforms DCP problems into a solver-friendly form.
• Interior point methods provide efficient solution approaches.
• If your problem can be formulated in a DCP-compliant manner, it is highly recommended to leverage DCP solvers for guaranteed optimal solutions.