Tutorials › Calculus III › Scaling Variables And Constraints In Design Optimization

Home  /  Tutorials  /  Applied Multivariable Optimization

Applied Multivariable Optimization · Tutorial 499 of 999

Scaling Variables And Constraints In Design Optimization

Use characteristic scales to write optimization models in better-balanced variables and constraints, while tracking how the optimum and constraint multipliers translate back to physical units.

Advanced 12 min read

What You'll Learn

  • Choose characteristic scales for variables with different units or typical magnitudes.
  • Rewrite objective functions and constraints in scaled coordinates.
  • Identify which scalings preserve a feasible set and an optimizer.
  • Translate Lagrange multipliers between scaled and physical models.
  • Diagnose how poor scaling can distort numerical calculations and sensitivity interpretations.

Why Scaling Matters in Design Optimization

Design variables can have very different units and typical sizes. A length might be measured in millimetres, a mass in kilograms, and a temperature in kelvins. Constraints can also have very different numerical magnitudes. Writing all these quantities directly in one system of equations is mathematically valid, but it can make calculations harder to interpret and numerical methods less reliable.

Scaling replaces physical variables and constraints with dimensionless or more comparable quantities. A suitable scaling does not change the physical design that solves the problem. It changes how the problem is represented, and it requires care when interpreting multipliers or sensitivity values.

Definition: Let \(x=(x_1,\ldots,x_n)\) be a vector of design variables, and choose positive characteristic scales \(s_1,\ldots,s_n\). Define scaled variables \(z_i=x_i/s_i\), or equivalently \(x_i=s_i z_i\). A constraint function \(g_j(x)\) may be scaled by a positive characteristic scale \(G_j\), giving \(\widetilde g_j(z)=g_j(Sz)/G_j\), where \(S\) is the diagonal matrix with entries \(s_i\).

The scales \(s_i\) should reflect meaningful sizes for the variables, such as a nominal design value or a typical allowable range. A constraint scale \(G_j\) should have the same units as \(g_j\), and can be chosen to represent a typical magnitude for that constraint. The purpose is not to make every equation look identical; it is to avoid numerical representations in which important quantities differ by many orders of magnitude without a good reason.

What Scaling Preserves

If each \(s_i\) is positive, the change of variables \(x=Sz\) is one-to-one: every \(z\) determines exactly one physical design \(x\), and every \(x\) corresponds to exactly one \(z\). Thus the change of variables alone does not add or remove designs. For an inequality written as \(g_j(x)\leq 0\), dividing by a positive \(G_j\) also preserves its direction and its feasible set. The same holds for an equality \(h_j(x)=0\). Scaling an objective by a positive constant preserves which feasible point minimizes it.

Rule: For a minimization problem, \(x=Sz\) with nonzero variable scales gives equivalent coordinates. Replacing \(g_j(x)\leq 0\) by \(g_j(x)/G_j\leq 0\) preserves the inequality when \(G_j>0\). Replacing an objective \(f\) by \(f/F_0\) preserves its minimizers when \(F_0>0\). A negative scale reverses an inequality direction and must be handled accordingly.

These equivalences concern the solution set and the optimizer, not the numerical values of the equations or their derivatives. The scaled equations have different coefficients, and multipliers associated with constraints change with the constraint scales.

Worked Example: Rescaling a Length and a Distance Constraint

Worked Example: Rescaling a Length and a Distance Constraint

Suppose \(x\) is a length in millimetres and \(y\) is a length in metres. Consider the dimensionless cost model

$$ \min_{x,y}\ f(x,y)=\left(\frac{x}{1000}-1\right)^2+\left(\frac{y}{0.5}-1\right)^2 \quad\text{subject to}\quad x+2000y=1500. $$

Here \(1000\) millimetres and \(0.5\) metres are the chosen characteristic lengths. Set \(u=x/1000\) and \(v=y/0.5\), so \(x=1000u\) and \(y=0.5v\). Substitution into the constraint gives

$$ 1000u+2000(0.5v)=1500, \qquad 1000u+1000v=1500, \qquad u+v=1.5. $$

The scaled problem is to minimize \((u-1)^2+(v-1)^2\) subject to \(u+v=1.5\). Write \(v=1.5-u\). The objective along the constraint is

$$ (u-1)^2+(0.5-u)^2 =2u^2-3u+1.25. $$

Its derivative is \(4u-3\), which is zero at \(u=0.75\). The second derivative is \(4>0\), so this point gives the minimum along the constraint. Then \(v=1.5-0.75=0.75\). Returning to physical variables gives

$$ x=1000(0.75)=750\text{ mm}, \qquad y=0.5(0.75)=0.375\text{ m}. $$

Check the constraint in physical units: \(750+2000(0.375)=750+750=1500\). Check the objective: each squared term is \((0.75-1)^2=0.0625\), so \(f=0.125\). In scaled coordinates, \(u=v=0.75\) also gives \(0.0625+0.0625=0.125\), as required.

The original equation mixes a length in millimetres with a length in metres after conversion by the coefficient \(2000\). The scaled equation \(u+v=1.5\) makes the relative contributions immediately visible. It also gives variables of order one near the solution, rather than values near \(10^3\) and \(10^{-1}\).

Why the Optimizer Is Unchanged

The preservation of the optimizer follows directly from the correspondence between designs. This short argument also explains why rescaling is a representation change rather than a different physical problem.

Derivation: Consider minimizing \(f(x)\) over the set of \(x\) satisfying the model's constraints. Define \(x=Sz\), where every diagonal entry of \(S\) is positive. Since \(S\) is invertible, the map \(z\mapsto Sz\) is a bijection. Define \(\widetilde f(z)=f(Sz)\), and replace each original constraint by the same constraint evaluated at \(Sz\), divided by a positive scale. A point \(z\) satisfies the transformed constraints exactly when \(x=Sz\) satisfies the original constraints. Moreover, \(\widetilde f(z)=f(x)\). Therefore, if \(x^*\) has objective value no greater than that of every original feasible point, its corresponding \(z^*=S^{-1}x^*\) has objective value no greater than that of every transformed feasible point. The reverse implication follows from the same bijection. Thus the two problems have corresponding minimizers.

If the objective is also divided by \(F_0>0\), all objective values are multiplied by the same positive constant. Their ordering is unchanged, so the minimizing design is still unchanged. This argument does not say that an arbitrary change to the model preserves its solution: the equivalence depends on the one-to-one variable transformation and on scaling constraints and objectives in ways that preserve their meanings.

Constraint Scaling and Multipliers

In a constrained optimization calculation, a multiplier is attached to a particular written form of a constraint. Multiplying a constraint by a constant changes the numerical value of its multiplier even though the feasible set stays the same. A multiplier therefore cannot be compared across differently scaled models without converting its scale.

Worked Example: A Multiplier Changes When a Constraint Is Rescaled

Minimize \(f(x,y)=(x-1)^2+(y-1)^2\) subject to \(g(x,y)=3-x-y\leq 0\). The point \((1.5,1.5)\) is feasible and lies on the boundary. Its objective gradient is \(\nabla f=(2(x-1),2(y-1))=(1,1)\), while \(\nabla g=(-1,-1)\). The stationarity equation for this active inequality is \(\nabla f+\lambda\nabla g=\mathbf 0\), with \(\lambda\geq 0\). It gives \((1,1)+\lambda(-1,-1)=\mathbf 0\), so \(\lambda=1\).

Now write the same constraint as \(\widetilde g=g/1000\leq 0\). Its gradient at the same point is \((-0.001,-0.001)\). The stationarity equation is now \((1,1)+\widetilde\lambda(-0.001,-0.001)=\mathbf 0\), so \(\widetilde\lambda=1000\). Both calculations describe the same optimizer and feasible boundary. The scaled constraint multiplier is 1000 times the original multiplier; the factor comes from dividing the constraint by 1000.

Check feasibility: \(g(1.5,1.5)=3-1.5-1.5=0\), and \(\widetilde g(1.5,1.5)=0/1000=0\). Check optimality directly on the feasible region: every feasible point satisfies \(x+y\geq 3\), and the squared distance \((x-1)^2+(y-1)^2\) is minimized on this half-plane at its boundary point closest to \((1,1)\), namely \((1.5,1.5)\). Its objective value is \(0.5\).

More generally, suppose the objective is scaled to \(\widetilde f(z)=f(Sz)/F_0\) and a constraint is scaled to \(\widetilde g_j(z)=g_j(Sz)/G_j\), with positive \(F_0\) and \(G_j\). If \(\mu_j\) is the original constraint multiplier and \(\widetilde\lambda_j\) is the multiplier in the scaled stationarity equation, then

$$ \widetilde\lambda_j=\frac{\mu_jG_j}{F_0}, \qquad\text{or equivalently}\qquad \mu_j=\frac{F_0}{G_j}\widetilde\lambda_j. $$

This relation follows by applying the chain rule to the scaled objective and constraints in the stationarity equation. It matters for interpretation: if the original multiplier measures a change in objective per unit change in a constraint's right-hand side, its units and numerical value depend on the original units and on the objective scale. The scaled multiplier belongs to the scaled formulation instead.

Worked Example: Scaling Improves the Balance of a Quadratic Model

Worked Example: Scaling Improves the Balance of a Quadratic Model

Let \(x\) be measured in millimetres and \(y\) in kilograms. Consider the dimensionless objective and equality constraint

$$ \min_{x,y}\ f(x,y)=\frac12\left[\left(\frac{x}{1000}\right)^2+\left(\frac{y}{0.001}\right)^2\right] \quad\text{subject to}\quad \frac{x}{1000}+\frac{y}{0.001}=3. $$

Define \(u=x/1000\) and \(v=y/0.001\). The scaled problem is to minimize \(\frac12(u^2+v^2)\) subject to \(u+v=3\). Substituting \(v=3-u\) gives

$$ \frac12\bigl(u^2+(3-u)^2\bigr)=u^2-3u+4.5. $$

The derivative is \(2u-3\), so \(u=1.5\). Then \(v=3-1.5=1.5\); the second derivative is \(2>0\), confirming a minimum. Thus \(x=1000(1.5)=1500\) millimetres and \(y=0.001(1.5)=0.0015\) kilograms. The constraint checks: \(1500/1000+0.0015/0.001=1.5+1.5=3\). The objective is \(\frac12(2.25+2.25)=2.25\).

In physical coordinates, the objective's second derivatives are \(1/1000^2=0.000001\) with respect to \(x\), and \(1/0.001^2=1{,}000{,}000\) with respect to \(y\). Their ratio is \(10^{12}\). In scaled coordinates, both second derivatives are \(1\). This large disparity in the original coefficients can make numerical calculations poorly balanced. Scaling does not change the underlying physical design; it expresses its two coordinates on comparable numerical scales.

Choosing Scales and Diagnosing Sensitivity

Useful scales are based on the problem rather than on a desire to make coefficients cosmetically similar. A variable scale can be a nominal value, a characteristic operating level, or a representative allowable variation. A constraint scale can be a typical nonzero value of that constraint's left-hand side or residual. Scales should be nonzero, and positive scales are the convenient choice for inequality constraints.

Poor scaling may obscure sensitivity in two ways. First, a large or small multiplier can reflect the units or numerical scale of the constraint, rather than a dramatic physical effect. Second, variables and gradients with very different numerical magnitudes can make a system of equations harder to solve accurately. Examine the scaled model and the units of any reported sensitivity before drawing a physical conclusion. If a multiplier is used to report a change in the original objective per unit change in an original constraint limit, convert it back to the original objective and constraint scales.

Scaling is not a substitute for checking the model. It does not fix an incorrect constraint, remove genuine nonlinear behaviour, or guarantee that every numerical algorithm will work well. It is a way to represent the same problem with more balanced quantities, so that calculations and their interpretation are less easily distorted by arbitrary choices of units.

A Common Scaling Mistake

Suppose the inequality \(2-x\leq 0\) expresses a lower bound on \(x\). It means \(x\geq 2\). Dividing by \(-2\) gives \((x-2)/2\), but division by a negative number reverses the inequality. The correct equivalent statement is \((x-2)/2\geq 0\). If one instead writes \((x-2)/2\leq 0\), one has changed the feasible set to \(x\leq 2\).

Takeaway: Positive scaling preserves an inequality's direction. Negative scaling is possible only if the inequality direction is reversed at the same time.

Check Your Understanding

Use these questions to check your understanding of variable and constraint scaling.

  1. If a length variable has a typical value of 500 millimetres, what scaled variable would you define using 500 millimetres as its characteristic scale?
  2. Why does dividing an inequality constraint by a positive number preserve its feasible set?
  3. A constraint is divided by 10 while the objective is unchanged. How does its multiplier change in the stationarity equation?
  4. What must happen to an inequality direction when both sides are divided by a negative number?
  5. Why should a very large multiplier in a scaled model not automatically be interpreted as a large physical sensitivity?

Previous

Next

Back to Tutorials