you need to read this
As someone who did a Bachelors in Mathematics I used to be first introduced to L¹ and L² as a measure of Distance… now it appears to be a measure of error — where have we gone flawed? But jokes aside, there appears to be this misconception that L₁ and L₂ serve the identical function — and while that will sometimes be true — each norm shapes its models in drastically other ways.
In this text we’ll travel from plain-old points on a line all of the strategy to L∞, stopping to see why L¹ and L² matter, how they differ, and where the L∞ norm shows up in AI.
Our Agenda:
- When to make use of L¹ versus L² loss
- How L¹ and L² regularization pull a model toward sparsity or smooth shrinkage
- Why the tiniest algebraic difference blurs GAN images — or leaves them razor-sharp
- Easy methods to generalize distance to Lᵖ space and what the L∞ norm represents
A Temporary Note on Mathematical Abstraction
You would possibly have have had a conversation (perhaps a confusing one) where the term popped up, and you would possibly have left that conversation feeling slightly more confused about what mathematicians are really doing. Abstraction refers to extracting underlying patters and properties from an idea to generalize it so it has wider application. This might sound really complicated but take a take a look at this trivial example:
A degree in 1-D is ; in 2-D: ; in 3-D: . Now I don’t learn about you but I can’t visualize 42 dimensions, but the identical pattern tells me some extent in 42 dimensions can be
This might sound trivial but this idea of abstraction is essential to get to L∞, where as an alternative of some extent we abstract distance. Any longer let’s work with otherwise known by its formal title: x∈ℝⁿ. And any vector is
The “Normal” Norms: L1 and L2
The key takeaway is easy but powerful: since the L¹ and L² norms behave in another way in a number of crucial ways, you possibly can mix them in a single objective to juggle two competing goals. In regularization, the L¹ and L² terms contained in the loss function help strike one of the best spot on the bias-variance spectrum, yielding a model that’s each accurate generalizable. In Gans, the L¹ pixel loss is paired with adversarial loss so the generator makes images that (i) look realistic and (ii) match the intended output. Tiny distinctions between the 2 losses explain why Lasso performs feature selection and why swapping L¹ out for L² in a GAN often produces blurry images.
L¹ vs. L² Loss — Similarities and Differences
- In case your data may contain many outliers or heavy-tailed noise, you often reach for L¹.
- In case you care most about overall squared error and have reasonably clean data, L² is superb — and easier to optimize since it is smooth.
Because MAE treats each error proportionally, models trained with L¹ sit nearer the median commentary, which is precisely why L¹ loss keeps texture detail in GANs, whereas MSE’s quadratic penalty nudges the model toward a mean value that appears smeared.
L¹ Regularization (Lasso)
Optimization and Regularization pull in opposite directions: optimization tries to suit the training set perfectly, while regularization deliberately sacrifices slightly training accuracy to achieve . Adding an L¹ penalty 𝛼∥w∥₁ promotes sparsity — many coefficients collapse all of the strategy to zero. A much bigger α means harsher feature pruning, simpler models, and fewer noise from irrelevant inputs. With Lasso, you get built-in feature selection since the ∥w∥₁ term literally turns small weights off, whereas L² merely shrinks them.
L2 Regularization (Ridge)
Change the regularization term to
and you could have Ridge regression. Ridge shrinks weights toward zero without often hitting exactly zero. That daunts any single feature from dominating while still keeping every feature in play — handy while you imagine inputs matter but you would like to curb overfitting.
Each Lasso and Ridge improve generalization; with Lasso, once a weight hits zero, the optimizer feels no strong reason to go away — it’s like standing still on flat ground — so zeros naturally “stick.” Or in additional technical terms they simply mold the coefficient space in another way — Lasso’s diamond-shaped constraint set zeroes coordinates, Ridge’s spherical set simply squeezes them. Don’t worry for those who didn’t understand that, there is kind of numerous theory that’s beyond the scope of this text, but when it interests you this reading on Lₚ space should help.
But back to point. Notice how after we train each models on the identical data, Lasso removes some input features by setting their coefficients exactly to zero.
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso, Ridge
X, y = make_regression(n_samples=100, n_features=30, n_informative=5, noise=10)
model = Lasso(alpha=0.1).fit(X, y)
print("Lasso nonzero coeffs:", (model.coef_ != 0).sum())
model = Ridge(alpha=0.1).fit(X, y)
print("Ridge nonzero coeffs:", (model.coef_ != 0).sum())
Notice how if we increase α to 10 so much more features are deleted. This may be quite dangerous as we could possibly be eliminating informative data.
model = Lasso(alpha=10).fit(X, y)
print("Lasso nonzero coeffs:", (model.coef_ != 0).sum())
model = Ridge(alpha=10).fit(X, y)
print("Ridge nonzero coeffs:", (model.coef_ != 0).sum())
L¹ Loss in Generative Adversarial Networks (GANs)
GANs pit 2 networks against one another, a Generator G (the “forger”) against a Discriminator D (the “detective”). To make G produce convincing faithful images, many image-to-image GANs use a hybrid loss
where
- — input image (e.g., a sketch)
- — real goal image (e.g., a photograph)
- — balance knob between realism and fidelity
Swap the pixel loss to L² and also you square pixel errors; large residuals dominate the target, so G plays it secure by predicting the of all plausible textures — result: smoother, blurrier outputs. With L¹, every pixel error counts the identical, so G gravitates to the texture patch and keeps sharp boundaries.
Why tiny differences matter
- In regression, the kink in L¹’s derivative lets Lasso weak predictors, whereas Ridge only nudges them.
- In vision, the linear penalty of L¹ keeps high-frequency detail that L² blurs away.
- In each cases you possibly can mix L¹ and L² to trade robustness, sparsity, and smooth optimization — precisely the balancing act at the center of recent machine-learning objectives.
Generalizing Distance to Lᵖ
Before we reach L∞, we’d like to speak concerning the the 4 rules every must satisfy:
- Non-negativity — A distance can’t be negative; no person says “I’m –10 m from the pool.”
- Positive definiteness — The gap is zero on the zero vector, where no displacement has happened
- Absolute homogeneity (scalability) — Scaling a vector by α scales its length by |α|: for those who double your speed you double your distance
- Triangle inequality — A detour through y isn’t shorter than going straight from start to complete (x + y)
Firstly of this text, the mathematical abstraction we performed was quite straightforward. But now, as we take a look at the next norms, you possibly can see we’re doing something similar at a deeper level. There’s a transparent pattern: the exponent contained in the sum increases by one every time, and the exponent outside the sum does too. We’re also checking whether this more abstract notion of distance still satisfies the core properties we mentioned above. It does. So what we’ve done is successfully abstract the concept of distance into Lᵖ space.
as a single of distances — the Lᵖ space. Taking the limit as p→∞ squeezes that family all of the strategy to the L∞ norm.
The L∞ Norm
The L∞ norm goes by many names supremum norm, max norm, uniform norm, Chebyshev norm, but they’re all characterised by the next limit:
By generalizing our norm to p — space, in two lines of code, we are able to write a function that calculates distance in any norm possible. Quite useful.
def Lp_norm(v, p):
return sum(abs(x)**p for x in v) ** (1/p)We are able to now consider how our measure for distance changes as increases. the graphs bellow we see that our measure for distance monotonically decreases and approaches a really specific point: The most important absolute value within the vector, represented by the dashed line in black.
Actually, it doesn’t only approach the biggest absolute coordinate of our vector but
The max-norm shows up any time you would like a uniform guarantee or worst-case control. In less technical terms, If no individual coordinate can transcend a certain threshold than the L∞ norm must be used. If you would like to set a tough cap on every coordinate of your vector then this can also be your go to norm.
This just isn’t only a quirk of theory but something quite useful, and well applied in plethora of various contexts:
- Maximum absolute error — sure every prediction so none drifts too far.
- Max-Abs feature scaling — squashes each feature into [−1,1][-1,1][−1,1] without distorting sparsity.
- Max-norm weight constraints — keep all parameters inside an axis-aligned box.
- Adversarial robustness — restrict each pixel perturbation to an ε-cube (an L∞ ball).
- Chebyshev distance in k-NN and grid searches — fastest strategy to measure “king’s-move” steps.
- Robust regression / Chebyshev-center portfolio problems — linear programs that minimize the worst residual.
- Fairness caps — limit the biggest per-group violation, not only the common.
- Bounding-box collision tests — wrap objects in axis-aligned boxes for quick overlap checks.
Conclusion
Abstracting the thought of distance can feel unwieldy, even needlessly theoretical, but distilling it to its core properties frees us to ask questions that might otherwise be inconceivable to border. Doing so reveals recent norms with concrete, real-world uses. It’s tempting to treat all distance measures as interchangeable, yet small algebraic differences give each norm distinct properties that shape the models built on them. From the bias-variance trade-off in regression to the selection between crisp or blurry images in GANs, it matters the way you measure distance.
