Interpreting Weight Regularization In Machine Learning

Why do L1 and L2 regularization lead to model sparsity and weight shrinkage? What about L3 regularization? Keep reading to search out out more!

Co-authored with Naresh Singh.

After reading this text, you’ll be thoroughly equipped with the tools and reasoning capability to think concerning the effects of any Lk regularization term and judge if it applies to your situation.

What’s regularization in machine learning?

Let’s take a look at some definitions on the web and generalize based on those.

1. Regularization is a set of methods for reducing overfitting in machine learning models. Typically, regularization trades a marginal decrease in training accuracy for a rise in generalizability. (IBM)
2. Regularization makes models stable across different subsets of the info. It reduces the sensitivity of model outputs to minor changes within the training set. (geeksforgeeks)
3. Regularization in machine learning serves as a way to forestall a model from overfitting. (simplilearn)

On the whole, regularization is a way to stop the model from overfitting and to permit the model to generalize its predictions on unseen data. Let’s take a look at the role of weight regularization particularly.

Why use weight regularization?

One could employ many types of regularization while training a machine learning model. Weight regularization is one such technique, which is the main target of this text. Weight regularization means applying some constraints on the learnable weights of your machine learning model in order that they permit the model to generalize to unseen inputs.

Weight regularization improves the performance of neural networks by penalizing the load matrices of nodes. This penalty discourages the model from having large parameter (weight) values. It helps control the model’s ability to suit the noise within the training data. Typically, the biases within the machine learning model are usually not subject to regularization.

How is regularization implemented in deep neural networks?

Typically, a regularization loss is added to the model’s loss during training. It allows us to manage the model’s weights during training. The formula looks like this:

Figure-1: Total loss as a sum of the model loss and regularization loss. k is a floating point value and indicates the regularization norm. Alpha is the weighting factor for the regularization loss.

Typical values of k utilized in practice are 1 and a couple of. These are called the L1 and L2 regularization schemes.

But why will we use just these two values for essentially the most part, when in reality there are infinitely many values of k one could use? Let’s answer this query with an interpretation of the L1 and L2 regularization schemes.

The 2 commonest kinds of regularization used for machine learning models are L1 and L2 regularization. We’ll start with these two, and proceed to debate some unusual regularization types equivalent to L0.5 and L3 regularization. We’ll take a take a look at the gradients of the regularization losses and plot them to intuitively understand how they affect the model weights.

L1 regularization

L1 regularization adds the typical of absolutely the value of the weights together because the regularization loss.

Figure-2: L1 regularization loss and its partial derivative with respect to every weight Wi.

It has the effect of adjusting the weights by a relentless (on this case alpha times the educational rate) within the direction that minimizes the loss. Figure 3 shows a graphical representation of the function and its derivative.

Figure-3: The blue line is |w| and the red line is the derivative of |w|.

You may see that the derivative of the L1 norm is a relentless (depending on the sign of w), which suggests that the gradient of this function only will depend on the sign of w and never its magnitude. The gradient of the L1 norm is just not defined at w=0.

It signifies that the weights are moved towards zero by a relentless value at each step during backpropagation. Throughout training, it has the effect of driving the weights to converge at zero. That’s the reason the L1 regularization makes a model sparse (i.e. among the weights develop into 0). It’d cause an issue in some cases if it finally ends up making a model too sparse. The L2 regularization doesn’t have this side-effect. Let’s discuss it in the subsequent section.

L2 regularization

L2 regularization adds the typical of the square of absolutely the value of the weights together because the regularization loss.

Figure-4: L2 regularization loss and its partial derivative with respect to every weight Wi.

It has the effect of adjusting each weight by a multiple of the load itself within the direction that minimizes the loss. Figure 5 shows a graphical representation of the function and its derivative.

Figure-5: The blue line is pow(|w|, 2) and the red line is the derivative of pow(|w|, 2).

You may see that the derivative of the L2 norm is just the sign-adjusted square root of the norm itself. The gradient of the L2 norm will depend on each the sign and magnitude of the load.

Which means that at every gradient update step, the weights will likely be adjusted toward zero by an amount that’s proportional to the load’s value. Over time, this has the effect of drawing the weights toward zero, but never exactly zero, since subtracting a relentless factor of a worth from the worth itself never makes the result exactly zero unless it’s zero to start with. The L2 norm is often used for weight decay during machine learning model training.

Let’s consider L0.5 regularization next.

L0.5 regularization

L0.5 regularization adds the typical of the square root of absolutely the value of the weights together because the regularization loss.

Figure-6: L0.5 regularization loss and its partial derivative with respect to every weight Wi.

This has the effect of adjusting each weight by a multiple (on this case alpha times the educational rate) of the inverse square root of the load itself within the direction that minimizes the loss. Figure 7 shows a graph of the function and its derivative.

Figure-7: The blue line is pow(|w|, 0.5) and the red line is the derivative of pow(|w|, 0.5).

You may see that the derivative of the L0.5 norm is a discontinuous function, which peaks on the positive values of w near 0 and it reaches negative infinity for the negative values of w near 0. Further, we are able to draw the next conclusions from the graph:

1. As |w| tends to 0, the magnitude of the gradient tends to infinity. During backpropagation, these values of w will quickly swing to past 0 because large gradients will cause a big change in the worth of w. In other words, negative w will develop into positive and vice-versa. This cycle of flip flops will proceed to repeat itself.
2. As |w| increases, the magnitude of the gradient decreases. These values of w are stable due to small gradients. Nonetheless, with each backpropagation step, the worth of w will likely be drawn closer to 0.

That is hardly what one would want from a weight regularization routine, so it’s protected to say that L0.5 isn’t an amazing weight regularizer. Let’s consider L3 regularization next.

L3 regularization

L3 regularization adds the typical of the cube of absolutely the value of the weights together because the regularization loss.

Figure-8: L3 regularization loss and its partial derivative with respect to every weight Wi.

This has the effect of adjusting each weight by a multiple (on this case alpha times the educational rate) of the square of the load itself within the direction that minimizes the loss.

Graphically, that is what the function and its derivative appear to be.

Figure-9: The blue line is pow(|w|, 3) and the red line is the derivative of pow(|w|, 3).

To actually understand what’s occurring here, we’d like to zoom in to the chart across the w=0 point.

Figure-10: The blue line is pow(|w|, 3) and the red line is the derivative of pow(|w|, 3), zoomed in at small values of w around 0.0.

You may see that the derivative of the L3 norm is a continuous and differentiable function (despite the presence of |w| within the derivative), which has a big magnitude at large values of w and a small magnitude for small values of w.

Interestingly, the gradient could be very near zero for very small values of w across the 0.0 mark.

The interpretation of the gradient for L3 is interesting.

1. For big values of w, the magnitude of the gradient is large. During backpropagation, these values will likely be pushed towards 0.
2. Once the load w reaches an inflection point (near 0.0), the gradient almost vanishes, and the weights will stop getting updated.

The effect is that it’ll drive the weights with large magnitudes near 0, but not exactly 0.

Let’s consider higher norms to see how this plays out within the limiting case.

Beyond L3 regularization

To know what happens for Linfinity, we’d like to see what happens within the case of the L10 regularization case.

Figure-11: The blue line is pow(|w|, 10) and the red line is the derivative of pow(|w|, 10), zoomed in at small values of w around 0.0.

One can see that the gradients for values of |w| < 0.5 are extremely small, which suggests that regularization won’t be effective for those values of w.

Exercise

Based on every little thing we saw above, L1 and L2 regularization are fairly practical based on what you must achieve. As an exercise, attempt to reason concerning the behavior of the L1.5 regularization, whose chart is shown below.

Figure-12: The blue line is pow(|w|, 1.5) and the red line is the derivative of pow(|w|, 1.5).

We took a visible and intuitive take a look at the L1 and L2 (and generally Lk) regularization terms to know why L1 regularization leads to sparse model weights and L2 regularization leads to model weights near 0. Framing the answer as inspecting the resulting gradients is amazingly worthwhile during this exercise.

We explored L0.5, L3, and L10 regularization terms and graphically, and also you (the reader) reasoned about regularization terms between L1 and L2 regularization, and developed an intuitive understanding of what implications it might have on a model’s weights.

We hope that this text has added to your toolbox of tricks you need to use when considering regularization strategies during model training to fine-tuning.

All of the charts in this text were created using the net desmos graphing calculator. Here’s a link to the functions utilized in case you would like to play with them.

All the photographs were created by the writer(s) unless otherwise mentioned.

We found the next articles useful while researching the subject, and we hope that you just find them useful too!

1. Stackexchange discussion
2. TDS: Demystifying L1 & L2 Regularization (part 3)
3. Visual explanation of L1 and L2 regularization
4. Deep Learning by Ian Goodfellow
5. An introduction to statistical learning by Gareth James