The bias-variance tradeoff, part 1 of three

Must you read this text? Should you understand all of the words in the subsequent section, then no. Should you don’t care to grasp them, then also no. Should you want the bolded bits explained, then yes.

“The bias-variance tradeoff” is a preferred phrase you’ll hear within the context of ML/AI. Should you’re a statistician, you would possibly think it’s about summarizing this formula:

It isn’t.

Well, it’s loosely related, however the phrase actually refers to a practical recipe for easy methods to pick a model’s . It’s most useful whenever you’re tuning a .

Should you’ve never heard of the MSE, you would possibly need a little bit of help with among the jargon. Once you hit a latest term you wish explained in additional detail, you may follow the links to my other articles where I introduce the words I’m using.

The mean squared error (MSE) is the preferred (and vanilla) selection for a model’s loss function and it tends to be the primary one you’re taught. You’ll likely take a complete bunch of stats classes before it occurs to anyone to inform you that you simply’re welcome to reduce other loss functions should you like. (But let’s be real: parabolae are super easy to optimize. Remember d/dx x²? 2x. That convenience is sufficient to keep most of you loyal to the MSE.)

When you learn concerning the MSE, it’s often mere moments until someone mentions the bias and variance formula:

MSE = Bias² + Variance

I did it too and, like a garden variety data science jerk, left the proof as homework for the interested reader.

Let’s make amends — should you’d like me to derive it for you while making snide comments within the margins, take a small detour to here. Should you decide to skip the mathy stuff, you then’ll should put up with my jazz hands and just take my word for it.

Want me to inform the important thing thing to you bluntly? Notice that the formula consists of two terms that .

The amount (MSE) you’re attempting to optimize whenever you suit your predictive ML/AI models may be decomposed into that involve bias only and variance only.

MSE = Bias² + Variance = (Bias)² + (Standard Deviation)²

Much more bluntly? Okay, sure.

A greater model has a lower MSE. E stands for error and fewer errors are higher, so the best model has a zero MSE: it makes no mistakes. Which means it also has no bias and no variance.

As an alternative of perfect models, let’s have a look at going from good to higher. Should you became a greater archer, you became a greater archer. No tradeoff. (You almost certainly needed more practice — data! — to get there.)

As Tolstoy would say, all perfect models are alike, but each unhappy model may be unhappy in its own way.

As Tolstoy would say, all perfect models are alike, but each unhappy model (for a given MSE) may be unhappy in its own way. You’ll be able to get two equally rubbish yet different models with the identical MSE: one model can have really good (low) bias but high variance while the opposite can have really good (low) variance but high bias, and yet each can have the identical MSE (overall rating).

If we measure the performance of an archer by MSE, we’re saying that decreasing an archer’s standard deviation is well worth the same as an equivalent decrease in bias. We’re saying we’re indifferent between the 2. (Wait, what if you’re not indifferent between them? Then the MSE won’t be your best option here. Don’t just like the MSE’s way of scoring performance? Not an issue. Make your personal loss function.)

Now that we’ve set the table, head over to Part 2 where we dig into the center of the matter: Is there an actual tradeoff? (Yes! But not where you would possibly think.) And what does overfitting should do with it? (Hint: every part!)

Should you rejoiced here and also you’re searching for a complete applied AI course designed to be fun for beginners and experts alike, here’s the one I made on your amusement: