The bias-variance tradeoff, part 2 of three

In Part 1, we covered much of the fundamental terminology in addition to a couple of key insights concerning the bias-variance formula (), including this misquote from Anna Karenina:

All perfect models are alike, but each unhappy model could be unhappy in its own way.

To take advantage of this article, I suggest taking a take a look at Part 1 to make certain you’re well-situated to soak up this one.

Let’s say you will have a model that’s pretty much as good as you’re going to get for the data you will have.

To have a fair higher model, you wish higher data. In other words, more data (quantity) or more relevant data (quality).

After I say as good as you’re going to get, I mean in “good” terms of MSE performance on data your model hasn’t seen before. (It’s speculated to predict, not postdict.) You’ve done an ideal job of getting what you may from the data you will have — the remainder is error you may’t do anything about together with your information.

Reality = Best Model + Unavoidable Error

But here’s the issue… we’ve jumped ahead;

All you will have is a pile of old data to learn this model from. Eventually, in case you’re smart, you’ll validate this model on data it hasn’t seen before, but first you will have to learn the model by finding useful patterns in data and attempting to inch closer and closer to the stated objective: an MSE that’s as little as possible.

Unfortunately, throughout the learning process, you don’t get to look at the MSE you’re after (the one which comes from reality). You simply get to compute a shoddy version out of your current training dataset.

Oh, and in addition, in this instance “you” usually are not a human, you’re an optimization algorithm that was told by your human boss to twiddle the dials within the model’s settings until the MSE is as little as it would go.

You say, “Sweet! I can do that!! Boss, in case you give me a particularly flexible model with numerous settings to fiddle (neural networks, anyone?), I can provide you with an ideal training MSE. No bias and no variance.”

The option to get a greater training MSE than the true model’s test MSE is to suit all of the noise (errors you will have no predictively-useful details about) together with the signal. How do you achieve this little miracle? By making the model more complicated. Connecting the dots, essentially.

This known as overfitting. Such a model has a wonderful training MSE but a whopper of a variance once you try to make use of it for anything practical. That’s what you get for attempting to cheat by creating an answer with more complexity than your information supports.

The boss is just too smart to your tricks. Knowing that a flexible, complicated model means that you can rating too well in your training set, the boss changes the scoring function to penalize complexity. This known as regularization. (Frankly, I wish we had more regularization of engineers’ antics, to stop them from doing complicated things for complexity’s sake.)