2️⃣. Judging Model Performance Only By Test Scores

You bought a test rating over 0.85 — do you have to be celebrating? Big, fat NO!

Regardless that high test scores generally mean robust performance, there are necessary caveats to interpreting test results. First and most significantly, whatever the value, test scores should only be judged based on the rating you get from training.

The one time you ought to be joyful together with your model is when the training rating is higher than the test rating, and each are high enough to satisfy the expectations of your unique case. Nevertheless, this doesn’t imply that the upper the difference between train and test scores, the higher.

For instance, 0.85 training rating and 0.8 test rating suggest a very good model that’s neither overfit nor underfit. But, if the training rating is over 0.9 and the test rating is 0.8, your model is overfitting. As a substitute of generalizing during training, the model memorized a number of the training data leading to a much lower test rating.

You’ll often see such cases with tree-based and ensemble models. For instance, algorithms resembling Random Forest are likely to achieve very high training scores if their tree depth shouldn’t be controlled, resulting in overfitting. You possibly can read this discussion on StackExchange to learn more in regards to the difference between train and test scores.

There may be also the case where the test rating is higher than the train. If the test rating is higher than training even within the slightest, feel alarmed since you made a mistake! The key reason for such scenarios is data leakage, and we discussed an example of that within the last section.

Sometimes, it’s also possible to get a very good training rating and a particularly low testing rating. When the difference between train and test scores is large, the issue will often be related to the test set slightly than overfitting. This might occur through the use of different preprocessing steps for the train and test sets or just forgetting to use preprocessing to the test set.

In summary, all the time examine the gap between train and test scores closely. Doing so will inform you whether you need to apply regularization to beat overfitting, search for possible mistakes you made during preprocessing or the best-case scenario, prepare the model for final evaluation and deployment.