## All it’s worthwhile to find out about Linear Regression is here (including an application in Python)

If you’re approaching Machine Learning, one among the primary models you might encounter is Linear Regression. It’s probably the best model to know, but don’t underestimate it: there are lots of things to know and master.

For those who’re a beginner in Data Science or an aspiring Data Scientist, you’re probably facing some difficulties because there are lots of resources on the market, but are fragmented. I understand how you’re feeling, and this is the reason I created this entire guide: I would like to offer you all of the knowledge you would like without trying to find anything.

So, if you ought to have complete knowledge of Linear Regression this text is for you. You possibly can study it deeply and re-read it every time you would like it probably the most. Also, consider that, to cover this topic, we’ll need some knowledge generally related to regression evaluation: we’ll cover it in deep.

And…you’ll excuse me if I’ll link a resource you’ll need: up to now, I’ve created an article on some topics related to Linear Regression so, to have an entire overview, I counsel you to read it (I’ll link later after we’ll need it).

What can we mean by "regression evaluation"?

Understanding correlation

The difference between correlation and regression

The Linear Regression model

Assumptions for the Linear Regression model

Finding the road that most closely fits the info

Graphical methods to validate your model

An example in Python

Here we’re studying Linear Regression, but what can we mean by “regression evaluation”? Paraphrasing from Wikipedia:

Regression evaluation is a mathematical technique used to search out a functional relationship between a dependent variable and a number of independent variable(s).

In other words, we all know that in mathematics we will define a function like so: `y=f(x)`

. Generally, `y`

is known as the dependent variable and `x`

the independent. So, we express `y`

in relationship with `x`

, using a certain function `f`

. The aim of regression evaluation is, then, to search out the function `f`

.

Now, this seems easy but will not be. And I do know it. And the rationale why will not be easy is:

- We all know
`x`

and`y`

. For instance, if we’re working with tabular data (with`Pandas`

, for instance)`x`

are the features and`y`

is the label. - Unfortunately, the info rarely follow a really clear path. So our job is to search out one of the best function
`f`

that the connection between`x`

and`y`

.

So, let me summarize it: regression evaluation goals to search out an estimated relationship (a very good one!) between the dependent and the independent variable(s).

Now, let’s visualize why this process could also be difficult. Consider the next code and its final result:

`import numpy as np`

import matplotlib.pyplot as plt# Create random linear data

a = 130

x = 6*np.random.rand(a,1)-3

y = 0.5*x+5+np.random.rand(a,1)

# Labels

plt.xlabel('x')

plt.ylabel('y')

# Plot a scatterplot

plt.scatter(x,y)

Now, tell me: can the connection between `x`

and `y`

be a line? So…can this data be approximated by a line? Just like the following, for instance:

Stop reading for a moment and take into consideration that.

Well, it could. And the way in regards to the following one?

Well, even this might! So, what’s one of the best one? And why not one other one?

That is the aim of regression: to search out the best-estimated function that may approximate the given data. And it does so using some methodologies: we’ll cover them later in this text. We’ll apply them to the Linear Regression model but a few of them may be used with another regression technique. Don’t worry: I’ll be very specific so that you don’t get confused.

Quoting from Wikipedia:

In statistics, correlation is any statistical relationship, whether causal or not, between two random variables. Although within the broadest sense, “correlation” may indicate any style of association, in statistics it often refers back to the degree to which a pair of variables are linearly related.

In other words, is a statistical measure that expresses the .

We are able to say that two variables are correlated if each value of the primary variable corresponds to a worth for the second variable, following a path. If two variables are highly correlated, the trail could be linear, since the correlation describes the linear relation between the variables.

## The mathematics behind the correlation

It is a comprehensive guide, as promised. So, I would like to cover the mathematics behind the correlation, but don’t worry: we’ll make it easy so you can understand it even should you’re not specialized in math.

We generally check with the correlation coefficient, also often known as the . This offers an estimate of the correlation between two variables. Suppose we have now two variables, `a`

and `b`

they usually can reach `n`

values. We are able to calculate the correlation coefficient as follows:

Where we have now:

- the mean value of
`a`

(but it surely applies to each variables,`a`

and`b`

):

If we have now a 0 correlation coefficient, it implies that the info points don’t are likely to increase or decrease following a linear path, because we have now no correlation.

Allow us to have a have a look at some plots of correlation coefficients with different values (image from Wikipedia here):

As we will see, when the correlation coefficient is the same as 1 or -1 the tendency of the info points is clearly to be along a line. But, because the correlation coefficient deviates from the 2 extreme values, the distribution of the info points deviates from a linear path. Finally, for the correlation coefficient of 0, the distribution of the info may be anything.

So, after we get a correlation coefficient of 0 we will’t say anything in regards to the distribution of the info, but we will investigate it (if needed) with a regression evaluation.

So, correlation and regression are linked but are different:

- Correlation analyzes the tendency of variables to be linearly distributed.
- Regression is the study of the connection between variables.

We’ve two sorts of Linear Regression models: the Easy and the Multiple ones. Let’s see them each.

## The Easy Linear Regression model

The goal of the Easy Linear Regression is to model the connection between a single feature and a continuous label. That is the mathematical equation that describes this ML model:

`y = wx + b`

The parameter `b`

(also called “bias”) represents the y-axis intercept (is the worth of `y`

when `X=0`

), and `w`

is the burden coefficient. Our goal is to learn the burden `w`

that describes the connection between `x`

and `y`

. This weight will later be used to predict the response for brand new values of `x`

.

Let’s consider a practical example:

`import numpy as np`

import matplotlib.pyplot as plt# Create data

x = np.array([1, 1, 2, 3, 4, 4, 5, 6, 7, 7, 8, 9])

y = np.array([13, 14, 17, 12, 23, 24, 25, 25, 24, 28, 32, 33])

# Show scatterplot

plt.scatter(x, y)

The query is: can this data distribution be approximated with a line? Well, we could create something like that:

`import numpy as np`

import matplotlib.pyplot as plt# Create data

x = np.array([1, 1, 2, 3, 4, 4, 5, 6, 7, 7, 8, 9])

y = np.array([13, 14, 17, 12, 23, 24, 25, 25, 24, 28, 32, 33])

# Create basic scatterplot

plt.plot(x, y, 'o')

# Obtain m (slope) and b (intercept) of a line

m, b = np.polyfit(x, y, 1)

# Add linear regression line to scatterplot

plt.plot(x, m*x+b)

# Labels

plt.xlabel('x variable')

plt.ylabel('y variable')

Well, as in the instance we’ve seen above, it could possibly be a line but it surely could possibly be a general curve.

And, in a moment we’ll see how we will say if the info distribution may be higher described by a line or by a general curve.

## The Multiple Linear Regression model

Since reality is complex, the standard cases we’ll face are related to the Multiple Linear Regression case. We mean that the feature `x`

will not be a single one: we’ll have multiple features. For instance, if we work with tabular data, a knowledge frame with 9 columns has 8 features and 1 label: because of this our problem is eight-dimensional.

As we will understand, this case may be very complicated to visualise and the equation of the road must be expressed with vectors and matrices, becoming:

So, the equation of the road becomes the sum of all of the weights (`w`

) multiplied by the independent variable (`x`

) and it may well even be written because the product of two matrices.

Now, to use the Linear Regression model, our data should respect some assumptions. These are:

- : the connection between the dependent variable and independent variables must be linear. Because of this a change within the independent variable should end in a proportional change within the dependent variable, following a linear path.
- : the observations within the dataset must be independent of one another. Because of this the worth of 1 commentary mustn’t depend upon the worth of one other commentary.
- : the variance of the residuals must be constant across all levels of the independent variable. In other words, the spread of the residuals must be roughly the identical across all levels of the independent variable.
- : the residuals must be normally distributed. In other words, the distribution of the residuals must be a traditional (or bell-shaped) curve.
- : the independent variables mustn’t be highly correlated with one another. If two or more independent variables are highly correlated, it may well be difficult to differentiate the person effects of every variable on the dependent variable.

Unfortunately, testing all these hypotheses will not be all the time possible, especially within the case of the Multiple Linear Regression model. Anyway, there may be a solution to test all of the hypotheses. It’s called the `p-value`

test, and perhaps you heard of that before. Anyway, we won’t cover this test here for 2 reasons:

- It’s a general test, not specifically related to the Linear Regression model. So, it needs a selected treatment in a dedicated article.
- I’m one among those (perhaps one among the few) who believes that calculating the
`p-value`

will not be all the time a must when we want to investigate data. Because of this, I’ll create in the long run a dedicated article on this controversial topic. But only for the sake of curiosity, since I’m an engineer I even have a really practical approach, and I like applied mathematics. I wrote an article on this topic here:

So, above we were reasoning which one among the next may be one of the best fit:

To know if one of the best model is the left one (the road) or the suitable one (a general curve) we proceed as follows:

- We split the info we have now into the training and the test set.
- We validate each models on each sets, testing how well our models generalize their learning.

We won’t cover the polynomial model here (useful for general curves), but consider that there are two approaches to validate ML models:

- The analytical one.
- The graphical one.

Generally speaking, we’ll use each to get a greater understanding of the performance of the model. Anyway, implies that our ML model learns from the training set and . If it doesn’t, we try one other ML model. Here’s the method:

Because of this .

I’ve discussed the analytical solution to validate an ML model within the case of linear regression in the next article:

I counsel you to read it because we’ll use some metrics discussed there in the instance at the tip of this text.

After all, the metrics discussed may be applied to any ML model within the case of a regression problem. But you’re lucky: I’ve used the linear model for example.

The graphical ways to validate an ML model within the case of a regression problem are discussed in the subsequent paragraph.

Let’s see three graphical ways to validate our ML models.

## 1. The residual evaluation plot

This method is restricted to the Linear Regression model and consists in visualizing how the residuals are distributed. Here’s what we expect:

To plot this we will use the built-in function `sns.residplot()`

in `Seaborn`

(here’s the documentation).

A plot like that is nice because we wish to see randomly distributed data points along the horizontal axis. One among the , in actual fact, is that the (assumption n°4 listed above). If the residuals are normally distributed, it implies that the errors of the observed values from the expected ones are randomly distributed around zero, with no clear pattern or trend; and this is precisely the case in our plot. So, in these cases, our ML model could also be a very good one.

As a substitute, if there may be a selected pattern in our residual plot, our model will not be good for our ML problem. For instance, consider the next:

On this case, we will see that there’s a parabolic trend: because of this our model (the Linear model) will not be good to resolve our ML problem.

## 2. The actual vs. predicted values plot

One other plot we may use to validate our ML model is the . On this case, we plot a graph having the actual values on the horizontal axis and the expected values on the vertical axis. The goal is to search out the info points distributed as much as possible to a line, within the case of Linear Regression. We are able to even use the strategy within the case of a polynomial regression: on this case, we’d expect the info distributed as much as possible to a generic curve.

Suppose we have now a result as follows:

The above graph shows that the expected data points are distributed along a line. It will not be an ideal linear distribution, so the linear model is probably not ideal.

If, for our specific problem, we have now`y_train`

(the label on the training set) and we’ve calculated `y_train_pred`

(the prediction on the training set), we will plot the next graph like so:

`import matplotlib.pyplot as plt`# Scatterplot of y_train and y_train_pred

plt.scatter(y_train, y_train_pred)

plt.plot(y_test, y_test, color='r') # Plot the road

# Labels

plt.title('ACTUAL VS PREDICTED VALUES')

plt.xlabel('ACTUAL VALUES')

plt.ylabel('PREDICTED VALUES')

## 3. The Kernel Density Estimation (KDE) plot

The last graph we wish to discuss to validate our ML models is the Kernel Density Estimation (KDE) plot. It is a general method and may be used to validate each regression and classification models.

The KDE is the appliance of a for probability density estimation. A kernel smoother is a statistical method that’s used to estimate a function because the weighted average of the neighbor observed data. The kernel defines the burden, giving a better weight to closer data points.

To know the usefulness of a smoother function, see the graph below:

It is useful to approximate our data points with a smoothing function if we wish to check two quantities. Within the case of an ML problem, in actual fact, we typically prefer to see the comparison between the actual labels and the labels predicted by our model, so we use the KDE to check two smoothed functions.

Let’s say we have now predicted our labels using a linear regression model. We wish to check the KDE for our training set’s actual and predicted labels. We are able to accomplish that with `Seaborn`

invoking the strategy `sns.kdeplot()`

(here’s the documentation).

Suppose we have now the next result:

As we will see, the comparison between the actual and the expected label is simple to do, since we’re comparing two smoothed functions; in a case like that, our model is nice since the curves are very similar.

In reality, what we expect from a “good” ML model are:

- The curves are much like bell curves, as much as possible.
- The 2 curves are similar between them, as much as possible.

Now, let’s apply all of the things we’ve learned to this point here. We’ll use the famous “Ames Housing” dataset, which is ideal for our scopes.

This dataset has 80 features, but for simplicity, we’ll work with only a subset of them that are:

`Overall Qual`

: it’s the rating of the general material and finish of the home on a scale from 1 (bad) to 10 (excellent).`Overall Cond`

: it’s the rating of the general condition of the home on a scale from 1 (bad) to 10 (excellent).`Gr Liv Area`

: it’s the above-ground living area, measured in squared feet.`Total Bsmt SF`

: it’s the overall basement area, measured in squared feet.`SalePrice`

: it’s the sale price, in USD $.

We’ll consider our `SalePrice`

column because the goal (label) variable, and the opposite columns because the features.

## Exploratory Data Evaluation EDA

Let’s import our data, create a subset with the mentioned features, and display some statistics:

`import pandas as pd`# Define the columns

columns = ['Overall Qual', 'Overall Cond', 'Gr Liv Area',

'Total Bsmt SF', 'SalePrice']

# Create dataframe

df = pd.read_csv('http://jse.amstat.org/v19n3/decock/AmesHousing.txt',

sep='t', usecols=columns)

# Show statistics

df.describe()

A very important commentary here is that the mean values for all labels have a special range (the `Overall Qual`

mean value is `6.09`

while `Gr Liv Area`

mean value is `1499.69`

). This tells us a vital fact: we have now to scale the features.

## Data preparation

What does “” mean?

Scaling a feature implies that the feature range is scaled between 0 and 1 or between 1 and -1. There are two typical methods to scale the features:

- Mean normalization is a technique of scaling numeric data in order that it has a minimum value of zero and a maximum value of every person the values are normalized across the mean value. Suppose
*c*is a worth reached by our feature; to scale across the mean (*c*′ is the brand new value of*c*after the normalization process):

Let’s see an example in Python:

`import numpy as np`# Create a listing of numbers

data = [1, 2, 3, 4, 5]

# Find min and max values

data_min = min(data)

data_max = max(data)

# Normalize the info

data_normalized = [(x - data_min) / (data_max - data_min) for x in data]

# Print the normalized data

print(f'normalized data: {data_normalized}')

>>>

normalized data: [0.0, 0.25, 0.5, 0.75, 1.0]

- (or z-score normalization): This method transforms a variable in order that it has a mean of zero and a regular deviation of 1. The formula is the next (c′c’c′ is the brand new value of ccc after the normalization process):

Let’s see an example in Python:

`import numpy as np`# Original data

data = [1, 2, 3, 4, 5]

# Calculate mean and standard deviation

mean = np.mean(data)

std = np.std(data)

# Standardize the info

data_standardized = [(x - mean) / std for x in data]

# Print the standardized data

print(f'standardized values: {data_standardized}')

print(f'mean of standardized values: {np.mean(data_standardized)}')

print(f'std. dev. of standardized values: {np.std(data_standardized): .2f}')

>>>

standardized values: [-1.414213562373095, -0.7071067811865475, 0.0, 0.7071067811865475, 1.414213562373095]

mean of standardized values: 0.0

std. dev. of standardized values: 1.00

As we will see, the normalized data have a mean of 0 and a regular deviation of 1, as we wanted. The excellent news is that we will use the library `scikit-learn`

to standardize the features, and we will do it in a moment.

Features scaling is a vital thing to do when working on an ML problem, for a straightforward reason:

- If we perform exploratory data evaluation with features that usually are not scaled, when calculating the mean values (for instance, in the course of the calculation of the coefficient of correlation) we’ll get numbers which are very different from one another. If we take a have a look at the statistics we’ve got above after we’ve invoked the
`df.describe()`

method, we will see that, for every column, we get a really different value of the mean. If we scale or normalize the features, as an alternative, we’ll get 0s, 1s, and -1s: and this can help us mathematically.

Now, this dataset has some `NaN`

values. We won’t show it for brevity (try it on your individual), but we’ll remove them. Also, we’ll calculate the correlation matrix:

`import seaborn as sns`

import matplotlib.pyplot as plt

import numpy as np# Drop NaNs from dataframe

df = df.dropna(axis=0)

# Apply mask

mask = np.triu(np.ones_like(df.corr()))

# Heat map for correlation coefficient

sns.heatmap(df.corr(), annot=True, fmt="0.1", mask=mask)

So, with `np.triu(np.ones_like(df.corr()))`

we have now created a mask that it’s useful to display a triangular correlation matrix, which is more readable (especially when we have now way more features than on this case).

So, there may be a moderate `0.6`

correlation between `Total Bsmt SF`

and `SalePrice`

, quite a high `0.7`

correlation between `Gr Liv Area`

and `SalePrice`

, and a high correlation `0.8`

between `Overall Qual`

and `SalePrice`

; Also, there may be a moderate correlation between `Overall Qual`

and `Gr Liv Area`

`0.6`

and `0.5`

between `Overall Qual`

and `Total Bsmt SF`

.

Here there’s no multicollinearity, so no features are highly correlated with one another (so, our features satisfy the hypothesis n°5 listed above). If we’d found some highly correlated features, we could delete them because ().

Finally, we subdivide the info frame `df`

into `X`

( the features) and `y`

(the label) and scale the features:

`from sklearn.preprocessing import StandardScaler`# Define the features

X = df.iloc[:,:-1]

# Define the label

y = df.iloc[:,-1]

# Scale the features

scaler = StandardScaler() # Call the scaler

X = scaler.fit_transform(X) # Fit the features to scale them

## Fitting the linear regression model

Now we have now to separate the features `X`

into the training and the test set and we’re fitting them with the Linear Regression model. Then, we calculate R² for each sets:

`from sklearn.model_selection import train_test_split`

from sklearn.linear_model import LinearRegression

from sklearn import metrics# Split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Fit the LR model

reg = LinearRegression().fit(X_train, y_train)

# Calculate R^2

coeff_det_train = reg.rating(X_train, y_train)

coeff_det_test = reg.rating(X_test, y_test)

# Print metrics

print(f" R^2 for training set: {coeff_det_train}")

print(f" R^2 for test set: {coeff_det_test}")

>>>

R^2 for training set: 0.77

R^2 for test set: 0.73

1) your results may be barely different resulting from the stocastical

nature of the ML models.2) here we will see generalization on motion:

we fitted the Linear Regression model to the train set with

*reg = LinearRegression().fit(X_train, y_train)*.

The, we have calculated R^2 on the training and test sets with:

*coeff_det_train = reg.rating(X_train, y_train)*

coeff_det_test = reg.rating(X_test, y_test

In other words: we do not fit the info to the test set.

We fit the info to the training set and we calculate the scores

and predictions (see next snippet of code with KDE) on each sets

to see the generalization of our modelon latest unseen data

(the info of the test set).

So we get R² of 0.77 on the training test and 0.73 on the test set that are quite good, suggesting the Linear model is a very good one to resolve this ML problem.

Let’s see the KDE plots for each sets:

`# Calculate predictions`

y_train_pred = reg.predict(X_train) # train set

y_test_pred = reg.predict(X_test) # test set# KDE train set

ax = sns.kdeplot(y_train, color='r', label='Actual Values') #actual values

sns.kdeplot(y_train_pred, color='b', label='Predicted Values', ax=ax) #predicted values

# Show title

plt.title('Actual vs Predicted values')

# Show legend

plt.legend()

`# KDE test set`

ax = sns.kdeplot(y_test, color='r', label='Actual Values') #actual values

sns.kdeplot(y_test_pred, color='b', label='Predicted Values', ax=ax) #predicted values# Show title

plt.title('Actual vs Predicted values')

# Show legend

plt.legend()

Whatever the undeniable fact that we’ve obtained an R² of 0.73 on the test set which is nice (but remember: the upper, the higher), this plot shows us that the linear model is indeed a very good model to resolve this ML problem. This is the reason I like the KDE plot: is a really powerful tool, as we will see.

Also, this shows why shouldn’t depend on only one method to validate our ML model: a mix of 1 analytical method with one graphical one generally gives us the suitable insights to come to a decision whether to alter our ML model or not. On this case, the Linear Regression model is ideal to make predictions.

I hope you’ll find useful this text. I comprehend it’s very long, but I wanted to offer you all of the knowledge you would like on this topic, so you can return to it every time you would like it probably the most.

A number of the things we’ve discussed listed here are general topics, while others are specific to the Linear Regression model. Let’s summarize them:

- The definition of is, after all, a general definition.
- is usually known as the Linear modelIn fact, as we said before, correlation is the tendency of two variables to be linearly dependent.Howeverthere are ways to define non-linear correlations, but we leave them for other articles (but, as knowledge for you: just consider that they exist).
- We’ve discussed the Easy and the Multiple Linear Regression models with their assumptions (the assumptions apply to each models).
- When talking about find out how to find the road that most closely fits the info, we’ve referred to the article “Mastering the Art of Regression Evaluation: 5 Key Metrics Every Data Scientist Should Know”. Here, we discover all of the metrics to know to resolve a regression evaluation. So, this can be a generical topic that applies to any regression model, including the Linear one, after all.
- We’ve shown three methods to validate our ML models: 1) : which applies to Linear Regression models, 2) : which may be applied to Linear and Polynomial models, 3) the : this may be applied to any ML model, even within the case of a classification problem

Finally, I would like to remind you that we’ve spent a few lines stressing the undeniable fact that we will avoid using `p-values`

to check the hypotheses of our ML models. I’m writing an article on this topic very soon, but, as you possibly can see, the KDE has shown us that our Linear model is nice to resolve this ML problem, and we haven’t validated our hypothesis with `p-values`

.

*Thus far in this text, we’ve used some plots. You possibly can **clone this repo** I’ve created so you can import the code and use it to simply plot the graphs. If you’ve gotten some difficulties, you discover examples of usages on my projects on GitHub. If you’ve gotten another difficulties, you possibly can **contact me** and I’ll aid you.*

*Subscribe to**my newsletter**to get more on Python & Data Science.**Found it useful?**Liked the article? Join Medium through**: unlock all of the content on Medium for five$/month (with no additional fee).**Find/contact me**here**.*