Forecasting with deep neural networks

Supervised learning involves training a machine learning model with an input data set. This data set is often a matrix: A two-dimensional data structure composed of rows (samples) and columns (features).

A time series is a sequence of values ordered in time. So, it must be transformed for supervised learning.

In a previous article, we learned the right way to transform a univariate time series from a sequence right into a matrix. This is completed with a sliding window. Each commentary of the series is modeled based on past recent values, also called lags.

Here’s an example of this transformation using the sequence from 1 to 10:

This transformation enables a sort of modeling called auto-regression. In auto-regression, a model is built using the past recent values (lags) of a time series as explanatory variables. These are used to predict future observations (goal variable). The intuition for the name auto-regression is that the time series is regressed with itself.

In the instance above, the lags are the initial 5 columns. The goal variable is the last column (the worth of the series in the following time step).

While most methods work with matrices, deep neural networks need a unique structure.

The input to deep neural networks equivalent to LSTMs or CNNs is a three-dimensional array. The actual data is identical because the one you’d put in a matrix. But, it’s structured otherwise.

Besides rows (samples) and columns (lags), the additional dimension refers back to the variety of variables within the series. In a matrix, you concatenate all attributes together no matter their source. Neural networks are a bit tidier. The input is organized by each variable within the series using a 3rd dimension.

Let’s do a practical example to make this clear.

On this tutorial, you’ll learn the right way to transform a time series for supervised learning with an LSTM (Long Short-Term Memory). An LSTM is a sort of neural network that is very useful to model time series.

We’ll split the time series transformation process into two steps:

1. From a sequence of values right into a matrix;
2. From a matrix right into a three-D array for deep learning.

First, we’ll do an example with a univariate time series. Multivariate time series are covered next.

Univariate Time Series

Let’s start by reading the info. We’ll use a time series related to the sales of various sorts of wine. You possibly can check the source in reference [1].

import pandas as pd
# https://github.com/vcerqueira/blog/tree/predominant/data
data = pd.read_csv('data/wine_sales.csv', parse_dates=['date'])
data.set_index('date', inplace=True)
series = data['Sparkling']

We concentrate on the sales of sparkling wine to do an example for the univariate case. This time series looks like this:

From a sequence of values right into a matrix

We apply a sliding window to remodel this series for supervised learning. You possibly can learn more about this process in a previous article.

# src module here: https://github.com/vcerqueira/blog/tree/predominant/src
from src.tde import time_delay_embedding
# using 3 lags as explanatory variables
N_LAGS = 3
# forecasting the following 2 values
HORIZON = 2
# using a sliding window method called time delay embedding
X, Y = time_delay_embedding(series, n_lags=N_LAGS, horizon=HORIZON, return_Xy=True)

Here’s a sample of the explanatory variables (X) and corresponding goal variables (Y):

This data set is the premise for training traditional machine learning methods. For instance, a linear regression or an xgboost.

from sklearn.linear_model import RidgeCV
# training a ridge regression model
model = RidgeCV()
model.fit(X, Y)

From a matrix right into a three-D structure for deep learning

You have to reshape this data set to coach a neural network like an LSTM. The next function may be used to do that:

import re
import pandas as pd
import numpy as np
def from_matrix_to_3d(df: pd.DataFrame) -> np.ndarray:
"""
Transforming a time series from matrix into three-D structure for deep learning
:param df: (pd.DataFrame) Time series within the matrix format after embedding
:return: Reshaped time series into three-D structure 
"""    
cols = df.columns
# getting unique variables within the time series
# this list has a single element for univariate time series
var_names = np.unique([re.sub(r'([^)]*)', '', c) for c in cols]).tolist()
# getting commentary for every variable
arr_by_var = [df.loc[:, cols.str.contains(v)].values for v in var_names]
# reshaping the info of every variable right into a three-D format
arr_by_var = [x.reshape(x.shape[0], x.shape[1], 1) for x in arr_by_var]
# concatenating the arrays of every variable right into a single array
ts_arr = np.concatenate(arr_by_var, axis=2)
return ts_arr
# transforming the matrices
X_3d = from_matrix_to_3d(X)
Y_3d = from_matrix_to_3d(Y)

Finally, you’ll be able to train an LSTM using the resulting data set:

from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import (Dense,
LSTM,
TimeDistributed,
RepeatVector)
# variety of variables within the time series
# 1 since the series is univariate
N_FEATURES = 1
# creating an easy stacked LSTM
model = Sequential()
model.add(LSTM(8, activation='relu', input_shape=(N_LAGS, N_FEATURES)))
model.add(RepeatVector(HORIZON))
model.add(LSTM(4, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(N_FEATURES)))
model.compile(optimizer='adam', loss='mse')
# compiling the model
model.compile(optimizer='adam', loss='mse')
# basic train/validation split 
X_train, X_valid, Y_train, Y_valid = train_test_split(X_3d, Y_3d, test_size=.2, shuffle=False)
# training the model
model.fit(X_train, Y_train, epochs=100, validation_data=(X_valid, Y_valid))
# making predictions
preds = model.predict_on_batch(X_valid)

Multivariate Time Series

Now, let’s take a look at a multivariate time series example. On this case, the goal is to forecast the longer term values of several variables, not only one. So, you wish a model for multivariate and multi-step forecasting.

The transformation process is like before.

To rework the multivariate time series right into a matrix format, you’ll be able to apply the sliding window approach to every variable. Then, you mix all resulting matrices right into a single one.

Here’s an example:

# transforming each variable right into a matrix format
mat_by_variable = []
for col in data:
col_df = time_delay_embedding(data[col], n_lags=N_LAGS, horizon=HORIZON)
mat_by_variable.append(col_df)
# concatenating all variables
mat_df = pd.concat(mat_by_variable, axis=1).dropna()
# defining goal (Y) and explanatory variables (X)
predictor_variables = mat_df.columns.str.comprises('(t-|(t)')
target_variables = mat_df.columns.str.comprises('(t+')
X = mat_df.iloc[:, predictor_variables]
Y = mat_df.iloc[:, target_variables]

The explanatory variables appear like this for 2 of the variables (others are omitted for conciseness):

You should utilize the identical function to remodel the info into three dimensions:

X_3d = from_matrix_to_3d(X)
Y_3d = from_matrix_to_3d(Y)

The training part can also be like before. The knowledge concerning the variety of variables within the series is provided within the N_FEATURES constant. Because the name implied, this constant is the variety of variables within the time series.

model = Sequential()
model.add(LSTM(8, activation='relu', input_shape=(N_LAGS, N_FEATURES)))
model.add(Dropout(.2))
model.add(RepeatVector(HORIZON))
model.add(LSTM(4, activation='relu', return_sequences=True))
model.add(Dropout(.2))
model.add(TimeDistributed(Dense(N_FEATURES)))
model.compile(optimizer='adam', loss='mse')
X_train, X_valid, Y_train, Y_valid = train_test_split(X_3d, Y_3d, test_size=.2, shuffle=False)
model.fit(X_train, Y_train, epochs=500, validation_data=(X_valid, Y_valid))
preds = model.predict_on_batch(X_valid)

The next plot shows a sample of one-step ahead forecasts.

The forecasts aren’t that good. The time series is small and we didn’t optimize the model in any way. Deep learning methods are known to be data-hungry. So, in case you go for this type of approach, ensure you will have enough data.