Discussing theory and implementation with Python and Scikit-learn

I even have already discussed various kinds of dimensionality reduction techniques intimately.

Principal Component Evaluation (PCA), Factor Evaluation (FA), Linear Discriminant Evaluation (LDA), Autoencoders (AEs), and Kernel PCA are the most well-liked ones.

Non-Negative Matrix Factorization (NMF or NNMF) can also be a linear dimensionality reduction technique that will be used to scale back the dimensionality of the feature matrix.

All dimensionality reduction techniques fall under the category of unsupervised machine learning during which we are able to reveal hidden patterns and necessary relationships in the information without requiring labels.

So, dimensionality reduction algorithms take care of unlabeled data. When training such an algorithm, the fit() method only needs the feature matrix, X because the input and it doesn’t require the label column, y.

As its name implies, non-negative matrix factorization (NMF) needs the feature matrix to be non-negative.

For this reason non-negativity constraint, the usage of NMF is restricted to data with non-negative values equivalent to image data (pixel values all the time lie between 0 and 255, hence there are not any negative values in image data!).

What you'll learn:
----------------------------------------------------
1. Maths behind NMF
2. NMF equation
3. Feature matrix, V
4. Transformed data matrix, W
5. Factorization matrix, H
6. Scikit-learn NMF() class
7. Arguments, methods and attributes of NMF() class
8. Load the MNIST with Scikit-learn
9. Perform dimensionality reduction in image data
Other matrix decomposition methods:
----------------------------------------------------
1. Eigendecomposition
2. Singular value decomposition

Non-negative matrix factorization comes from linear algebra. In easy words, it’s the technique of decomposing a matrix into the product of two small matrices.

To be more precise,

Non-negative matrix factorization (NMF) is the technique of decomposing a non-negative feature matrix, V (nxp) right into a product of two non-negative matrices called W (nxd) and H (dxp). All three matrices should contain non-negative elements.

The product of W and H matrices only gives an approximation to the matrix V. So, it is best to expect some information loss when applying NMF.

• V (n x p): Represents the feature matrix where n is the variety of observations (samples) and p is the variety of features (variables). That is the information matrix we decompose.
• W (n x d): Represents the transformed data matrix after applying NMF. We are able to use this transformed matrix rather than the unique feature matrix, V. So, W is crucial output of NMF. It’s obtained by calling Scikit-learn NMF’s fit_transform() method. n is the variety of observations (samples) and d is the variety of latent aspects or components. In other words, d describes the quantity of dimensionality that we wish to maintain. It is definitely a hyperparameter that we want to specify within the Scikit-learn NMF’s n_components argument. That is an integer value that must be lower than the variety of features, p and greater than 0. Choosing the appropriate value for d is an actual challenge when performing NMF. We’d like to think about the balance between the quantity of knowledge and the variety of components that we wish to maintain.

from sklearn.decomposition import NMF
# W = transformed data matrix, V = original feature matrix
W = NMF(n_components=d).fit_transform(V)

• H (d x p): Represents the factorization matrix. d and p have the identical definitions above. This matrix will not be very necessary. Nonetheless, this will be obtained by calling the Scikit-learn NMF’s components_ attribute.

from sklearn.decomposition import NMF
# H = factorization matrix
H = NMF(n_components=d).fit(V).components_

In Python, NMF is implemented through the use of Scikit-learn’s NMF() class. As you already know, Scikit-learn is the Python machine learning library.

All you should do is to import the NMF() class and create an instance of it by specifying the required arguments.

# Import
from sklearn.decomposition import NMF
# Create an instance
nmf_model = NMF(n_components, init, random_state)

Vital arguments of NMF() class

• n_components: An integer that defines the variety of components or latent aspects or the quantity of dimensionality that we wish to maintain. Crucial hyperparameter! The worth is lower than the variety of original features and greater than 0.
• init: A technique of the initialization process. To learn more about this, check with the Scikit-learn documentation here. The output returned by the NMF model will significantly vary depending on the init method you select.
• random_state: Used when the initialization method is ‘nndsvdar’ or ‘random’. Use an integer to get the identical results across different executions. Learn more here.

Note: There are various arguments within the NMF() class. If we don’t specify them, they take their default values when calling the NMF() function. To learn more about those arguments, check with the Scikit-learn documentation.

Vital methods of NMF() class

• fit(V): Learns an NMF model from the feature matrix, V. No transformation is applied here.
• fit_transform(V): Learns an NMF model from the feature matrix, V and returns the transformed data matrix, W.

W = nmf_model.fit_transform(V)

• transform(V): Returns the transformed data matrix, W after fitting the model.

nmf_model.fit(V) # Fitted model
W = nmf_model.transform(V)

• inverse_transform(W): Transforms (recovers) the information matrix, W back to the unique space. Very useful for visualization purposes!

recovered_data = nmf_model.inverse_transform(W)

Vital attributes of NMF() class

• components_: Returns the factorization matrix, H. This matrix will not be very necessary.

H = nmf_model.components_

• reconstruction_err_: Returns the beta divergence as a float that measures the gap between V and the product of WH. The solver tries to attenuate this error through the training process. Analyzing this error by setting different values for n_components is an excellent way of selecting the appropriate variety of components, d.

We’ll use the MNIST digits dataset for this task. We’ll perform NMF on MNIST data to scale back dimensionality by selecting different numbers of components after which we compare each output with the unique one.

Step 1: Load the MNIST dataset using Scikit-learn

The MNIST digits dataset will be loaded as follows using Scikit-learn.

from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1)
image_data = mnist['data']
print("Shape:", image_data.shape)
print("Type:", type(image_data))

The dataset is loaded as a Pandas data frame. The form is (70000, 784). There are 70000 observations (images) within the dataset. Each statement has 784 features (pixel values). The scale of a picture is 28 x 28. When loading the MNIST dataset in this manner, each image is represented as a 1D array that incorporates 784 (28 x 28) elements. That is the format we want for this task and no further modification is required for the dataset.

Alternatively, you may load the MNIST dataset using Keras. There, you’ll get a 28 x 28 2D array for every image as an alternative of a 1D array. You may learn more here.

Step 2: Visualize a sample of the unique images

Now, we’ll visualize a sample of the primary five images within the MNIST dataset. This sample will be used to match with the outputs of the NMF model.

import matplotlib.pyplot as plt
n = 5
plt.figure(figsize=(6.75, 1.5))
for i in range(n):
ax = plt.subplot(1, n, i+1)
plt.imshow(image_data.iloc[i].values.reshape(28, 28), cmap="binary")
ax.axis('off')
plt.show()

Step 3: Apply NMF with 9 components (d = 9)

from sklearn.decomposition import NMF
nmf_model = NMF(n_components=9, init='random', random_state=0)
image_data_nmf = nmf_model.fit_transform(image_data)
print("Shape:", image_data_nmf.shape)
print("Type:", type(image_data_nmf))

Now, the brand new dimensionality is 9. The unique dimensionality was 784. Due to this fact, the dimensionality has been significantly reduced!

To get the form of V, W and H matrices, we are able to run the next code.

print("V_shape:", image_data.shape)
print("W_shape:", image_data_nmf.shape)
print("H_shape", nmf_model.components_.shape)

To get the reconstruction error or beta divergence between V and the product of WH, we are able to run the next code.

nmf_model.reconstruction_err_

The reconstruction error may be very high. It is because we’ve got chosen only 9 components out of 784. We are able to confirm this by visualizing the output.

image_data_nmf_recovered = nmf_model.inverse_transform(image_data_nmf)
n = 5
plt.figure(figsize=(6.75, 1.5))
for i in range(n):
ax = plt.subplot(1, n, i+1)
plt.imshow(image_data_nmf_recovered[i, :].reshape(28, 28), cmap="binary")
ax.axis('off')
plt.show()

NMF output: 9 components or d = 9

The digits aren’t clear. You may compare this output with the sample of the unique images.

I even have run the NMF algorithm by selecting 100, 225 and 784 components. Listed below are the outcomes.

NMF output: 100 components or d = 100

The reconstruction error is 174524.20.

NMF output: 225 components or d = 225

The reconstruction error is 104024.62.

NMF output: 784 components or d = 784 (all components)

The reconstruction error is 23349.67.

When the variety of components is increased when running non-negative matrix factorization (NMF), the photographs are getting clear and the reconstruction error is getting low.

By taking a look at the output and reconstruction error, a right value for d will be chosen. For this, you should run the NMF algorithm a couple of times which could also be time-consuming depending on the pc resources you will have.

With d = 784 (all components), you’ll still get a reconstruction error of 23349.67 as an alternative of zero.

It is clear that the product of W and H matrices only gives a non-negative matrix approximation to the feature matrix, V.

Can we run NMF with a negative matrix?

The reply is no. In the event you try to make use of NMF with a feature matrix that has negative values, you’ll get the next ValueError!

import numpy as np
from sklearn.decomposition import NMF
V = np.array([[1, 1, -2, 1], [2, 1, -3, 2], [3, 1.2, -3.3, 5]])
nmf_model = NMF(n_components=2, init='random', random_state=0)
W = nmf_model.fit_transform(V)
print("V_shape:", V.shape)
print("W_shape:", W.shape)
print("Reconstruction error:", nmf_model.reconstruction_err_)

You may’t break the non-negativity constraint when running non-negative matrix factorization (NMF). The feature matrix should all the time contain non-negative elements.