Part 2 of the tutorial on the best way to run and evaluate experiments without leaving your IDE

Within the previous article of this series, I demonstrated the best way to use DVC’s VS Code extension to rework our IDE into an experimentation platform, allowing us to directly run and evaluate ML experiments. I also mentioned that the extension offers useful plotting functionalities, which enable us to visualise and evaluate the performance of our experiments using interactive plots. To make it even higher, the extension also offers live plotting of certain metrics through the training phase. You’ll be able to get a sneak peek of this feature in the next figure.

This text will reveal the best way to enhance the previously-introduced experimentation workflow by monitoring model performance and evaluating experiments with interactive plots, all inside VS Code. To attain this, we’ll tackle a binary image classification problem. First, we’ll provide an outline of transfer learning in computer vision and share some details in regards to the chosen dataset.

Image classification is one of the crucial popular tasks in the sphere of computer vision. For our example, we’ll use the cat vs dog classification problem, which has been widely utilized in the research community to benchmark different deep learning models. As you may have guessed, the goal of the project is to categorise an input image as either a cat or a dog.

To attain high accuracy even with limited training data, we’ll leverage transfer learning to hurry up the training process. Transfer learning is a robust deep learning technique that has recently gained significant popularity, especially in various domains of computer vision. With the vast amount of information available on the web, transfer learning allows us to leverage existing knowledge from one domain/problem and apply it to a unique one.

Considered one of the approaches to using transfer learning for computer vision relies on the thought of feature extraction. First, a model is trained on a big and general dataset (for instance, the ImageNet dataset). This model serves as a generic model of “vision”. Then, we are able to use the learned feature maps of such a model without having to begin the training of a custom network from scratch

For our use case, we’ll utilize a pre-trained model (ResNet50) to extract relevant features for our binary classification problem. The approach consists of a number of steps:

1. Obtain a pre-trained model, i.e., a saved network that was previously trained on a big dataset. You’ll find some examples here.
2. Use the feature maps learned by the chosen network to extract meaningful features from images that the network was not trained on.
3. Add a latest classifier on top of the pre-trained network. The classifier might be trained from scratch for the reason that classification component of the pre-trained model is particular to its original task.

We’ll show the best way to do all of this in the next sections. Nonetheless, please keep in mind that this shouldn’t be a tutorial on transfer learning. For those who would really like to learn more in regards to the theory and implementation, please consult with this text or this tutorial.

By utilizing the next snippet, we are able to download the cats vs. dogs dataset. The original dataset contained 12500 images of every class. Nonetheless, for our project, we might be using a smaller, filtered dataset that accommodates 1000 training images and 500 validation images per class. The extra advantage of downloading the filtered dataset via TensorFlow is that it doesn’t contain some corrupted images that were present in the unique dataset (please see here for more information).

import os
import tensorflow as tf
import shutil
DATA_URL = "https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip"
DATA_PATH = "data/raw"
path_to_zip = tf.keras.utils.get_file(
"cats_and_dogs.zip", origin=DATA_URL, extract=True
)
download_path = os.path.join(os.path.dirname(path_to_zip), "cats_and_dogs_filtered")
train_dir_from = os.path.join(download_path, "train")
validation_dir_from = os.path.join(download_path, "validation")
train_dir_to = os.path.join(DATA_PATH, "train")
validation_dir_to = os.path.join(DATA_PATH, "validation")
shutil.move(train_dir_from, train_dir_to)
shutil.move(validation_dir_from, validation_dir_to)

The next tree presents the structure of the directories containing the downloaded images:

📦data
┗ 📂raw
┣ 📂train
┃ ┣ 📂cats
┃ ┗ 📂dogs
┗ 📂validation
┣ 📂cats
┗ 📂dogs

In case you prefer to to make use of the entire dataset to your experiments, you’ll be able to load it using tensorflow_datasets.

On this section, we’ll show the code used for training and experimenting with our neural network classifier. Specifically, we’ll need the next three files:

• train.py — accommodates the code used for training the neural network.
• params.yaml — accommodates the parameters used for training the neural network, similar to the scale of the input images, batch size, learning rate, variety of epochs, etc.
• dvc.yaml — accommodates the DVC pipeline, which stores details about all of the steps which are executed inside our project, including their respective dependencies and outputs. For a more thorough description of this file and its structure, please consult with my previous article.

As a matter of fact, our current setup is more advanced than the bare minimum. While we could have began with just the training script, we selected to implement a more sophisticated setup right from the beginning. This can allow us to conveniently run experiments in a queue and simply parameterize them, amongst other advantages.

Let’s start with the dvc.yaml file because it accommodates this project’s pipeline. As this can be a relatively easy project, it only has one stage called train. Within the file, we are able to see which script accommodates the stage’s code, what its dependencies are, where the parameters are positioned, and what the outputs are. The outs step accommodates a directory that doesn’t exist yet (dvclive), which might be mechanically created while running our experiments.

stages:
train:
cmd: python src/train.py
deps:
- src/train.py
- data/raw
params:
- train
outs:
- models
- metrics.csv
- dvclive/metrics.json:
cache: False
- dvclive/plots

Let’s proceed to the params.yaml file. We have now already mentioned what it accommodates, so its contents shouldn’t come as a surprise:

train:
image_width: 180
image_height: 180
batch_size: 32
learning_rate: 0.01
n_epochs: 15

Naturally, the file can contain many more parameters for multiple stages of the project, that are defined within the DVC pipeline.

Finally, we proceed to the file used for training the neural network. To make it more readable, we’ll break it down into three code snippets. In the primary one, we execute the next steps:

• Import the needed libraries.
• Define the info directories individually for the training and validation datasets.
• Load the parameters from the params.yaml file.
• Define the training and validation datasets using the image_dataset_from_directory functionality of keras.

import os
from pathlib import Path
import numpy as np
import tensorflow as tf
from dvc.api import params_show
from dvclive.keras import DVCLiveCallback
# data directories
BASE_DIR = Path(__file__).parent.parent
DATA_DIR = "data/raw"
train_dir = os.path.join(DATA_DIR, "train")
validation_dir = os.path.join(DATA_DIR, "validation")
# get the params
params = params_show()["train"]
IMG_WIDTH, IMG_HEIGHT = params["image_width"], params["image_height"]
IMG_SIZE = (IMG_WIDTH, IMG_HEIGHT)
BATCH_SIZE = params["batch_size"]
LR = params["learning_rate"]
N_EPOCHS = params["n_epochs"]
# get image datasets
train_dataset = tf.keras.utils.image_dataset_from_directory(
train_dir, shuffle=True, batch_size=BATCH_SIZE, image_size=IMG_SIZE
)
validation_dataset = tf.keras.utils.image_dataset_from_directory(
validation_dir, shuffle=True, batch_size=BATCH_SIZE, image_size=IMG_SIZE
)

The second a part of the training script accommodates the definition of the neural network architecture that we wish to make use of for this project.

def get_model():
"""
Prepare the ResNet50 model for transfer learning.
"""
data_augmentation = tf.keras.Sequential(
[
tf.keras.layers.RandomFlip("horizontal"),
tf.keras.layers.RandomRotation(0.2),
]
)
preprocess_input = tf.keras.applications.resnet50.preprocess_input
IMG_SHAPE = IMG_SIZE + (3,)
base_model = tf.keras.applications.ResNet50(
input_shape=IMG_SHAPE, include_top=False, weights="imagenet"
)
base_model.trainable = False
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
prediction_layer = tf.keras.layers.Dense(1)
inputs = tf.keras.Input(shape=IMG_SHAPE)
x = data_augmentation(inputs)
x = preprocess_input(x)
x = base_model(x, training=False)
x = global_average_layer(x)
x = tf.keras.layers.Dropout(0.2)(x)
outputs = prediction_layer(x)
model = tf.keras.Model(inputs, outputs)
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=LR),
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=["accuracy"],
)
return model

We won’t dive deeply into the code used for transfer learning, because it is barely outside the scope of this text. Nonetheless, it’s value mentioning that:

• We used some quite simple image augmentation techniques: random horizontal flip and random rotation. These augmentations are only applied to the training set.
• While training the model, we wish to trace its accuracy. We selected this metric because we’re coping with a balanced dataset, but we could easily track additional metrics similar to precision and recall.

The third and last snippet accommodates the major body of our script:

def major():
model_path = BASE_DIR / "models"
model_path.mkdir(parents=True, exist_ok=True)
model = get_model()
callbacks = [
tf.keras.callbacks.ModelCheckpoint(
model_path / "model.keras", monitor="val_accuracy", save_best_only=True
),
tf.keras.callbacks.CSVLogger("metrics.csv"),
DVCLiveCallback(save_dvc_exp=True),
]
history = model.fit(
train_dataset,
epochs=N_EPOCHS,
validation_data=validation_dataset,
callbacks=callbacks,
)
if __name__ == "__main__":
major()

On this snippet, we do the next:

• We create the models directory if it doesn’t exist.
• We get the model using the get_model function defined within the previous snippet.
• We define the callbacks we wish to make use of. The primary two are standard callbacks used while training neural networks. The primary one is used for creating checkpoints while training. The second stores the chosen metrics (in our case, accuracy and loss) after each epoch right into a CSV file. We’ll cover the third callback in a moment.
• We fit the model to the training data and evaluate using the validation set.

The third callback we used, DVCLiveCallback, comes from a companion library called DVCLive. Basically, it’s a library that gives utilities for logging ML parameters, metrics, and other metadata in easy file formats. You’ll be able to consider it as an ML logger just like, for instance, MLFlow. The most important difference is that through the use of DVCLive, we wouldn’t have to make use of any additional services or servers. All the logged metrics and metadata are stored as plain text files, which will be versioned with Git.

On this particular case, we used a Keras-compatible callback provided by DVCLive. DVCLive provides similar utilities for the preferred machine and deep learning libraries, similar to TensorFlow, PyTorch, LightGBM, XGBoost, and more. You’ll find the entire list of supported libraries here. Additionally it is value mentioning that regardless that DVCLive provides many useful callbacks that we are able to use out-of-the-box, it doesn’t mean that is the one option to log the metrics. We will manually log whichever metrics/plots we wish at any point we wish.

After we specified the DVCLiveCallback, we set the save_dvc_exp argument to True. By doing so, we indicated that we would really like to mechanically track the outcomes using Git.

At this point, we’re able to run our first experiment. For that, we’ll use the parameters we’ve got initially laid out in the params.yaml file. To run the experiment, we are able to either press the Run Experiment button within the Experiments tab of the DVC panel or use the next command within the terminal:

dvc exp run

For more information on running the experiments and navigating the Experiments tab, please consult with my previous article.

After running the experiment, we notice that a latest directory was created —dvclive. The DVCLive callback we utilized in our code mechanically logged data and stored it in plain text files in that directory. In our case, the directory looks like this:

📦dvclive
┣ 📂plots
┃ ┗ 📂metrics
┃ ┃ ┣ 📂eval
┃ ┃ ┃ ┣ 📜accuracy.tsv
┃ ┃ ┃ ┗ 📜loss.tsv
┃ ┃ ┗ 📂train
┃ ┃ ┃ ┣ 📜accuracy.tsv
┃ ┃ ┃ ┗ 📜loss.tsv
┣ 📜.gitignore
┣ 📜dvc.yaml
┣ 📜metrics.json
┗ 📜report.html

We offer a temporary description of the generated files:

• The TSV files contain the accuracy and loss over epochs, individually for the training and validation datasets.
• metrics.json accommodates the requested metrics for the ultimate epoch.
• report.html accommodates plots of the tracked metrics in a type of an HTML report.

At this point, we are able to inspect the tracked metrics within the HTML report. Nonetheless, we may try this directly from VS Code by navigating to the Plots tab within the DVC extension.

Using the left-hand sidebar, we are able to select the experiments we wish to visualise. I actually have chosen the major one, but you’ll be able to see that I actually have already run a number of experiments before. Within the Plots menu, we are able to select which metrics we wish to plot. This functionality may be very handy after we track plenty of metrics, but we only wish to inspect a number of of them at a time.

Within the major view, we are able to see the visualized metrics. The upper plots present the metrics calculated using the validation set, while the lower ones are based on the training set. What you can’t see within the static image is that those plots are live plots. It signifies that the metrics are updated after each epoch of coaching is accomplished. We will use this tab to watch the progress of our training jobs in real-time.

For the second experiment, we increase the training rate from 0.01 to 0.1. We will run such an experiment using the next command:

dvc exp run -S train.learning_rate=0.1

To watch the model during training, we also chosen the workspace experiment within the Experiments menu. Within the image below, you’ll be able to see what the plots appear to be while the neural network remains to be within the training stage (you’ll be able to see that the method is running within the terminal window).

To this point, all of our plots were generated within the Data Series section of the Plots tab. In total, there are three sections, each with different sorts of plots:

• Data Series — accommodates visualizations of metrics stored in text files (JSON, YAML, CSV, or TSV).
• Images — accommodates side-by-side visualizations of stored images, similar to JPG files.
• Trends — accommodates mechanically generated and updated scalar metrics per epoch if DVC checkpoints are enabled.

We have now already explored the best way to track and visualize metrics using DVCLive’s callbacks. Using DVC also allows us to trace plots stored as images. For example, we could create a bar chart representing the feature importance obtained from a certain model. Or, to simplify, we could track a confusion matrix.

The final approach to trace and visualize custom plots using DVC is to create the plot manually, put it aside as a picture, after which track it. This enables us to trace any custom plot we create. Alternatively, for certain scikit-learn plots, we are able to use DVCLive’s log_sklearn_plot method and generate the plot using data (predictions vs. ground truth) stored in JSON files. This approach currently works for the next sorts of plots: probability calibration, confusion matrix, ROC curve, and precision-recall curve.

For this instance, we’ll reveal the best way to start tracking a confusion matrix. Within the code snippet below, you’ll be able to see the modified train.py script. We have now removed many things that didn’t change, making it easier to follow the modifications.

import os
from pathlib import Path
import numpy as np
import tensorflow as tf
from dvc.api import params_show
from dvclive.keras import DVCLiveCallback
from dvclive import Live
# data directories, parameters, datasets, and the model function didn't change
def major():
model_path = BASE_DIR / "models"
model_path.mkdir(parents=True, exist_ok=True)
model = get_model()
with Live(save_dvc_exp=True) as live:
callbacks = [
tf.keras.callbacks.ModelCheckpoint(
model_path / "model.keras", monitor="val_accuracy", save_best_only=True
),
tf.keras.callbacks.CSVLogger("metrics.csv"),
DVCLiveCallback(live=live),
]
history = model.fit(
train_dataset,
epochs=N_EPOCHS,
validation_data=validation_dataset,
callbacks=callbacks,
)
model.load_weights(str(model_path / "model.keras"))
y_pred = np.array([])
y_true = np.array([])
for x, y in validation_dataset:
y_pred = np.concatenate([y_pred, model.predict(x).flatten()])
y_true = np.concatenate([y_true, y.numpy()])
y_pred = np.where(y_pred > 0, 1, 0)
live.log_sklearn_plot("confusion_matrix", y_true, y_pred)
if __name__ == "__main__":
major()

As you’ll be able to see, this time we created an instance of a Live object, which we use each for the callback and the log_sklearn_plot method. To trace all of the metrics, we used a context manager (the with statement) to instantiate the Live instance. Without doing so, DVCLive would create an experiment when keras calls on_train_end. Consequently, any data logged after that (in our case, the confusion matrix plot) wouldn’t be tracked throughout the experiment.

After modifying the training script, we ran again the 2 experiments with different learning rates (0.1 vs. 0.01). Consequently, we are able to now see the confusion matrices within the Plots tab, right under the previously explored plots.

The final thing to say is that running the modified training script also modifies the dvc.yaml pipeline throughout the dvclive directory. As you’ll be able to see below, it now accommodates information in regards to the tracked confusion matrix, similar to the best way to construct it, which template to make use of, and what labels to make use of.

metrics:
- metrics.json
plots:
- plots/metrics
- plots/sklearn/confusion_matrix.json:
template: confusion
x: actual
y: predicted
title: Confusion Matrix
x_label: True Label
y_label: Predicted Label

Within the previous article of the series, we showed the best way to start using DVC and the dedicated VS Code extension to show your IDE into an ML experimentation platform. On this part, we continued where we left off and we explored various (live-) plotting capabilities of the extension. Using those, we are able to easily evaluate and compare experiments to decide on the perfect one.

In my view, there are two significant benefits of using a DVC-enhanced workflow. First, we don’t need any external services or setups to begin our experiments. The one requirement is a Git repo. Moreover, DVC works with Git in a clean way. Although every experiment is saved in a Git commit, those commits are hidden and don’t clutter our repository. The truth is, we don’t even have to create separate branches.

Secondly, all the things happens inside our IDE, enabling us to concentrate on our project without continually switching between the IDE, browser, and other tools. This fashion, we are able to avoid distractions and the ever-threatening context-switching.

As at all times, any constructive feedback is greater than welcome. You’ll be able to reach out to me on Twitter or within the comments. You’ll find all of the code used for this text in this repository.

Liked the article? Turn into a Medium member to proceed learning by reading without limits. For those who use this link to change into a member, you’ll support me at no extra cost to you. Thanks prematurely and see you around!

You may additionally be focused on certainly one of the next: