automobile stops suddenly. Worryingly, there isn’t a stop check in sight. The engineers can only make guesses as to why the automobile’s neural network became confused. It might be a tumbleweed rolling across the road, a automobile coming down the opposite lane or the red billboard within the background. To seek out the actual reason, they turn to Grad-CAM [1].
Grad-CAM is an explainable AI (XAI) technique that helps reveal a convolutional neural network (CNN) made a selected decision. The strategy produces a heatmap that highlights the regions in a picture which can be an important for a prediction. For our self-driving automobile example, this might show if the pixels from the weed, automobile or billboard caused the automobile to stop.
Now, Grad-CAM is one among many XAI methods for Computer Vision. Attributable to its speed, flexibility and reliability, it has quickly grow to be one of the popular. It has also inspired many related methods. So, if you happen to are eager about XAI, it’s price understanding exactly how this method works. To try this, we will probably be implementing Grad-CAM from scratch using Python.
Specifically, we will probably be counting on PyTorch Hooks. As you will notice, these allow us to dynamically extract gradients and activations from a network during forward and backwards passes. These are practical skills that is not going to only assist you to implement Grad-CAM but in addition any gradient-based XAI method. See the total project on GitHub.
The idea behind Grad-CAM
Before we get to the code, it’s price touching on the speculation behind Grad-CAM. When you desire a deep dive, then take a look at the video below. If you need to study other methods, then see this free XAI for Computer Vision course.
To summarise, when creating Grad-CAM heatmaps, we start with a trained CNN. We then do a forward go through this network with a single sample image. This can activate all convolutional layers within the network. We call these feature maps ($A^k$). They will probably be a set of 2D matrices that contain different features detected within the sample image.
With Grad-CAM, we’re typically eager about the maps from the last convolutional layer of the network. Once we apply the tactic to VGG16, you will notice that its final layer has 512 feature maps. We use these as they contain features with essentially the most detailed semantic information while still retaining spatial information. In other words, they tell us what was used for a prediction and where within the image it was taken from.
The issue is that these maps also contain features which can be vital for other classes. To mitigate this, we follow the method shown in Figure 1. Once we’ve got the feature maps ($A^k$), we weight them by how vital they’re to the category of interest ($y_c$). We do that using $a_k^c$ — the typical gradient of the rating for $y_c$ w.r.t. to the weather within the feature map. We then do element-wise summation. For VGG16, you will notice we go from 512 maps of 14×14 pixels to a single 14×14 map.
The gradients for a person element ($frac{partial y^c}{partial A_{ij}^k}$) tell us how much the rating will change with a small change within the element. Which means that large average gradients indicate that the whole feature map was vital and may contribute more to the ultimate heatmap. So, once we weight and sum the maps, those that contain features for other classes will likely contribute less.
The ultimate steps are to use the ReLU activation function to make sure all negative elements can have a price of zero. Then we upsample with interpolation so the heatmap has the identical dimensions because the sample image. The ultimate map is summarised by the formula below. You would possibly recognise it from the Grad-CAM paper [1].
$$ L_{Grad-CAM}^c = ReLUleft( sum_{k} a_k^c A^k right) $$
Grad-CAM from Scratch
Don’t worry if the speculation isn’t completely clear. We’ll walk through it step-by-step as we apply the tactic from scratch. You will discover the total project on GitHub. To begin, we’ve got our imports below. These are all common imports for computer vision problems.
import matplotlib.pyplot as plt
import numpy as np
import cv2
from PIL import Image
import torch
import torch.nn.functional as F
from torchvision import models, transforms
import urllib.requestLoad pretrained model from PyTorch
We’ll be applying Grad-CAM to VGG16 pretrained on ImageNet. To assist, we’ve got the 2 functions below. The primary will format a picture in the right way for input into the model. The normalisation values used are the mean and standard deviation of the pictures in ImageNet. The 224×224 size can also be standard for ImageNet models.
def preprocess_image(img_path):
"""Load and preprocess images for PyTorch models."""
img = Image.open(img_path).convert("RGB")
#Transforms utilized by imagenet models
transform = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
return transform(img).unsqueeze(0)ImageNet has many classes. The second function will format the output of the model so we display the classes with the very best predicted probabilities.
def display_output(output,n=5):
"""Display the highest n categories predicted by the model."""
# Download the categories
url = "https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt"
urllib.request.urlretrieve(url, "imagenet_classes.txt")
with open("imagenet_classes.txt", "r") as f:
categories = [s.strip() for s in f.readlines()]
# Show top categories per image
probabilities = torch.nn.functional.softmax(output[0], dim=0)
top_prob, top_catid = torch.topk(probabilities, n)
for i in range(top_prob.size(0)):
print(categories[top_catid[i]], top_prob[i].item())
return top_catid[0]We now load the pretrained VGG16 model (line 2), move it to a GPU (lines 5-8) and set it to evaluation mode (line 11). You’ll be able to see a snippet of the model output in Figure 2. VGG16 is fabricated from 16 weighted layers. Here, you possibly can see the last 2 of 13 convolutional layers and the three fully connected layers.
# Load the pre-trained model (e.g., VGG16)
model = models.vgg16(pretrained=True)
# Set the model to gpu
device = torch.device('mps' if torch.backends.mps.is_built()
else 'cuda' if torch.cuda.is_available()
else 'cpu')
model.to(device)
# Set the model to evaluation mode
model.eval()The names you see in Figure 2 are vital. Later, we’ll use them to reference a selected layer within the network to access its activations and gradients. Specifically, we’ll use . That is the ultimate convolutional layer within the network. As you possibly can see within the snapshot, this layer accommodates 512 feature maps.
Forward pass with sample image
We will probably be explaining a prediction from this model. To do that, we want a sample image that will probably be fed into the model. We downloaded one from Wikipedia Commons (lines 2-3). We then load it (lines 5-6), crop it to have equal height and width (line 7) and display it (lines 9-10). In Figure 3, you possibly can see we’re using a picture of a whale shark in an aquarium.
# Load a sample image from the online
img_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a1/Male_whale_shark_at_Georgia_Aquarium.jpg/960px-Male_whale_shark_at_Georgia_Aquarium.jpg"
urllib.request.urlretrieve(img_url, "sample_image.jpg")[0]
img_path = "sample_image.jpg"
img = Image.open(img_path).convert("RGB")
img = img.crop((320, 0, 960, 640)) # Crop to 640x640
plt.imshow(img)
plt.axis("off")
ImageNet has no dedicated class for whale sharks, so it is going to be interesting to see what the model predicts. To do that, we start by processing our image (line 2) and moving it to the GPU (line 3). We then do a forward pass to get a prediction (line 6) and display the highest 5 probabilities (line 7). You’ll be able to see these in Figure 4.
# Preprocess the image
img_tensor = preprocess_image(img_path)
img_tensor = img_tensor.to(device)
# Forward pass
predictions = model(img_tensor)
display_output(predictions,n=5)Given the available classes, these seem reasonable. They’re all marine life and the highest two are sharks. Now, let’s see how we are able to explain this prediction. We wish to grasp what regions of the image contribute essentially the most to the very best predicted class — hammerhead.
PyTorch hooks naming conventions
Grad-CAM heatmaps are created using each activations from a forward pass and gradients from a backwards pass. To access these, we’ll use PyTorch hooks. These are functions that assist you to save the inputs and outputs of a layer. We won’t do it here, but they even assist you to alter these features. For instance, Guided Backpropagation might be applied by ensuring only positive gradients are propagated using a backwards hook.
You’ll be able to see some examples of those functions below. A will probably be called during a forward pass. It should be registered on a given module (i.e. layer). By default, the function receives three arguments — the module, its input and its output. Similarly, a is triggered during a backwards pass with the module and gradients of the input and output.
# Example of a forwards hook function
def fowards_hook(module, input, output):
"""Parameters:
module (nn.Module): The module where the hook is applied.
input (tuple of Tensors): Input to the module.
output (Tensor): Output of the module."""
...
# Example of a backwards hook function
def backwards_hook(module, grad_in, grad_out):
"""Parameters:
module (nn.Module): The module where the hook is applied.
grad_in (tuple of Tensors): Gradients w.r.t. the input of the module.
grad_out (tuple of Tensors): Gradients w.r.t. the output of the module."""
...To avoid confusion, let’s make clear the parameter names utilized by these functions. Take a have a look at the overview of the usual backpropagation procedure for a convolutional layer in Figure 5. This layer consists of a set of kernels, $K$, and biases, $b$. The opposite parts are the:
- – a set of feature maps or a picture
- – set of feature maps
- is the gradient of the loss w.r.t. the layer’s input.
- is the gradient of the loss w.r.t. the layer’s output.
We have now labelled these using the identical names of the arguments used to call the hook functions that we apply later.
Bear in mind, we won’t use the gradients in the identical way as backpropagation. Normally, we use the gradients of a of images to update $K$ and $b$. Now, we’re only eager about of a sample image. This can give us the gradients of the weather within the layer’s feature maps. In other words, the gradients we use to weight the feature maps.
Activations with PyTorch forward hook
Our VGG16 network has been created using ReLU with These modify tensors in memory, so the unique values are lost. That’s, tensors used as input are overwritten by the ReLU function. This could result in problems when applying hooks, as we might have the unique input. So we use the code below to switch all ReLU functions with ones. This can not impact the output of the model, but it is going to increase its memory usage.
# Replace all in-place ReLU activations with out-of-place ones
def replace_relu(model):
for name, child in model.named_children():
if isinstance(child, torch.nn.ReLU):
setattr(model, name, torch.nn.ReLU(inplace=False))
print(f"Replacing ReLU activation in layer: {name}")
else:
replace_relu(child) # Recursively apply to submodules
# Apply the modification to the VGG16 model
replace_relu(model)Below we’ve got our first hook function — . This can append the output from a module (line 6) to an inventory of activations (line 2). In our case, we’ll only register the hook onto one module (i.e. the last convolutional layer), so this list will only contain one element. Notice how we format the output (line 6). We detach it from the computational graph so the network isn’t affected. We also format them as a numpy array and squeeze the batch dimension.
# List to store activations
activations = []
# Function to avoid wasting activations
def save_activations(module, input, output):
activations.append(output.detach().cpu().numpy().squeeze())To make use of the hook function, we register it on the last convolutional layer — This is finished using the function.
# Register the hook to the last convolutional layer
hook = model.features[28].register_forward_hook(save_activations)Now, once we do a forward pass (line 2), the hook function will probably be called for this layer. In other words, its output will probably be saved to the list.
# Forward go through the model to get activations
prediction = model(img_tensor)
Finally, it is nice practice to remove the hook function when it is not any longer needed (line 2). This implies the forward hook function is not going to be triggered if we do one other forward pass.
# Remove the hook after use
hook.remove() The form of those activations is (512, 14, 14). In other words, we’ve got 512 feature maps and every map is 14×14 pixels. You’ll be able to see some examples of those in Figure 6. A few of these maps may contain features vital for other classes or people who decrease the probability of the expected class. So let’s see how we are able to find gradients to assist discover an important maps.
act_shape = np.shape(activations[0])
print(f"Shape of activations: {act_shape}") # (512, 14, 14)
Gradients with PyTorch backwards hooks
To get gradients, we follow the same process to before. The important thing difference is that we now use the to register the function (line 7). This can be sure that it is known as during a backwards pass. Importantly, we do the backwards pass (line 16) from the output for the category with the very best rating (line 13). This effectively sets the rating for this class to 1 and all other scores to 0. In other words, we get the gradients of the hammerhead class w.r.t. to the weather of the feature maps.
gradients = []
def save_gradient(module, grad_in, grad_out):
gradients.append(grad_out[0].cpu().numpy().squeeze())
# Register the backward hook on a convolutional layer
hook = model.features[28].register_full_backward_hook(save_gradient)
# Forward pass
output = model(img_tensor)
# Pick the category with highest rating
rating = output[0].max()
# Backward pass from the rating
rating.backward()
# Remove the hook after use
hook.remove()We can have a gradient for each element of the feature maps. So, again, the form is (512, 14, 14). Figure 7 visualises a few of these. You’ll be able to see some are likely to have higher values. Nevertheless, we usually are not so concerned with the person gradients. Once we create a Grad-CAM heatmap, we’ll use the typical gradient of every feature map.
grad_shape = np.shape(gradients[0])
print(f"Shape of gradients: {grad_shape}") # (512, 14, 14)
Finally, before we move on, it is nice practice to reset the model’s gradients (line 2). This is especially vital if you happen to plan to run the code for multiple images, as gradients might be amassed with each backwards pass.
# Reset gradients
model.zero_grad() Creating Grad-CAM heatmaps
First, we discover the mean gradients for every feature map. There will probably be 512 of those average gradients. Plotting a histogram of them, you possibly can see most are likely to be around 0. In other words, these don’t have much impact on the expected rating. There are a number of that are likely to have a negative impact and a positive impact. It’s these feature maps we wish to provide more weight to.
# Step 1: aggregate the gradients
gradients_aggregated = np.mean(gradients[0], axis=(1, 2))
We mix all of the activations by doing element-wise summation (lines 2-4). Once we do that, we weight each feature map by its average gradient (line 3). In the long run, we can have one 14×14 array.
# Step 2: weight the activations by the aggregated gradients and sum them up
weighted_activations = np.sum(activations[0] *
gradients_aggregated[:, np.newaxis, np.newaxis],
axis=0)These weighted activations will contain each positive and negative pixels. We are able to consider the negative pixels to be suppressing the expected rating. In other words, a rise in the worth of those regions tends to diminish the rating. Since we’re only eager about the positive contributions—regions that support the category prediction—we apply a ReLU activation to the ultimate heatmap (line 2). You’ll be able to see the difference within the heatmaps in Figure 9.
# Step 3: ReLU summed activations
relu_weighted_activations = np.maximum(weighted_activations, 0)
You’ll be able to see the heatmap in Figure 9 is sort of coarse. It could be more useful if it had the scale of the unique image. Because of this the last step for creating Grad-CAM heatmaps is to upsample to the dimension of the input image (lines 2-4). On this case, we’ve got a 224×224 image.
#Step 4: Upsample the heatmap to the unique image size
upsampled_heatmap = cv2.resize(relu_weighted_activations,
(img_tensor.size(3), img_tensor.size(2)),
interpolation=cv2.INTER_LINEAR)
print(np.shape(upsampled_heatmap)) # Must be (224, 224)Figure 10 gives us our final visualisation. We display the sample image (lines 5-7) next to the heatmap (lines 10-15). For the latter, we create a transparent visualisation with the assistance of Canny Edge detection (line 10). This offers us an edge map (i.e. outline) of the sample image. We are able to then overlay the heatmap on top of this (line 14).
# Step 5: visualise the heatmap
fig, ax = plt.subplots(1, 2, figsize=(8, 8))
# Input image
resized_img = img.resize((224, 224))
ax[0].imshow(resized_img)
ax[0].axis("off")
# Edge map for the input image
edge_img = cv2.Canny(np.array(resized_img), 100, 200)
ax[1].imshow(255-edge_img, alpha=0.5, cmap='gray')
# Overlay the heatmap
ax[1].imshow(upsampled_heatmap, alpha=0.5, cmap='coolwarm')
ax[1].axis("off")our Grad-CAM heatmap, there may be some noise. Nevertheless, it appears the model is counting on the tail fin and, to a lesser extent, the pectoral fin to make its predictions. It’s beginning to make sense why the model classified this shark as a hammerhead. Perhaps each animals share these characteristics.
For some further investigation, we apply the identical process but now using an actual image of a hammerhead. On this case, the model appears to be counting on the identical features. This can be a bit concerning. Would we not expect the model to make use of one among the shark’s defining features— the hammerhead? Ultimately, this may increasingly lead VGG16 to confuse various kinds of sharks.
With this instance, we see how Grad-CAM can highlight potential flaws in our model. We are able to not only get their predictions but in addition understand how they made them. We are able to understand if the features used will result in unexpected predictions down the road. This could potentially save us lots of time, money and within the case of more consequential applications, lives!
If you need to learn more about XAI for CV take a look at one among these articles. Or see this Free XAI for CV course.
I hope you enjoyed this text! See the course page for more XAI courses. You may as well find me on Bluesky | Threads | YouTube | Medium
References
[1] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
