In my previous article, I discussed how morphological feature extractors mimic the best way biological experts visually assess images.
time, I need to go a step further and explore a brand new query:
Can different architectures complement one another to construct an AI that “sees” like an authority?
Introduction: Rethinking Model Architecture Design
While constructing a high accuracy visual recognition model, I bumped into a key challenge:
How will we get AI to not only “see” a picture, but actually understand the features that matter?
Traditional CNNs excel at capturing local details like fur texture or ear shape, but they often miss the larger picture. Transformers, however, are great at modeling global relationships, how different regions of a picture interact, but they’ll easily overlook fine-grained cues.
This insight led me to explore combining the strengths of each architectures to create a model that not only captures tremendous details but additionally comprehends the larger picture.
While developing PawMatchAI, a 124-breed dog classification system, I went through three major architectural phases:
1. Early Stage: EfficientNetV2-M + Multi-Head Attention
I began with EfficientNetV2-M and added a multi-head attention module.
I experimented with 4, 8, and 16 heads—eventually deciding on 8, which gave the perfect results.
This setup reached an F1 rating of 78%, however it felt more like a technical combination than a cohesive design.
2. Refinement: Focal Loss + Advanced Data Augmentation
After closely analyzing the dataset, I noticed a category imbalance, some breeds appeared way more incessantly than others, skewing the model’s predictions.
To handle this, I introduced Focal Loss, together with RandAug and mixup, to make the info distribution more balanced and diverse.
This pushed the F1 rating as much as 82.3%.
3. Breakthrough: Switching to ConvNextV2-Base + Training Optimization
Next, I replaced the backbone with ConvNextV2-Base, and optimized the training using OneCycleLR and a progressive unfreezing strategy.
The F1 rating climbed to 87.89%.
But during real-world testing, the model still struggled with visually similar breeds, indicating room for improvement in generalization.
4. Final Step: Constructing a Truly Hybrid Architecture
After reviewing the primary three phases, I noticed the core issue: stacking technologies isn’t the identical as getting them to work together.
What I needed was true collaboration between the CNN, the Transformer, and the morphological feature extractor, each playing to its strengths. So I restructured the whole pipeline.
ConvNextV2 was in control of extracting detailed local features.
The morphological module acted like a site expert, highlighting features critical for breed identification.
Finally, the multi-head attention brought all of it together by modeling global relationships.
This time, they weren’t just independent modules, they were a team.
CNNs identified the small print, the morphology module amplified the meaningful ones, and the eye mechanism tied all the pieces right into a coherent global view.
Key Result: The F1 rating rose to 88.70%, but more importantly, this gain got here from the model learning to understand morphology, not only memorize textures or colours.
It began recognizing subtle structural features—similar to an actual expert would—making higher generalizations across visually similar breeds.
💡 Should you’re interested, I’ve written more about morphological feature extractors here.
These extractors mimic how biological experts assess shape and structure, enhancing critical visual cues like ear shape and body proportions.
They’re an important a part of this hybrid design, filling the gaps traditional models are inclined to overlook.
In this text, I’ll walk through:
- The strengths and limitations of CNNs vs. Transformers—and the way they’ll complement one another
- Why I ultimately selected ConvNextV2 over EfficientNetV2
- The technical details of multi-head attention and the way I made a decision the variety of heads
- How all these elements got here together in a unified hybrid architecture
- And eventually, how heatmaps reveal that the AI is learning to “see” key features, similar to a human expert
1. The Strengths and Limitations of CNNs and Transformers
Within the previous section, I discussed how CNNs and Transformers can effectively complement one another. Now, let’s take a better have a look at what sets each architecture apart, their individual strengths, limitations, and the way their differences make them work so well together.
1.1 The Strength of CNNs: Great with Details, Limited in Scope
CNNs are like meticulous artists, they’ll draw tremendous lines beautifully, but often miss the larger composition.
✅ Strong at Local Feature Extraction
CNNs are excellent at capturing edges, textures, and shapes—ideal for distinguishing fine-grained features like ear shapes, nose proportions, and fur patterns across dog breeds.
✅ Computational Efficiency
With parameter sharing, CNNs process high-resolution images more efficiently, making them well-suited for large-scale visual tasks.
✅ Translation Invariance
Even when a dog’s pose varies, CNNs can still reliably discover its breed.
That said, CNNs have two key limitations:
⚠️ Limited Receptive Field:
CNNs expand their field of view layer by layer, but early-stage neurons only “see” small patches of pixels. Consequently, it’s difficult for them to attach features which might be spatially far apart.
🔹
⚠️ Lack of Global Feature Integration:
CNNs excel at local stacking of features, but they’re less adept at combining information from distant regions.
🔹
1.2 The Strength of Transformers: Global Awareness, But Less Precise
Transformers are like master strategists with a bird’s-eye view, they quickly spot patterns, but aren’t great at filling within the tremendous details.
✅ Capturing Global Context
Because of their self-attention mechanism, Transformers can directly link any two features in a picture, regardless of how far apart they’re.
✅ Dynamic Attention Weighting
Unlike CNNs’ fixed kernels, Transformers dynamically allocate focus based on context.
🔹
But Transformers even have two major drawbacks:
⚠️ High Computational Cost:
Self-attention has a time complexity of O(n²). As image resolution increases, so does the fee—making training more intensive.
⚠️ Weak at Capturing Superb Details:
Transformers lack CNNs’ “built-in intuition” that nearby pixels are often related.
🔹
1.3 Why a Hybrid Architecture Is Mandatory
Let’s take an actual world case:
How do you distinguish a Golden Retriever from a Labrador Retriever?
They’re each beloved family dogs with similar size and temperament. But experts can easily tell them apart by observing:
- Golden Retrievers have long, dense coats starting from golden to dark gold, more elongated heads, and distinct feathering around ears, legs, and tails.
- Labradors, however, have short, double-layered coats, more compact bodies, rounder heads, and thick otter-like tails. Their coats are available yellow, chocolate, or black.
Interestingly, for humans, this distinction is comparatively easy, “long hair vs. short hair” could be all you would like.
But for AI, relying solely on coat length (a texture-based feature) is commonly unreliable. Lighting, image quality, or perhaps a trimmed Golden Retriever can confuse the model.
When analyzing this challenge, we will see…
The issue with using only CNNs:
- While CNNs can detect individual features like “coat length” or “tail shape,” they struggle with mixtures like “head shape + fur type + body structure.” This issue worsens when the dog is in a distinct pose.
The issue with using only Transformers:
- Transformers can associate features across the image, but they’re not great at picking up fine-grained cues like slight variations in fur texture or subtle head contours. In addition they require large datasets to realize expert-level performance.
- Plus, their computational cost increases sharply with image resolution, slowing down training.
These limitations highlight a core truth:
Superb-grained visual recognition requires each local detail extraction and global relationship modeling.
A very expert system like a veterinarian or show judge must inspect features up close while understanding the general structure. That’s exactly where hybrid architectures shine.
1.4 The Benefits of a Hybrid Architecture
This is the reason we want hybrid systems architectures that mix CNNs’ precision in local features with Transformers’ ability to model global relationships:
- CNNs: Extract local, fine-grained features like fur texture and ear shape, crucial for spotting subtle differences.
- Transformers: Capture long-range dependencies (e.g., head shape + body size + eye color), allowing the model to reason holistically.
- Morphological Feature Extractors: Mimic human expert judgment by emphasizing diagnostic features, bridging the gap left by data-driven models.
Such an architecture not only boosts evaluation metrics just like the F1 Rating, but more importantly, it enables the AI to genuinely understand the subtle distinctions between breeds, getting closer to the best way human experts think. The model learns to weigh multiple features together, as a substitute of over-relying on one or two unstable cues.
In the subsequent section, I’ll dive into how I actually built this hybrid architecture, especially how I chosen and integrated the fitting components.
2. Why I Selected ConvNextV2: Key Innovations Behind the Backbone
Amongst the numerous visual recognition architectures available, why did I select ConvNextV2 because the backbone of my project?
Because its design effectively combines the perfect of each worlds: the CNN’s ability to extract precise local features, and the Transformer’s strength in capturing long-range dependencies.
Let’s break down three core innovations that made it the fitting fit.
2.1 FCMAE Self-Supervised Learning: Adaptive Learning Inspired by the Human Brain
Imagine learning to navigate along with your eyes covered, your brain becomes laser-focused on memorizing the small print you’ll be able to perceive.
ConvNextV2 uses a self-supervised pretraining strategy just like that of Vision Transformers.
During training, as much as 60% of input pixels are intentionally masked, and the model must learn to reconstruct the missing regions.
This “make learning harder on purpose” approach actually leads to a few major advantages:
- Comprehensive Feature Learning
The model learns the underlying structure and patterns of a picture—not only probably the most obvious visual cues.
Within the context of breed classification, this implies it pays attention to fur texture, skeletal structure, and body proportions, as a substitute of relying solely on color or shape. - Reduced Dependence on Labeled Data
By pretraining on unlabeled dog images, the model develops strong visual representations.
Later, with only a small amount of labeled data, it may possibly fine-tune effectively—saving significant annotation effort. - Improved Recognition of Rare Patterns
The reconstruction task pushes the model to learn generalized visual rules, enhancing its ability to discover rare or underrepresented breeds.
2.2 GRN Global Calibration: Mimicking an Expert’s Attention
Like a seasoned photographer who adjusts the exposure of every element to spotlight what truly matters.
GRN (Global Response Normalization) is arguably probably the most impactful innovation in ConvNextV2, giving CNNs a level of global awareness that was previously lacking:
- Dynamic Feature Recalibration
GRN globally normalizes the feature map, amplifying probably the most discriminative signals while suppressing irrelevant ones.
For example, when identifying a German Shepherd, it emphasizes upright ears and the sloped back while minimizing background noise. - Enhanced Sensitivity to Subtle Differences
This normalization sharpens feature contrast, making it easier to identify fine-grained differences—critical for telling apart breeds just like the Siberian Husky and Alaskan Malamute. - Concentrate on Diagnostic Features
GRN helps the model prioritize features that actually matter for classification, slightly than counting on statistically correlated but causally irrelevant cues.
2.3 Sparse and Efficient Convolutions: More with Less
Like a streamlined team where each member plays to their strengths, reducing redundancy while boosting performance.
ConvNextV2 incorporates architectural optimizations corresponding to depthwise separable convolutions and sparse connections, leading to three major gains:
- Improved Computational Efficiency
By breaking down convolutions into smaller, more efficient steps, the model reduces its computational load.
This permits it to process high-resolution dog images and detect tremendous visual differences without requiring excessive resources. - Expanded Effective Receptive Field
The layout of convolutions is designed to increase the model’s field of view, helping it analyze each overall body structure and local details concurrently. - Parameter Efficiency
The architecture ensures that each parameter carries more learning capability, extracting richer, more nuanced information using the identical amount of compute.
2.4 Why ConvNextV2 Was the Right Fit for a Hybrid Architecture
ConvNextV2 turned out to be the perfect backbone for this hybrid system, not simply because of its performance, but since it embodies the very philosophy of fusion.
It retains the local precision of CNNs while adopting key design concepts from Transformers to expand its global awareness. This duality makes it a natural bridge between CNNs and Transformers apable of preserving fine-grained details while understanding the broader context.
It also lays the groundwork for extra modules like multi-head attention and morphological feature extractors, ensuring the model starts with an entire, balanced feature set.
In brief, ConvNextV2 doesn’t just “see the parts”, it starts to understand how the parts come together. And in a task like dog breed classification, where each minute differences and overall structure matter, this sort of foundation is what transforms an odd model into one which can reason like an authority.
3. Technical Implementation of the MultiHeadAttention Mechanism
In neural networks, the core concept of the attention mechanism is to enable models to “focus” on key parts of the input, just like how human experts consciously deal with specific features (corresponding to ear shape, muzzle length, tail posture) when identifying dog breeds.
The Multi-Head Attention (MHA) mechanism further enhances this ability:
“Somewhat than having one expert evaluate all features, it’s higher to form a panel of experts, letting each deal with different details, after which synthesize a final judgment!”
Mathematically, MHA uses multiple linear projections to permit the model to concurrently learn different feature associations, further enhancing performance.
3.1 Understanding MultiHeadAttention from a Mathematical Perspective
The core idea of MultiHeadAttention is to make use of multiple different projections to permit the model to concurrently attend to patterns in several subspaces. Mathematically, it first projects input features into three roles: Query, Key, and Value, then calculates the similarity between Query (Q) and Key (K), and uses this similarity to perform weighted averaging of Values.
The essential formula will be expressed as:
[text{Attention}(Q, K, V) = text{softmax}left(frac{QK^T}{sqrt{d_k}}right)V]
3.2 Application of Einstein Summation Convention in Attention Calculation
Within the implementation, I used the torch.einsum function based on the Einstein summation convention to efficiently calculate attention scores:
energy = torch.einsum("nqd,nkd->nqk", [q, k])This implies:q has shape (batch_size, num_heads, query_dim)k has shape (batch_size, num_heads, key_dim)
The dot product is performed on dimension d, leading to (batch_size, num_heads, query_len, key_len) This is actually “calculating similarity between each Query and all Keys,” generating an attention weight matrix
3.3 Implementation Code Evaluation
Key implementation code for MultiHeadAttention:
def forward(self, x):
N = x.shape[0] # batch size
# 1. Project input, prepare for multi-head attention calculation
x = self.fc_in(x) # (N, input_dim) → (N, scaled_dim)
# 2. Calculate Query, Key, Value, and reshape into multi-head form
q = self.query(x).view(N, self.num_heads, self.head_dim) # query
k = self.key(x).view(N, self.num_heads, self.head_dim) # key
v = self.value(x).view(N, self.num_heads, self.head_dim) # value
# 3. Calculate attention scores (similarity matrix)
energy = torch.einsum("nqd,nkd->nqk", [q, k])
# 4. Apply softmax (normalize weights) and perform scaling
attention = F.softmax(energy / (self.head_dim ** 0.5), dim=2)
# 5. Use attention weights to perform weighted sum on Value
out = torch.einsum("nqk,nvd->nqd", [attention, v])
# 6. Rearrange output and go through final linear layer
out = out.reshape(N, self.scaled_dim)
out = self.fc_out(out)
return out3.3.1. Steps 1-2: Projection and Multi-Head Splitting
First, input features are projected through a linear layer, after which individually projected into query, key, and value spaces. Importantly, these projections not only change the feature representation but additionally split them into multiple “heads,” each attending to different feature subspaces.
3.3.2. Steps 3-4: Attention Calculation
3.3.3. Steps 5-6: Weighted Aggregation and Output Projection
Using the calculated attention weights, weighted summation is performed on the worth vectors to acquire the attended feature representation. Finally, outputs from all heads are concatenated and passed through an output projection layer to get the .
This implementation has the next simplifications and adjustments compared to plain Transformer MultiHeadAttention: Query, key, and value come from the identical input (self-attention), suitable for processing features obtained from CNN backbone networks.
It uses einsum operations to simplify matrix calculations.
The design of projection layers ensures dimensional consistency, facilitating integration with other modules.
3.4 How Attention Mechanisms Enhance Understanding of Morphological Feature Relationships
The multi-head attention mechanism brings three core benefits to dog breed recognition:
3.4.1. Feature Relationship Modeling
Just as knowledgeable veterinarian not only sees that ears are upright but additionally notices how this combines with tail curl degree and skull shape to form a dog breed’s “feature combination.”
It may well establish associations between different morphological features, capturing their synergistic relationships, not only seeing “what features exist” but observing “how these features mix.”
Application: The model can learn that a mixture of “pointed ears + curled tail + medium construct” points to specific Northern dog breeds.
3.4.2. Dynamic Feature Importance Assessment
Just as experts know to focus particularly on fur texture when identifying Poodles, while focusing mainly on the distinctive nose and head structure when identifying Bulldogs.
It dynamically adjusts deal with different features based on the precise content of the input.
Key features vary across different breeds, and the eye mechanism can adaptively focus.
Application: When seeing a Border Collie, the model might focus more on fur color distribution; when seeing a Dachshund, it would focus more on body proportions
3.4.3. Complementary Information Integration
Like a team of experts with different specializations, one specializing in skeletal structure, one other on fur features, one other analyzing behavioral posture, making a more comprehensive judgment together.
Through multiple attention heads, each concurrently captures several types of feature relationships. Each head can deal with a particular style of feature or relationship pattern.
Application: One head might primarily deal with color patterns, one other on body proportions, and yet one more on facial expression, ultimately synthesizing these perspectives to make a judgment.
By combining these three capabilities, the MultiHeadAttention mechanism goes beyond identifying individual features, it learns to model the complex relationships between them, capturing subtle patterns that emerge from their mixtures and enabling more accurate recognition.
4. Implementation Details of the Hybrid Architecture
4.1 The Overall Architectural Flow
When designing this hybrid architecture, my goal was easy yet ambitious:
Let each component do what it does best, and construct a complementary system where they enhance each other.
Very like a well-orchestrated symphony, each instrument (or module) plays its role, only together can they create harmony.
On this setup:
- The CNN focuses on capturing local details.
- The morphological feature extractor enhances key structural features.
- The multi-head attention module learns how these features interact.
As shown within the diagram above, the general model operates through five key stages:
4.1.1. Feature Extraction
Once a picture enters the model, ConvNextV2 takes charge of extracting foundational features, corresponding to fur color, contours, and texture. That is where the AI begins to “see” the essential shape and appearance of the dog.
4.1.2. Morphological Feature Enhancement
These initial features are then refined by the morphological feature extractor. This module functions like an authority’s eye—highlighting structural characteristics corresponding to ear shape and body proportions. Here, the AI learns to deal with what actually matters.
4.1.3. Feature Fusion
Next comes the feature fusion layer, which merges the local features with the improved morphological cues. But this isn’t just a straightforward concatenation, the layer also models how these features interact, ensuring the AI doesn’t treat them in isolation, but slightly understands how they mix to convey meaning.
4.1.4. Feature Relationship Modeling
The fused features are passed into the multi-head attention module, which builds contextual relationships between different attributes. The model begins to grasp mixtures like “ear shape + fur texture + facial proportions” slightly than taking a look at each trait independently.
4.1.5. Final Classification
In any case these layers of processing, the model moves to its final classifier, where it makes a prediction concerning the dog’s breed, based on the wealthy, integrated understanding it has developed.
4.2 Integrating ConvNextV2 and Parameter Setup
For implementation, I selected the pretrained ConvNextV2-base model because the backbone:
self.backbone = timm.create_model(
'convnextv2_base',
pretrained=True,
num_classes=0) # Use only the feature extractor; remove original classification headDepending on the input image size or backbone architecture, the feature output dimensions may vary. To construct a robust and versatile system, I designed a dynamic feature dimension detection mechanism:
with torch.no_grad():
dummy_input = torch.randn(1, 3, 224, 224)
features = self.backbone(dummy_input)
if len(features.shape) > 2:
features = features.mean([-2, -1]) # Global average pooling to provide a 1D feature vector
self.feature_dim = features.shape[1]This ensures the system mechanically adapts to any feature shape changes, keeping all downstream components functioning properly.
4.3 Intelligent Configuration of the Multi-Head Attention Layer
As mentioned earlier, I experimented with several head counts. Too many heads increased computation and risked overfitting. I ultimately settled on eight, but allowed the variety of heads to regulate mechanically based on feature dimensions:
self.num_heads = max(1, min(8, self.feature_dim // 64))
self.attention = MultiHeadAttention(self.feature_dim, num_heads=self.num_heads)4.4 Making CNN, Transformers, and Morphological Features Work Together
The morphological feature extractor works hand-in-hand with the eye mechanism.
While the previous provides structured representations of key traits, the latter models relationships amongst these features:
# Feature fusion
combined_features = torch.cat([
features, # Base features
morphological_features, # Morphological features
features * morphological_features # Interaction between features
], dim=1)
fused_features = self.feature_fusion(combined_features)
# Apply attention
attended_features = self.attention(fused_features)
# Final classification
logits = self.classifier(attended_features)
return logits, attended_featuresA special note concerning the third component features * morphological_features — this isn’t only a mathematical multiplication. It creates a type of dialogue between the 2 feature sets, allowing them to influence one another and generate richer representations.
For instance, suppose the model picks up “pointy ears” from the bottom features, while the morphological module detects a “small head-to-body ratio.”
Individually, these is probably not conclusive, but their interaction may strongly suggest a particular breed, like a Corgi or Finnish Spitz. It’s now not nearly recognizing ears or head size, the model learns to interpret how features work together, very like an authority would.
This full pipeline from feature extraction, through morphological enhancement and attention-driven modeling, to prediction is my vision of what a great architecture should appear like.
The design has several key benefits:
- The morphological extractor brings structured, expert-inspired understanding.
- The multi-head attention uncovers contextual relationships between traits.
- The feature fusion layer captures nonlinear interactions through element-wise multiplication.
4.5 Technical Challenges and How I Solved Them
Constructing a hybrid architecture like this was removed from smooth sailing.
Listed below are several challenges I faced and the way solving them helped me improve the general design:
4.5.1. Mismatched Feature Dimensions
- Challenge: Output sizes varied across modules, especially when switching backbone networks.
- Solution: Along with the dynamic dimension detection mentioned earlier, I implemented adaptive projection layers to unify the feature dimensions.
4.5.2. Balancing Performance and Efficiency
- Challenge: More complexity meant more computation.
- Solution: I dynamically adjusted the variety of attention heads, and used efficient
einsumoperations to optimize performance.
4.5.3. Overfitting Risk
- Challenge: Hybrid models are more vulnerable to overfitting, especially with smaller training sets.
- Solution: I applied LayerNorm, Dropout, and weight decay for regularization.
4.5.4. Gradient Flow Issues
- Challenge: Deep architectures often suffer from vanishing or exploding gradients.
- Solution: I introduced residual connections to make sure gradients flow easily during each forward and backward passes.
Should you’re keen on exploring the complete implementation, be happy to ascertain out the GitHub project here.
5. Performance Evaluation and Heatmap Evaluation
The worth of a hybrid architecture lies not only in its quantitative performance but additionally in the way it qualitatively “thinks.”
On this section, we’ll use confidence rating statistics and heatmap evaluation to reveal how the model evolved from CNN → CNN+Transformer → CNN+Transformer+MFE, and the way each stage brought its visual reasoning closer to that of a human expert.
To make sure that the performance differences got here purely from architecture design, I retrained each model using the very same dataset, augmentation methods, loss function, and training parameters. The one variation was the presence or absence of the Transformer and morphological modules.
When it comes to F1 rating, the CNN-only model reached 87.83%, the CNN+Transformer variant performed barely higher at 89.48%, and the ultimate hybrid model scored 88.70%. While the transformer-only version showed the very best rating on paper, it didn’t at all times translate into more reliable predictions. In actual fact, the hybrid model was more consistent in practice and handled similar-looking or blurry cases more reliably.
5.1 Confidence Scores and Statistical Insights
I tested 17 images of Border Collies, including standard photos, artistic illustrations, and various camera angles, to thoroughly assess the three architectures.
While other breeds were also included within the broader evaluation, I selected Border Collie as a representative case as a result of its distinctive features and frequent confusion with similar breeds.
Figure 1: Model Confidence Rating Comparison
As shown above, there are clear performance differences across the three models.
A notable example is Sample #3, where the CNN-only model misclassified the Border Collie as a Collie, with a low confidence rating of 0.2492.
While the CNN+Transformer corrected this error, it introduced a brand new one in Sample #5, misidentifying it as a Shiba Inu with 0.2305 confidence.
The ultimate CNN+Transformer+MFE model appropriately identified all samples without error. What’s interesting here is that each misclassifications occurred at low confidence levels (below 0.25).
This means that even when the model makes a mistake, it retains a way of uncertainty—a desirable trait in real world applications. We wish models to be cautious when unsure, slightly than confidently incorrect.
Figure 2: Confidence Rating Distribution
Taking a look at the distribution of confidence scores, the development becomes much more evident.
The CNN-only model mostly predicted within the 0.4–0.5 range, with few samples reaching beyond 0.6.
CNN+Transformer showed higher concentration around 0.5–0.6, but still had just one sample within the 0.7–0.8 high-confidence range.
The CNN+Transformer+MFE model stood out with 6 samples reaching the 0.7–0.8 confidence level.
This rightward shift in distribution reveals greater than just accuracy, it reflects certainty.
The model is evolving from “barely correct” to “confidently correct,” which significantly enhances its reliability in real-world deployment.
Figure 3: Statistical Summary of Model Performance
A deeper statistical breakdown highlights consistent improvements:
Mean confidence rating rose from 0.4639 (CNN) to 0.5245 (CNN+Transformer), and at last 0.6122 with the complete hybrid setup—a 31.9% increase overall.
Median rating jumped from 0.4665 to 0.6827, confirming the general shift toward higher confidence.
The proportion of high-confidence predictions (≥ 0.5) also showed striking gains:
- CNN: 41.18%
- CNN+Transformer: 64.71%
- CNN+Transformer+MFE: 82.35%
Which means with the ultimate architecture, most predictions aren’t only correct but confidently correct.
You would possibly notice a slight increase in standard deviation (from 0.1237 to 0.1616), which could seem to be a negative at first. But in point of fact, this reflects a more nuanced response to input complexity:
The model is highly confident on easier samples, and appropriately cautious on harder ones. The advance in maximum confidence value (from 0.6343 to 0.7746) further shows how this hybrid architecture could make more decisive and warranted judgments when presented with straightforward samples.
5.2 Heatmap Evaluation: Tracing the Evolution of Model Reasoning
While statistical metrics are helpful, they don’t tell the complete story.
To really understand how the model makes decisions, we want to see what it sees and heatmaps make this possible.
In these heatmaps, red indicates areas of high attention, highlighting the regions the model relies on most during prediction. By analyzing these attention maps, we will observe how each model interprets visual information, revealing fundamental differences of their reasoning styles.
Let’s walk through one representative case.
5.2.1 Frontal View of a Border Collie: From Local Eye Focus to Structured Morphological Understanding
When presented with a frontal image of a Border Collie, the three models reveal distinct attention patterns, reflecting how their architectural designs shape visual understanding.
The CNN-only model produces a heatmap with two sharp attention peaks, each centered on the dog’s eyes. This means a robust reliance on local features while overlooking other morphological traits just like the ears or facial outline. While eyes are indeed necessary, focusing solely on them makes the model more vulnerable to variations in pose or lighting. The resulting confidence rating of 0.5581 reflects this limitation.
With the CNN+Transformer model, the eye becomes more distributed. The heatmap forms a loose M-shaped pattern, extending beyond the eyes to incorporate the brow and the space between the eyes. This shift suggests that the model begins to grasp spatial relationships between features, not only the features themselves. This added contextual awareness results in a stronger confidence rating of 0.6559.
The CNN+Transformer+MFE model shows probably the most structured and comprehensive attention map. The warmth is symmetrically distributed across the eyes, ears, and the broader facial region. This means that the model has moved beyond feature detection and is now capturing how features are arranged as a part of a meaningful whole. The Morphological Feature Extractor plays a key role here, helping the model grasp the structural signature of the breed. This deeper understanding boosts the boldness to 0.6972.
Together, these three heatmaps represent a transparent progression in visual reasoning, from isolated feature detection, to inter-feature context, and at last to structural interpretation. Despite the fact that ConvNeXtV2 is already a strong backbone, adding Transformer and MFE modules enables the model to not only see features but to grasp them as a part of a coherent morphological pattern. This shift is subtle but crucial, especially for fine-grained tasks like breed classification.
5.2.2 Error Case Evaluation: From Misclassification to True Understanding
It is a case where the CNN-only model misclassified a Border Collie.
Taking a look at the heatmap, we will see why. The model focuses almost entirely on a single eye, ignoring many of the face. This type of over-reliance on one local feature makes it easy to confuse breeds that share similar traits on this case, a Collie, which also has similar eye shape and color contrast.
What the model misses are the broader facial proportions and structural details that outline a Border Collie. Its low confidence rating of 0.2492 reflects that uncertainty.
With the CNN+Transformer model, attention shifts in a more promising direction. It now covers each eyes and parts of the brow, making a more balanced attention pattern. This means the model is starting to connect multiple features, slightly than depending on only one.
Because of self-attention, it may possibly higher interpret relationships between facial components, resulting in the correct prediction — Border Collie. The boldness rating rises to 0.5484, greater than double the previous model’s.
The CNN+Transformer+MFE model takes this further by improving morphological awareness. The heatmap now extends to the nose and muzzle, capturing nuanced traits like facial length and mouth shape. These are subtle but necessary cues that help distinguish herding breeds from each other.
The MFE module seems to guide the model toward structural mixtures, not only isolated features. Consequently, confidence increases again to 0.5693, showing a more stable, breed-specific understanding.
This progression from a narrow deal with a single eye, to integrating facial traits, and at last to interpreting structural morphology, highlights how hybrid models support more accurate and generalizable visual reasoning.
In this instance, the CNN-only model focuses almost entirely on one side of the dog’s face. The remainder of the image is sort of ignored. This type of narrow attention suggests the model didn’t have enough visual context to make a robust decision. It guessed appropriately this time, but with a low confidence rating of 0.2238, it’s clear that the prediction wasn’t based on solid reasoning.
The CNN+Transformer model shows a broader attention span, however it introduces a distinct issue, the heatmap becomes scattered. You possibly can even spot a robust attention spike on the far right, completely unrelated to the dog. This type of misplaced focus likely led to a misclassification as a Shiba Inu, and the boldness rating was still low at 0.2305.
This highlights a crucial point:
Adding a Transformer doesn’t guarantee higher judgment unless the model learns where to look. Without guidance, self-attention can amplify the incorrect signals and create confusion slightly than clarity.
With the CNN+Transformer+MFE model, the eye becomes more focused and structured. The model now looks at key regions just like the eyes, nose, and chest, constructing a more meaningful understanding of the image. But even here, the boldness stays low at 0.1835, despite the right prediction. This image clearly presented an actual challenge for all three models.
That’s what makes this case so interesting.
It reminds us that an accurate prediction doesn’t at all times mean the model was confident. In harder scenarios unusual poses, subtle features, cluttered backgrounds even probably the most advanced models can hesitate.
And that’s where confidence scores change into invaluable.
They assist flag uncertain cases, making it easier to design review pipelines where human experts can step in and confirm tricky predictions.
5.2.3 Recognizing Artistic Renderings: Testing the Limits of Generalization
Artistic images pose a novel challenge for visual recognition systems. Unlike standard photos with crisp textures and clear lighting, painted artworks are sometimes abstract and distorted. This forces models to rely less on superficial cues and more on deeper, structural understanding. In that sense, they function an ideal stress test for generalization.
Let’s see how the three models handle this scenario.
Starting with the CNN-only model, the eye map is scattered, with focus diffused across either side of the image. There’s no clear structure — only a vague try and “see all the pieces,” which often means the model is unsure what to deal with. That uncertainty is reflected in its confidence rating of 0.5394, sitting within the lower-mid range. The model makes the right guess, however it’s removed from confident.
Next, the CNN+Transformer model shows a transparent improvement. Its attention sharpens and clusters around more meaningful regions, particularly near the eyes and ears. Even with the stylized brushstrokes, the model seems to infer, “this could possibly be an ear” or “that appears just like the facial outline.” It’s beginning to map anatomical cues, not only visual textures. The boldness rating rises to 0.6977, suggesting a more structured understanding is taking shape.
Finally, we have a look at the CNN+Transformer+MFE hybrid model. This one locks in with precision. The heatmap centers tightly on the intersection of the eyes and nose — arguably probably the most distinctive and stable region for identifying a Border Collie, even in abstract form. It’s now not guessing based on appearance. It’s reading the dog’s underlying structure.
This leap is basically due to the MFE, which helps the model deal with features that persist, even when style or detail varies. The result? A confident rating of 0.7457, the very best amongst all three.
This experiment makes something clear:
Hybrid models don’t just improve at recognition, they improve at reasoning.
They learn to look past visual noise and deal with what matters most: structure, proportion, and pattern. And that’s what makes them reliable, especially within the unpredictable, messy real world of images.
Conclusion
As Deep Learning evolves, we’ve moved from CNNs to Transformers—and now toward hybrid architectures that mix the perfect of each. This shift reflects a broader change in AI design philosophy: from in search of purity to embracing fusion.
Consider it like cooking. Great chefs don’t insist on one technique. They mix sautéing, boiling, and frying depending on the ingredient. Similarly, hybrid models mix different architectural “flavors” to suit the duty at hand.
This fusion design offers several key advantages:
- Complementary strengths: Like combining a microscope and a telescope, hybrid models capture each tremendous details and global context.
- Structured understanding: Morphological feature extractors bring expert-level domain insights, allowing models not only to see, but to actually understand.
- Dynamic adaptability: Future models might adjust internal attention patterns based on the image, emphasizing texture for spotted breeds, or structure for solid-colored ones.
- Wider applicability: From medical imaging to biodiversity and art authentication, any task involving fine-grained visual distinctions can profit from this approach.
This visual system—mixing ConvNeXtV2, attention mechanisms, and morphological reasoning proves that accuracy and intelligence don’t come from any single architecture, but from the fitting combination of ideas.
Perhaps the long run of AI won’t depend on one perfect design, but on learning to mix cognitive strategies just because the human brain does.
References & Data Source
Research References
Dataset Sources
- Stanford Dogs Dataset – Kaggle Dataset
Originally sourced from Stanford Vision Lab – ImageNet Dogs License: Non-commercial research and academic use only Citation: Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. FGVC Workshop, CVPR, 2011 - Unsplash Images – Additional images of 4 breeds (Bichon Frise, Dachshund, Shiba Inu, Havanese) were sourced from Unsplash for dataset augmentation.
Thanks for reading. Through developing PawMatchAI, I’ve learned many helpful lessons about AI vision systems and have recognition. If you could have any perspectives or topics you’d wish to discuss, I welcome the chance to exchange ideas. 🙌
📧 Email
💻 GitHub
Disclaimer
