A high-level overview of the newest convolutional kernel structures in Deformable Convolutional Networks, DCNv2, DCNv3
Because the remarkable success of OpenAI’s ChatGPT has sparked the boom of enormous language models, many individuals foresee the subsequent breakthrough in large image models. On this domain, vision models may be prompted to research and even generate images and videos in an identical manner to how we currently prompt ChatGPT.
The newest deep learning approaches for giant image models have branched into two important directions: those based on convolutional neural networks (CNNs) and people based on transformers. This text will deal with the CNN side and supply a high-level overview of those improved CNN kernel structures.
Traditionally, CNN kernels have been applied to fixed locations in each layer, leading to all activation units having the identical receptive field.
As within the figure below, to perform convolution on an input feature map x, the worth at each output location p0 is calculated as an element-wise multiplication and summation between kernel weight w and a sliding window on x. The sliding window is defined by a grid R, which can also be the receptive field for p0. The scale of R stays the identical across all locations throughout the same layer of y.
Each output value is calculated as follows:
where pn enumerates locations within the sliding window (grid R).
The RoI (region of interest) pooling operation, too, operates on bins with a set size in each layer. For (i, j)-th bin containing nij pixels, its pooling consequence is computed as:
Again shape and size of bins are the identical in each layer.
Each operations thus grow to be particularly problematic for high-level layers that encode semantics, e.g., objects with various scales.
DCN proposes deformable convolution and deformable pooling which can be more flexible to model those geometric structures. Each operate on the 2D spatial domain, i.e., the operation stays the identical across the channel dimension.
Deformable convolution
Given input feature map x, for every location p0 within the output feature map y, DCN adds 2D offsets △pn when enumerating each location pn in an everyday grid R.
These offsets are learned from preceding feature maps, obtained via an extra conv layer over the feature map. As these offsets are typically fractional, they’re implemented via bilinear interpolation.
Deformable RoI pooling
Much like the convolution operation, pooling offsets △pij are added to the unique binning positions.
As within the figure below, these offsets are learned through a totally connected (FC) layer after the unique pooling result.
Deformable Position-Sentitive (PS) RoI pooling
When applying deformable operations to PS RoI pooling (Dai et al., n.d.), as illustrated within the figure below, offsets are applied to every rating map as a substitute of the input feature map. These offsets are learned through a conv layer as a substitute of an FC layer.
Position-Sensitive RoI pooling (Dai et al., n.d.): Traditional RoI pooling loses information regarding which object part each region represents. PS RoI pooling is proposed to retain this information by converting input feature maps to k² rating maps for every object class, where each rating map represents a selected spatial part. So for C object classes, there are total k² (C+1) rating maps.
Although DCN allows for more flexible modelling of the receptive field, it assumes pixels inside each receptive field contribute equally to the response, which is commonly not the case. To higher understand the contribution behaviour, authors use three methods to visualise the spatial support:
- Effective receptive fields: gradient of the node response with respect to intensity perturbations of every image pixel
- Effective sampling/bin locations: gradient of the network node with respect to the sampling/bin locations
- Error-bounded saliency regions: progressively masking the parts of the image to search out the smallest image region that produces the identical response as all the image
To assign learnable feature amplitude to locations throughout the receptive field, DCNv2 introduces modulated deformable modules:
For location p0, the offset △pn and its amplitude △mn are learnable through separate conv layers applied to the identical input feature map.
DCNv2 revised deformable RoI pooling similarly by adding a learnable amplitude △mij for every (i,j)-th bin.
DCNv2 also expands the usage of deformable conv layers to switch regular conv layers in conv3 to conv5 stages in ResNet-50.
To scale back the parameter size and memory complexity from DCNv2, DCNv3 makes the next adjustments to the kernel structure.
- Inspired by depthwise separable convolution (Chollet, 2017)
Depthwise separable convolution decouples traditional convolution into: 1. depth-wise convolution: each channel of the input feature is convolved individually with a filter; 2. point-wise convolution: a 1×1 convolution applied across channels.
The authors propose to let the feature amplitude m be the depth-wise part, and the projection weight w shared amongst locations within the grid because the point-wise part.
2. Inspired by group convolution (Krizhevsky, Sutskever and Hinton, 2012)
Group convolution: Split input channels and output channels into groups and apply separate convolution to every group.
DCNv3 (Wang et al., 2023) propose splitting the convolution into G groups, each having separate offset △pgn and have amplitude △mgn.
DCNv3 is hence formulated as:
where G is the overall variety of convolution groups, wg is location irrelevant, △mgn is normalized by the softmax function in order that the sum over grid R is 1.
Up to now DCNv3 based InternImage has demonstrated superior performance in multiple downstream tasks akin to detection and segmentation, as shown within the table below, in addition to the leaderboard on papers with code. Discuss with the unique paper for more detailed comparisons.
In this text, now we have reviewed kernel structures for normal convolutional networks, together with their latest improvements, including deformable convolutional networks (DCN) and two newer versions: DCNv2 and DCNv3. We discussed the constraints of traditional structures and highlighted the advancements in innovation built upon previous versions. For a deeper understanding of those models, please consult with the papers within the References section.
Special due to Kenneth Leung, who inspired me to create this piece and shared amazing ideas. An enormous thanks to Kenneth, Melissa Han, and Annie Liao, who contributed to improving this piece. Your insightful suggestions and constructive feedback have significantly impacted the standard and depth of the content.
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H. and Wei, Y. (n.d.). Deformable Convolutional Networks. [online] Available at: https://arxiv.org/pdf/1703.06211v3.pdf.
Zhu, X., Hu, H., Lin, S. and Dai, J. (n.d.). Deformable ConvNets v2: More Deformable, Higher Results. [online] Available at: https://arxiv.org/pdf/1811.11168.pdf.
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., Wang, X. and Qiao, Y. (n.d.). InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions. [online] Available at: https://arxiv.org/pdf/2211.05778.pdf [Accessed 31 Jul. 2023].
Chollet, F. (n.d.). Xception: Deep Learning with Depthwise Separable Convolutions. [online] Available at: https://arxiv.org/pdf/1610.02357.pdf.
Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), pp.84–90. doi:https://doi.org/10.1145/3065386.
Dai, J., Li, Y., He, K. and Sun, J. (n.d.). R-FCN: Object Detection via Region-based Fully Convolutional Networks. [online] Available at: https://arxiv.org/pdf/1605.06409v2.pdf.