Lecture
«Bi-Real Net (a binary neural network): Enhancing the Performance of 1-bit CNNs with Improved Representational Capability and Advanced Training Algorithm» This paper introduces a number of improvements to the network structure and training optimization in response to the shortcomings of XNOR-Net, including introducing a shortcut connection in the form of one layer per block, using a quadratic function to approximate the derivative of the sign activation function for real-valued numbers, incorporating the magnitude of the real-valued weights when updating the real-valued weights, and pretraining the model using a clip function instead of ReLU for training, thereby obtaining binarized network weights and activations. Although the output is binarized, this yields high inference accuracy, especially on large datasets (ILSVRC ImageNet).
As shown in the figure above, once the weights and activations of a deep network are binarized, the memory footprint of the parameters and activations can be significantly reduced. Second, the convolution computation is also simplified to XNOR operations and bit-counting, which greatly facilitates deploying deep networks for inference, especially on mobile devices with limited resources and extremely demanding energy-efficiency requirements. XNOR-Net, ABC-Net, and others are currently the leading binary networks. Bi-Real Net, proposed in the paper, is a further development and improvement of XNOR-Net. First, a shortcut in the form of one layer per block is implemented in the network structure: that is, the real-valued output of the current 1-bit convolution or BN layer is added directly to the real-valued output of the next 1-bit convolution or BN layer, thereby increasing the network's range of values. The structure is as follows:
In a binary network, binarization of the activation A and the weight W is usually implemented using the Sign operation:
The training scheme for the binary network proposed in the paper is shown in the figure below, including the forward process (sign operation on the real-valued activations, magnitude-aware sign operation on the real-valued weights, 1-bit Conv, BN, etc.) and the backward process (gradients with respect to the binary weights, updating of the real-valued weights, etc.):
During training, the network's real-valued weights are updated directly, and the gradient of the loss with respect to the real-valued weights can be expanded according to the chain rule (the product terms are the gradient of the loss with respect to the binary weight, and the gradient of the binary weight with respect to the real-valued weight):
Among these, the gradient of the loss with respect to the binary weight can be expanded according to the chain rule as follows (the product terms are the gradient of the loss with respect to the real-valued activation, the gamma coefficient of the BN layer, and the binary activation):
At this point, the gradient of the loss with respect to the real-valued activation must first be resolved (it can be expressed, by the chain rule, as the product of the gradient of the loss with respect to the binary activation and the gradient of the binary activation with respect to the real-valued activation), but since the Sign function is not differentiable (its derivative is a unit impulse response), a differentiable approximation function F of the Sign function needs to be designed as a straight-through estimator:
Approximating the Sign function introduces a discrepancy between the approximate gradient obtained during backpropagation and the real activation, so the degree of approximation between the approximation function and the Sign function affects the model's performance. XNOR-Net chooses the piecewise-linear function clip(-1, x, 1) as its approximation of the Sign function, which has a large deviation. This paper constructs a quadratic function as a straight-through approximation of the Sign function, which can further reduce the deviation and improve model performance, and whose corresponding derivative curve has a triangular shape, better suited for modeling impulse signals:
During training, since the gradient of the loss with respect to the binary weights is usually small, it is difficult to induce changes in the real-valued weights. This paper introduces information about the magnitude of the real-valued weights, i.e., the sign operation on the real-valued weights is replaced during training by a magnitude-aware sign operation, which can increase the gradient of the binary weights with respect to the real-valued weights, thereby increasing the size of the update step for the real-valued weight. This helps accelerate the model's convergence rate:
The final recursive formula used to update the real-valued weights is as follows:
The product term in the update step is the gradient of the loss with respect to the real-valued activation, the gamma coefficient of the BN layer, the binary activation, and the gradient of the magnitude-aware Sign operation. The gradient of the magnitude-aware Sign operation, i.e., the gradient of the binary weight with respect to the real-valued weight, is expressed as (the derivative of the Sign function is still approximated by the Clip function):
Since the model is at the inference-deployment stage, binarization of the real-valued weights still uses the ordinary Sign operation, so after the magnitude-aware Sign operation has been used to train the model to convergence, it is necessary to use the plain Sign operation to train for a few more epochs in order to update the moving parameters of the BN layer (with the learning rate set to zero at this stage) so as to meet the requirements of actual deployment.
In addition, since the binarized output of the binary network is {-1, 1} and contains no zero elements, when pretraining the model, choosing the clip function instead of ReLU as the non-linear activation function can achieve a better initialization effect. At the same time, during training of the binary network, weight decay is set to zero, i.e., there is no need to impose L1 or L2 regularization constraints on the real-valued weights. Also, a shortcut structure with one connection per block outperforms a shortcut structure with two connections per block; see the experimental analysis for details.
Experimental results show that Bi-Real Net outperforms XNOR-Net, ABC-Net, and others on datasets such as CIFAR-10/100 and ImageNet, while having fewer parameters. It is very well suited for deployment on mobile terminals and can be accelerated with the help of the NEON instruction set:
[[b6252]]
Comments