正则化是用来解决神经网络过拟合的问题，通过降低模型的复杂性和约束权值，迫使神经网络学习可泛化的特征

• 对权重进行约束
• 对目标函数添加额外项（间接约束权值）：L1 & L2正则
• 降低网络复杂度：dropout，stochastic depth
• early stopping
• 我们在对网络进行正则化时不考虑网络的bias：正则表达式只是权值的表达式，不包含bias
  • bias比weight具有更少的参数量
  • 对bias进行正则化可能引入太多的方差，引入大量的欠拟合
  • 不是随机的将隐层节点的输出变成0，而是将节点中的每个与其相连的输入权值以1-p的概率变成0。（一个是输出一个是输入）
  • 训练阶段，对每个example／mini-batch, 每个epoch都随机sample一个mask矩阵

  • Dropconnect在测试期间采用了与标准dropout不同的方法。作者提出了dropconnect在每个神经元处的高斯近似，然后从这个高斯函数中抽取一个样本并传递给神经元激活函数。这使得dropconnect在测试时和训练时都是一种随机方法。

  • 伯努利分布：0-1分布

  • dropblock：是针对卷积层的正则化方法，相比较于dropout的random mute，能够更有效地remove掉部分语义信息，block size=1的时候退化成dropout

  • papers

    [dropout] Improving neural networks by preventing co-adaptation of feature detectors，丢节点

    [drop connect] Regularization of neural networks using dropconnect，丢weight path

    [Stochastic Depth] Deep Networks with Stochastic Depth，丢layer

    [DropBlock] A regularization method for convolutional networks

  • drop大法一句话汇总

  • dropout：各维度完全随机扔
  • spacial dropout：按照channel随机扔
  • stochastic depth：按照res block随机扔
  • dropblock：在feature map上按照spacial块随机扔
  • cutout：在input map上按照spacial块随机扔
  • dropconnect：扔连接不扔神经元

  Deep Networks with Stochastic Depth

  1. propose a training procedure：stochastic depth，train short and test deep
  2. for each mini-batch
     • randomly drop a subset of layers
     • and bypass them with the identity function
     • short：reduces training time
     • reg：improves the test error
     • can increase the network depth

     • deeper

     • expressiveness
     • vanishing gradients
     • diminishing feature reuse

     • resnet

     • skip connection

     • when输入输出channel数不match：redefine id(·) as a linear projection to reduce the dimensions

     • Dropout reduces the effect known as “co- adaptation” of hidden nodes
     • Dropout loses effectiveness when used in combination with Batch Normalization

     • our approach

     • higher diversity
     • shorter instead of thinner
     • work with Batch Normalization
     • all res path are active
     • each res path is weighted by its survival probability
     • $H_l^{Test} = ReLU(b_l Res_l(H_{l-1}, W_l) + id(H_{l-1}))$
     • 跟dropout一样