add fourth chapter
This commit is contained in:
parent
09726aadb7
commit
f8b4defb75
|
|
@ -674,7 +674,7 @@ layers.4.0.weight has gradient mean of 1.3258541822433472
|
|||
>
|
||||
> - **有快捷连接**时,假设我们在每一层之间都添加快捷连接,梯度的传播路径就多了一条直接路径:
|
||||
>
|
||||
> $$\frac{\partial L}{\partial X_{1}}=\frac{\partial L}{\partial\left(X_{1}+F\left(X_{1}\right)\right)} \cdot\left(1+\frac{\partial F\left(X_{1}\right)}{\partial X_{1}}\right)$$
|
||||
> $$\frac{\partial L}{\partial X_{1}}=\frac{\partial L}{\partial\left(X_{1}+F\left(X_{1}\right)\right)} \cdot\left(1+\frac{\partial F\left(X_{1}\right)}{\partial X_{1}}\right)$$
|
||||
>
|
||||
> 这样,即使 $` \frac{\partial F\left(X_{1}\right)}{\partial X_{1}} `$ 很小,梯度依然可以通过 111 这条路径直接传递到更前面的层。
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue