charieswei
/
Build-A-Large-Language-Model-CN
mirror of https://github.com/skindhu/Build-A-Large-Language-Model-CN.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add fourth chapter

This commit is contained in:
skindhu 2024-11-04 19:49:10 +08:00
parent 09726aadb7
commit f8b4defb75
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
cn-Book/4.从零开始实现一个用于文本生成的 GPT 模型.md
View File

@ -674,7 +674,7 @@ layers.4.0.weight has gradient mean of 1.3258541822433472
> >
> - **有快捷连接**时，假设我们在每一层之间都添加快捷连接，梯度的传播路径就多了一条直接路径： > - **有快捷连接**时，假设我们在每一层之间都添加快捷连接，梯度的传播路径就多了一条直接路径：
> >
> ​ $$\frac{\partial L}{\partial X_{1}}=\frac{\partial L}{\partial\left(X_{1}+F\left(X_{1}\right)\right)} \cdot\left(1+\frac{\partial F\left(X_{1}\right)}{\partial X_{1}}\right)$$ > $$\frac{\partial L}{\partial X_{1}}=\frac{\partial L}{\partial\left(X_{1}+F\left(X_{1}\right)\right)} \cdot\left(1+\frac{\partial F\left(X_{1}\right)}{\partial X_{1}}\right)$$
> >
> 这样，即使 $` \frac{\partial F\left(X_{1}\right)}{\partial X_{1}} `$ 很小，梯度依然可以通过 111 这条路径直接传递到更前面的层。 > 这样，即使 $` \frac{\partial F\left(X_{1}\right)}{\partial X_{1}} `$ 很小，梯度依然可以通过 111 这条路径直接传递到更前面的层。