add third chapter

This commit is contained in:
skindhu 2024-11-02 14:57:42 +08:00
parent b446670385
commit 5498cf5935
1 changed files with 3 additions and 1 deletions

View File

@ -998,7 +998,9 @@ tensor([[2., 2., 0., 2., 2., 0.],
>
> 在应用 dropout 时,一部分注意力权重被随机置零(假设 dropout 率为 p。剩余的权重会被放大其放大倍数为 $` \frac{1}{1-p} `$。放大后的权重记为 z
>
> $$ \frac{z_{i}}{1-p} \quad \text { (对于未被置零的权重) } $$
> $$ \text{z_{i}}^{\prime}=\frac{z_{i}}{1-p} \quad \text { (对于未被置零的权重) } $$
>
> $$ \text{softmax}\left(z_{i}\right)=\frac{e^{z_{i}}}{\sum_{j} e^{z_{j}}} $$
>
>
>