typo fix: <unk> corrected to <|unk|>

This commit is contained in:
voltage-poppy 2025-06-16 23:59:08 +08:00 committed by GitHub
parent 3306177695
commit 7d85deacb6
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions

View File

@ -346,7 +346,7 @@ KeyError: 'Hello'
<img src="../Image/chapter2/figure2.10.png" width="75%" />
现在让我们修改词汇表将这两个特殊token <unk><|endoftext|> 包含在内,方法是将它们添加到我们在上一节中创建的唯一单词列表中:
现在让我们修改词汇表将这两个特殊token <|unk|><|endoftext|> 包含在内,方法是将它们添加到我们在上一节中创建的唯一单词列表中:
```python
all_tokens = sorted(list(set(preprocessed)))