openai模型BPE标记，压缩文本数据 https://github.com/openai/tiktoken
天问
ef915bb425
Update 'README.md'
|
1 year ago | |
|---|---|---|
| README.md | 1 year ago |
README.md
tiktoken
openai模型BPE标记
Usage
pip install tiktoken
import tiktoken
import torch