This website requires JavaScript.
explore
Help
Register
Sign In
mirror
/
minGPT
Watch
1
Star
0
Fork
0
You've already forked minGPT
mirror of
https://github.com/karpathy/minGPT
synced
2026-05-04 06:00:47 +02:00
Code
Issues
Activity
Files
master
minGPT
/
tests
T
Add File
New File
Upload File
Apply Patch
Copy Permalink
Download directory as ZIP
Download directory as TAR.GZ
History
Andrej
9642f40b83
add a refactored BPE encoder from openai. Basically I dont super trust the huggingface tokenizer, the implementation sprawls multiple files and inheritance and has special magic handling around AddedTokens that I don't fully follow. Prefer to roll our own explicit implementation here that exactly mirrors the code of OpenAI and nothing else
2022-07-12 02:01:41 +00:00
..
test_huggingface_import.py
add a refactored BPE encoder from openai. Basically I dont super trust the huggingface tokenizer, the implementation sprawls multiple files and inheritance and has special magic handling around AddedTokens that I don't fully follow. Prefer to roll our own explicit implementation here that exactly mirrors the code of OpenAI and nothing else
2022-07-12 02:01:41 +00:00