4 Commits

Author	SHA1	Message	Date
Andrej	e2065c59c6	use a bit more extended example that has my last name too because nice to show how it breaks up into more tokens	2022-07-12 04:31:31 +00:00
Andrej	d8dd157f9c	add a full example into the script as well	2022-07-12 04:25:17 +00:00
Nat Friedman	e9f6e3d448	Typos ... Fixed some small typos.	2022-07-11 20:55:38 -07:00
Andrej	9642f40b83	add a refactored BPE encoder from openai. Basically I dont super trust the huggingface tokenizer, the implementation sprawls multiple files and inheritance and has special magic handling around AddedTokens that I don't fully follow. Prefer to roll our own explicit implementation here that exactly mirrors the code of OpenAI and nothing else	2022-07-12 02:01:41 +00:00