3 Commits

Author	SHA1	Message	Date
Andrej	9642f40b83	add a refactored BPE encoder from openai. Basically I dont super trust the huggingface tokenizer, the implementation sprawls multiple files and inheritance and has special magic handling around AddedTokens that I don't fully follow. Prefer to roll our own explicit implementation here that exactly mirrors the code of OpenAI and nothing else	2022-07-12 02:01:41 +00:00
Andrej	acaadacd59	refactor sequence generation into the model and match the huggingface/transformers api. touches everything but this makes a lot more sense to me aesthetically	2022-07-11 18:50:53 +00:00
Andrej	803f38800d	refactor pretrained weight loading into from_pretrained and add unit tests	2022-07-08 22:56:15 +00:00