78d8f43a88
TestPlan: - go test -run TestTokenize ./internal/tokenizer - go test -tags flex -run TestTokenize ./internal/tokenizer (shold fail as default fixtures are from regex-based tokenizer) |
||
---|---|---|
.. | ||
flex | ||
common.go | ||
tokenize_c.go | ||
tokenize_test.go | ||
tokenize.go |