Alexander Bezzubov 78d8f43a88 tokenizer: hide flex-based impl, avoid build failures on win ... TestPlan: - go test -run TestTokenize ./internal/tokenizer - go test -tags flex -run TestTokenize ./internal/tokenizer (shold fail as default fixtures are from regex-based tokenizer)		2020-03-19 19:58:48 +01:00
..
flex	tokenizer: hide flex-based impl, avoid build failures on win	2020-03-19 19:58:48 +01:00
common.go	address review feedback	2019-04-16 19:38:48 +02:00
tokenize_c.go	*: module rename to go-enry/go-enry/v4	2020-03-19 17:31:29 +01:00
tokenize_test.go	tokenizer: hide flex-based impl, avoid build failures on win	2020-03-19 19:58:48 +01:00
tokenize.go	tokenizer: fix a bug and regenerate the code \w latest Go	2020-03-19 19:08:21 +01:00