11 Commits

Author	SHA1	Message	Date
Clément Renault	2d8f9a9849	Add support for back/slashes	2019-11-11 21:23:08 +01:00
Clément Renault	de2b8672d4	Make the tokenizer understand strange whitespaces/quotes	2019-11-04 16:41:58 +01:00
Clément Renault	9dce41ed6b	Cargo clippy pass	2019-10-18 13:30:06 +02:00
Clément Renault	ca26a0f2e4	Cargo fmt pass	2019-10-18 13:30:06 +02:00
Clément Renault	cdd69290c3	test: Make the tests work with new separator limits	2019-09-24 20:49:42 +02:00
Clément Renault	175b3dcb75	fix: Do not consider underscores and middle dash hard	2019-09-24 20:14:20 +02:00
Clément Renault	6b9426a051	feat: Support underscore as a split character	2019-09-24 13:56:32 +02:00
Clément Renault	abf7191eec	feat: Make the Tokenizer able to support tokenizing sequences	2019-04-29 14:32:00 +02:00
Clément Renault	a745819ddf	feat: Simplify the Tokenizer to use the LinearStrGroupBy type	2019-04-29 14:31:37 +02:00
Clément Renault	19e67dcf0b	feat: Move query splitting into the tokenizer workspace	2019-04-29 14:31:37 +02:00
Clément Renault	1897da5348	feat: Move tokenizer things into the meilidb-tokenizer workspace	2019-04-29 14:31:37 +02:00