106 Commits

Author	SHA1	Message	Date
Clément Renault	1bd37d213a	Lowercase quoted words	2020-08-10 14:49:09 +02:00
Clément Renault	883a8109c8	Show both database and documents database sizes	2020-08-10 14:37:18 +02:00
Clément Renault	394844062f	Move the documents MTBL database inside the Index	2020-08-10 13:47:19 +02:00
Clément Renault	91282c8b6a	Move the documents into another file	2020-08-07 13:11:31 +02:00
Clément Renault	fae694a102	Put the documents into an MTBL database	2020-08-07 12:14:40 +02:00
Clément Renault	d3b1096510	Compute the word attribute postings lists on each threads	2020-08-06 11:50:27 +02:00
Kerollmops	9ade00e27b	Highlight all the matching words	2020-07-14 11:53:21 +02:00
Kerollmops	3d144e62c4	Search for best proximities in multiple attributes	2020-07-13 19:06:56 +02:00
Kerollmops	576dd011a1	Compute the candidates but not by attribute	2020-07-13 18:16:05 +02:00
Kerollmops	6b14b20369	Introduce a method to retrieve the number of attributes of the documents	2020-07-13 17:50:16 +02:00
Kerollmops	12358476da	Use the log crate instead of stderr	2020-07-12 10:55:09 +02:00
Kerollmops	d31da26a51	Avoid cloning RoraringBitmaps when unecessary	2020-07-11 23:51:32 +02:00
Kerollmops	b12bfcb03b	Reduce the deepness of the word position document ids ... This helps reduce the number of allocations.	2020-07-07 12:30:05 +02:00
Kerollmops	7178b6c2c4	First basic version using MTBL again	2020-07-07 11:32:33 +02:00
Kerollmops	ec1023e790	Intersect document ids by inverse popularity of the words ... This reduces the worst request we had which took 56s to now took 3s ("the best of the do").	2020-07-05 19:33:51 +02:00
Kerollmops	2fcae719ad	Use another LRU impl which uses hashbrown	2020-06-29 22:26:06 +02:00
Kerollmops	f98b615bf3	Replace the LRU by an Arc cache	2020-06-29 20:48:57 +02:00
Kerollmops	5f0088594b	Index by writing directly into LMDB	2020-06-29 13:54:47 +02:00
Kerollmops	63cbeca64e	Skip all derived words when too short	2020-06-28 12:13:12 +02:00
Kerollmops	736f0f7560	Use the proximity instead of the attributes when searching for <= 7 proximities	2020-06-28 12:13:12 +02:00
Kerollmops	fe3be8f18a	Replace the HashMap by a Vec for attributes documents ids	2020-06-28 12:13:12 +02:00
Kerollmops	7e16afbdce	Ignore documents which are not part of the candidates when exploring with A*	2020-06-24 15:06:45 +02:00
Kerollmops	1c7a9a4132	Remove the found documents from the candidates list	2020-06-24 15:00:26 +02:00
Kerollmops	50169b9798	Compute the full list of ids we are willing to find by attribute	2020-06-24 14:48:04 +02:00
Kerollmops	374ec6773f	Introduce a database to store all docids for a word and attribute	2020-06-22 19:24:20 +02:00
Kerollmops	ba3e805981	Document the Index types and the internal LMDB databases	2020-06-22 18:09:22 +02:00
Kerollmops	2f0e1afd16	Introduce the roaring bitmap heed codec	2020-06-22 17:56:07 +02:00
Kerollmops	8148210860	Use the cache when retrieving the documents at the end	2020-06-21 12:25:19 +02:00
Kerollmops	1628a31efa	Cache the unions of the derived words positions	2020-06-20 15:38:10 +02:00
Kerollmops	115e0142d9	Add a feature flags to enable the export of stats	2020-06-20 13:25:42 +02:00
Kerollmops	beb49b24f6	Skip looking at connections for proximity 0	2020-06-20 13:19:03 +02:00
Kerollmops	55a8941922	Optimize things	2020-06-19 17:48:17 +02:00
Kerollmops	3577de04b8	Reduce the number of KV lookups to the sucessfulls only	2020-06-16 12:58:29 +02:00
Kerollmops	e974e6b3c9	Acquire search intersections metrics	2020-06-16 12:10:23 +02:00
Kerollmops	8db16ff306	Add a cache to the contains_documents success function	2020-06-14 13:39:39 +02:00
Kerollmops	a8cda248b4	Introduce a customized A* algorithm. ... This custom algo lazily compute the intersections between words, to avoid too much set operations and database reads	2020-06-14 12:51:57 +02:00
Kerollmops	69285b22d3	Check that an edges combination contains results	2020-06-13 11:16:02 +02:00
Kerollmops	b9cc6c10af	Introduce a function to ignore useless paths	2020-06-13 00:17:43 +02:00
Kerollmops	d02c5cb023	Fix node skipping by computing the accumulated proximity	2020-06-12 14:08:46 +02:00
Kerollmops	0a83a86e65	Fix multiple bugs	2020-06-11 11:55:03 +02:00
Kerollmops	4e86ecf807	Retrieve the words before the intersect loops	2020-06-10 22:05:01 +02:00
Kerollmops	6ca3579cc0	Add more time debug measurements	2020-06-10 21:35:01 +02:00
Kerollmops	66a4b26811	Introduce a proximity based documents retriever	2020-06-10 16:54:28 +02:00
Kerollmops	2a6d6a7f69	Introduce a first draft of the best_proximity algorithm	2020-06-09 10:11:43 +02:00
Kerollmops	dfdaceb410	Introduce a first basic working positions-based engine	2020-06-05 20:13:19 +02:00
Kerollmops	f51a63e4ef	Store documents ids under attribute ids	2020-06-05 16:32:14 +02:00
Kerollmops	ce86a43779	Make the query tokenizer a real Iterator	2020-06-05 09:49:28 +02:00
Kerollmops	f55f4cb02a	Not fetch the cached prefix postings when prefix is disabled	2020-06-04 21:22:45 +02:00
Kerollmops	eefc6d7c44	Add support for quoted query phrases	2020-06-04 20:25:51 +02:00
Kerollmops	1f7035f18f	Just do a little clean-up	2020-06-04 19:13:28 +02:00
Kerollmops	71dc6a3828	Disable prefix search when query is ended by a whitespace	2020-06-04 18:37:20 +02:00
Kerollmops	5404776f7a	Add a little bit more debug	2020-06-01 17:52:43 +02:00
Kerollmops	dff68a339a	Use OnceCell to cache levenshtein builders	2020-05-31 19:27:11 +02:00
Kerollmops	dde3e01a59	Introduce prefix postings ids for better perfs	2020-05-31 18:20:49 +02:00
Kerollmops	a26553c90a	Reintroduce a simple HTTP server	2020-05-31 17:48:13 +02:00
Kerollmops	6c726df9b9	Support multiple space seperated words	2020-05-31 16:09:34 +02:00