Kerollmops
68f4af7d2e
Improve the display of the number of processed documents
2020-09-29 16:08:58 +02:00
Kerollmops
59a127d022
Improve the indexing process
...
We now store the words pairs proximity in a cache and only compute the
shortest proximity between pairs of words in a document.
2020-09-29 15:09:18 +02:00
Kerollmops
6ddb3e722c
Depth-first search cache the docids unions
2020-09-28 16:55:21 +02:00
Kerollmops
a3821a0b33
Introduce the depth_first_search path resolution function
2020-09-28 16:34:12 +02:00
Clément Renault
d8354f6f02
Fix the word_docids capacity limit detection
2020-09-27 11:52:05 +02:00
Clément Renault
25b2853b70
Move the words pairs proximities compute into the write document function
2020-09-23 15:02:40 +02:00
Clément Renault
ed05999f63
Replace the arc cache by a simple linked hash map
2020-09-23 14:50:52 +02:00
Clément Renault
4d22d80281
Display only the key on heed error
2020-09-23 14:13:51 +02:00
Clément Renault
5178b3d59d
Make the search system be aware of query words typos
2020-09-23 12:01:39 +02:00
Clément Renault
b597a92487
Add a default max-memory value to the indexer
2020-09-23 12:00:36 +02:00
Clément Renault
1f6e00878d
Use the words pair proximities in the search algorithm
2020-09-22 18:47:55 +02:00
Clément Renault
31224a8425
Index the word pair proximities for both orders of the pair
2020-09-22 14:49:22 +02:00
Clément Renault
a58ae5eb2a
Introduce the word-pair-proximities-docids infos subcommand
2020-09-22 14:04:34 +02:00
Clément Renault
d6fa9c0414
Index the intra documents word pair proximities
2020-09-22 14:04:33 +02:00
Clément Renault
7b67ae6972
Introduce the StrStrU8 heed codec
2020-09-22 12:44:17 +02:00
Clément Renault
e34437b2d7
Move the proximity function to a module
2020-09-22 10:54:59 +02:00
Clément Renault
15208c7d3d
Simplify the indexer record loop
2020-09-22 10:33:30 +02:00
Clément Renault
e5adfaade0
Replace the token filter by a filter mapper
2020-09-22 10:24:31 +02:00
Clément Renault
d21c80b865
Apply the chunk compression parameters on all the MTBL writers
2020-09-21 18:30:54 +02:00
Clément Renault
944df52e2a
Simplify the indexer main loop
2020-09-21 14:59:48 +02:00
Kerollmops
3ded98e5fa
Bump the roaring version that fix a deserialization bug
2020-09-10 22:37:51 +02:00
Kerollmops
d5e5baa20f
Bump the oxidized-mtbl dependency
2020-09-10 13:29:12 +02:00
Kerollmops
aed0704404
Remove the temporary optimisation
2020-09-08 14:48:33 +02:00
Kerollmops
072382fa61
Sort the word docids to make intersections much faster
2020-09-07 22:38:49 +02:00
Kerollmops
ad11c5fb3f
Introduce the words-docids command for the infos binary
2020-09-07 22:36:35 +02:00
Kerollmops
5664c37539
Introduce an heed codec that reduce the size of small amount of serialized integers
2020-09-07 20:06:23 +02:00
Kerollmops
3e2250423c
Introduce the average-number-of-positions infos subcommand
2020-09-07 15:26:42 +02:00
Kerollmops
ea605b499c
Introduce two new infos subcommands
2020-09-07 14:56:48 +02:00
Clément Renault
bb1ab428db
Use another function to define the proximity
2020-09-06 17:55:07 +02:00
Clément Renault
dec460ce52
Fix the infos binary and add commands
2020-09-06 17:14:20 +02:00
Clément Renault
daa3673c1c
Invert the word docid positions key order
2020-09-06 10:30:53 +02:00
Clément Renault
c2405bcae2
Prefer using the word_docids db to create the words-fst
2020-09-06 10:23:56 +02:00
Kerollmops
4ca9472e02
Fix the minimum proximity len
2020-09-06 10:19:34 +02:00
Clément Renault
1c504471d3
Introduce the plane-sweep algorithm
2020-09-05 18:25:27 +02:00
Clément Renault
dc88a86259
Store the word positions under the documents
2020-09-05 18:03:06 +02:00
Kerollmops
580ed1119a
Make the engine to return csv string records as documents and headers
2020-08-31 19:02:00 +02:00
Clément Renault
bad0663138
Come back to the old tokenizer
2020-08-31 13:34:38 +02:00
Clément Renault
4afc4d0751
Use the groups of four positions to speed up disjunctions tests
2020-08-30 16:25:11 +02:00
Clément Renault
605f75b56f
Add the words grouped by four positions in the infos binary
2020-08-29 18:23:33 +02:00
Clément Renault
ad5cafbfed
Introduce a database to store docids in groups of four positions
2020-08-29 17:42:55 +02:00
Clément Renault
3db517548d
Move the documents back into the LMDB database
2020-08-29 15:14:04 +02:00
Clément Renault
816db7a0aa
Improve the RoaringBitmap codec to reserve enough vector space
2020-08-29 11:21:30 +02:00
Clément Renault
3fe497e129
Improve the Mtbl heed codec to only encode MTBL databases
2020-08-29 11:20:39 +02:00
Clément Renault
21aafd603c
Make sure the first document is associated to the document id 0
2020-08-29 10:56:40 +02:00
Clément Renault
0a44ff86ab
Put the documents MTBL back into LMDB
...
We makes sure to write the documents into a file before
memory mapping it and putting it into LMDB, this way we avoid
moving it to RAM
2020-08-28 15:43:24 +02:00
Clément Renault
d784d87880
Remove the prefix LMDB databases
2020-08-28 14:41:43 +02:00
Clément Renault
7cde312f14
Introduce the StrBEU32Codec heed codec
2020-08-28 14:16:37 +02:00
Clément Renault
34db376ae5
Rename the RoaringBitmapCodec module
2020-08-28 13:31:16 +02:00
Kerollmops
38ddc71b83
Simplify the search algorithm
2020-08-26 15:16:41 +02:00
Kerollmops
ba2eb0d7ad
Take the words-fst into account when retrieving the biggests values
2020-08-26 14:36:22 +02:00
Clément Renault
32da07ccee
Introduce the word-positions-doc-ids and words-positions infos commands
2020-08-23 10:52:47 +02:00
Clément Renault
d19f394630
Make the indexer support gzipped CSV as input
2020-08-21 18:10:24 +02:00
Clément Renault
ff479c865d
Replace pipe by ringtail to improve stdin read performances
2020-08-21 17:45:52 +02:00
Clément Renault
ada30c2789
Introducing more arguments to specify the different compression algorithms
2020-08-21 16:41:26 +02:00
Clément Renault
02335ee72d
Introduce the biggest-value-sizes command on the infos binary
2020-08-21 14:44:42 +02:00
Clément Renault
1e3e756c19
Introduce the words-frequencies command on the infos binary
2020-08-21 14:44:42 +02:00
Kerollmops
6a230fe803
Move the contains_documents logic to a function
2020-08-21 14:44:42 +02:00
Kerollmops
e55a569629
Compress much more the documents database
2020-08-21 14:44:42 +02:00
Kerollmops
962bad3cea
Introduce an infos binary to fetch stats
2020-08-17 19:41:49 +02:00
Clément Renault
8806fcd545
Introduce a better query and document lexer
2020-08-16 14:36:54 +02:00
Clément Renault
1e358e3ae8
Introduce the AstarBagIter that iterates through best paths
2020-08-15 16:24:06 +02:00
Clément Renault
7dc594ba4d
Introduce the Search builder struct
2020-08-13 14:27:51 +02:00
Clément Renault
bfb46cbfbe
Introduce the Crtierion enum
2020-08-12 10:43:02 +02:00
Clément Renault
6d04a285dc
Retrieve and display the distances of the words found
2020-08-11 15:18:02 +02:00
Clément Renault
1bd37d213a
Lowercase quoted words
2020-08-10 14:49:09 +02:00
Clément Renault
883a8109c8
Show both database and documents database sizes
2020-08-10 14:37:18 +02:00
Clément Renault
a4e0f3f724
Remove the useless TransitiveArc from the serve binary
2020-08-10 14:06:27 +02:00
Clément Renault
edc06a97d6
Remove the useless stats binary
2020-08-10 13:55:02 +02:00
Clément Renault
ae77fe5a69
Introduce an option to specify the maximum database size
2020-08-10 13:53:53 +02:00
Clément Renault
394844062f
Move the documents MTBL database inside the Index
2020-08-10 13:47:19 +02:00
Clément Renault
ecd2b2f217
Make the final merge done in parallel
2020-08-07 15:44:04 +02:00
Clément Renault
91282c8b6a
Move the documents into another file
2020-08-07 13:11:31 +02:00
Clément Renault
fae694a102
Put the documents into an MTBL database
2020-08-07 12:14:40 +02:00
Clément Renault
405a71d3a4
Accept csv from stdin
2020-08-06 13:38:21 +02:00
Clément Renault
d3b1096510
Compute the word attribute postings lists on each threads
2020-08-06 11:50:27 +02:00
Clément Renault
8d734941af
Clean up some lines
2020-08-06 10:20:26 +02:00
Clément Renault
6508d497ce
Replace the regex highlighting by a simple algorithm
2020-08-05 13:52:27 +02:00
Clément Renault
4873abe145
Introduce option flags to toggle the indexing engine
2020-08-05 12:10:41 +02:00
Clément Renault
bd4b18541c
Introduce a new indexer which uses an MTBL sorter
2020-08-04 15:44:37 +02:00
Kerollmops
ee305c9284
Replace the title by the milli logo
2020-07-15 23:55:28 +02:00
Kerollmops
9ade00e27b
Highlight all the matching words
2020-07-14 11:53:21 +02:00
Kerollmops
085c376655
Use the regex crate to highlight "hello"
2020-07-14 11:28:40 +02:00
Kerollmops
aa92311d4e
Add a dark theme to the dashboard
2020-07-13 23:51:41 +02:00
Kerollmops
3d144e62c4
Search for best proximities in multiple attributes
2020-07-13 19:06:56 +02:00
Kerollmops
576dd011a1
Compute the candidates but not by attribute
2020-07-13 18:16:05 +02:00
Kerollmops
6b14b20369
Introduce a method to retrieve the number of attributes of the documents
2020-07-13 17:50:16 +02:00
Kerollmops
92c2b1dd2d
Refine the help message of the binaries
2020-07-12 11:06:45 +02:00
Kerollmops
f757df5dfd
Introduce the stderr logger to the project
2020-07-12 11:04:35 +02:00
Kerollmops
12358476da
Use the log crate instead of stderr
2020-07-12 10:55:09 +02:00
Kerollmops
2c62eeea3c
Rename the project milli
2020-07-12 00:16:41 +02:00
Kerollmops
d31da26a51
Avoid cloning RoraringBitmaps when unecessary
2020-07-11 23:51:32 +02:00
Kerollmops
b8a1fc0126
Clean up the CSS style custom bulma rules
2020-07-11 14:51:59 +02:00
Kerollmops
f6eae91c7d
Pretty print the new dashboard numbers
2020-07-11 14:17:37 +02:00
Kerollmops
d44428fa90
Display more informations on the dashboard
2020-07-11 11:51:56 +02:00
Kerollmops
11c7fef80a
Implement a memory dumper
...
It moves the in memory HashMaps used when indexing to a disk based MTBL file
2020-07-07 16:48:49 +02:00
Kerollmops
b12bfcb03b
Reduce the deepness of the word position document ids
...
This helps reduce the number of allocations.
2020-07-07 12:30:05 +02:00
Kerollmops
7178b6c2c4
First basic version using MTBL again
2020-07-07 11:32:33 +02:00
Kerollmops
adb1038b26
Add a jobs parameter to set the number of threads the indexer uses
2020-07-06 12:17:17 +02:00
Kerollmops
ec1023e790
Intersect document ids by inverse popularity of the words
...
This reduces the worst request we had which took 56s to now took 3s ("the best of the do").
2020-07-05 19:33:51 +02:00
Kerollmops
cd7e64b2b3
Allow users to set the arc cache size when indexing
2020-07-04 18:12:41 +02:00
Kerollmops
ac8353a64f
Merge pre-computed word attribute documents ids
2020-07-04 17:02:27 +02:00
Kerollmops
fea7cac206
Display the time it took to compute the word attribute documents ids
2020-07-04 15:18:38 +02:00
Kerollmops
46ced5c828
Introduce the RwIter append heed API
2020-07-04 12:34:10 +02:00
Kerollmops
7e7440c431
Finalize the LMDB indexing design
2020-07-01 22:45:43 +02:00
Kerollmops
2ae3f40971
Make the indexer ignore certain words
...
This is a preparation for making the indexing fully parallel by making the
indexer only be aware of certain words for each threads to avoid postings lists
conflicts for each words
2020-07-01 17:49:46 +02:00
Kerollmops
a3ac2623d5
Introduce multiple functions to clean up the code
2020-07-01 17:24:55 +02:00
Kerollmops
ac5cc7ddad
Introduce an Iterator yielding owned entries for the LruCache
2020-07-01 17:21:52 +02:00
Kerollmops
014a25697d
Use only one ARC cache based on the words
2020-07-01 12:03:18 +02:00
Kerollmops
fc4013a43f
Fix the ARC cache
2020-07-01 10:35:07 +02:00
Kerollmops
2fcae719ad
Use another LRU impl which uses hashbrown
2020-06-29 22:26:06 +02:00
Kerollmops
f98b615bf3
Replace the LRU by an Arc cache
2020-06-29 20:48:57 +02:00
Kerollmops
07abebfc46
Introduce a (too big) LRU cache
2020-06-29 18:15:03 +02:00
Kerollmops
5f0088594b
Index by writing directly into LMDB
2020-06-29 13:54:47 +02:00
Kerollmops
63cbeca64e
Skip all derived words when too short
2020-06-28 12:13:12 +02:00
Kerollmops
736f0f7560
Use the proximity instead of the attributes when searching for <= 7 proximities
2020-06-28 12:13:12 +02:00
Kerollmops
fe3be8f18a
Replace the HashMap by a Vec for attributes documents ids
2020-06-28 12:13:12 +02:00
Kerollmops
6a2834f2b0
Add a jobs parameter to set the number of threads the indexer uses
2020-06-28 12:13:10 +02:00
Kerollmops
7e16afbdce
Ignore documents which are not part of the candidates when exploring with A*
2020-06-24 15:06:45 +02:00
Kerollmops
1c7a9a4132
Remove the found documents from the candidates list
2020-06-24 15:00:26 +02:00
Kerollmops
50169b9798
Compute the full list of ids we are willing to find by attribute
2020-06-24 14:48:04 +02:00
Kerollmops
374ec6773f
Introduce a database to store all docids for a word and attribute
2020-06-22 19:24:20 +02:00
Kerollmops
a044cb6cc8
Clean up the warnings for prefix postings
2020-06-22 18:10:31 +02:00
Kerollmops
ba3e805981
Document the Index types and the internal LMDB databases
2020-06-22 18:09:22 +02:00
Kerollmops
2f0e1afd16
Introduce the roaring bitmap heed codec
2020-06-22 17:56:07 +02:00
Kerollmops
8148210860
Use the cache when retrieving the documents at the end
2020-06-21 12:25:19 +02:00
Kerollmops
1628a31efa
Cache the unions of the derived words positions
2020-06-20 15:38:10 +02:00
Kerollmops
115e0142d9
Add a feature flags to enable the export of stats
2020-06-20 13:25:42 +02:00
Kerollmops
beb49b24f6
Skip looking at connections for proximity 0
2020-06-20 13:19:03 +02:00
Kerollmops
c84012d655
Accept queries from standard input when not given as argument
2020-06-20 12:01:15 +02:00
Kerollmops
55a8941922
Optimize things
2020-06-19 17:48:17 +02:00
Kerollmops
a3ca80d20d
Ignore every proximities bigger or equal to 8
2020-06-18 15:42:46 +02:00
Kerollmops
3577de04b8
Reduce the number of KV lookups to the sucessfulls only
2020-06-16 12:58:29 +02:00
Kerollmops
e974e6b3c9
Acquire search intersections metrics
2020-06-16 12:10:23 +02:00
Kerollmops
8db16ff306
Add a cache to the contains_documents success function
2020-06-14 13:39:39 +02:00
Kerollmops
a8cda248b4
Introduce a customized A* algorithm.
...
This custom algo lazily compute the intersections between words, to avoid too much set operations and database reads
2020-06-14 12:51:57 +02:00
Kerollmops
69285b22d3
Check that an edges combination contains results
2020-06-13 11:16:02 +02:00
Kerollmops
b9cc6c10af
Introduce a function to ignore useless paths
2020-06-13 00:17:43 +02:00
Kerollmops
d02c5cb023
Fix node skipping by computing the accumulated proximity
2020-06-12 14:08:46 +02:00
Kerollmops
37a48489da
Reworked the best proximity algo a little bit
2020-06-12 12:53:08 +02:00
Kerollmops
302866ad73
Make the algo don't work with an astar
2020-06-11 17:43:06 +02:00
Kerollmops
0a83a86e65
Fix multiple bugs
2020-06-11 11:55:03 +02:00
Kerollmops
4e86ecf807
Retrieve the words before the intersect loops
2020-06-10 22:05:01 +02:00
Kerollmops
6ca3579cc0
Add more time debug measurements
2020-06-10 21:35:01 +02:00
Kerollmops
66a4b26811
Introduce a proximity based documents retriever
2020-06-10 16:54:28 +02:00
Kerollmops
78f27c0465
squash-me: Remove debugs
2020-06-10 16:29:46 +02:00
Kerollmops
3ad883d7c7
squash-me: Make the dijkstra work even with different attributes
2020-06-10 16:27:02 +02:00
Kerollmops
fecd8ca54a
squash-me: It works! we must remove the debug after having added more tests
2020-06-10 14:20:35 +02:00
Kerollmops
13977d9338
squash-me
2020-06-09 23:06:59 +02:00
Kerollmops
5d5b827f1a
Squash-me
2020-06-09 17:32:25 +02:00
Kerollmops
2a6d6a7f69
Introduce a first draft of the best_proximity algorithm
2020-06-09 10:11:43 +02:00
