287: Add benchmarks for indexing r=Kerollmops a=irevoire
closes#274
I don't really know how much time this will take on our bench machine. I'm afraid the wiki dataset will take a really long time to bench (it takes 1h30 on my computer).
If you are ok with it, I would like to merge this first PR since it introduces a first set of benchmarks and see how much time it takes in reality on our setup.
Co-authored-by: Tamo <tamo@meilisearch.com>
285: Support documents with at most 65536 fields r=Kerollmops a=Kerollmops
Fixes#248.
In this PR I updated the `obkv` crate, it now supports arbitrary key length and therefore I was able to use an `u16` to represent the fields instead of a single byte. It was impressively easy to update the whole codebase 🍡🍔
Co-authored-by: Kerollmops <clement@meilisearch.com>
283: Use the AlwaysFreePages flag when opening an index r=irevoire a=Kerollmops
We introduced a new flag in our fork of LMDB, this `AlwaysFreePages` flag forces LMDB to always free the single pages it uses before writing to the disk instead of keeping them in a linked list.
Declaring this flag reduces the memory print (leak) we have on memory after indexing a lot of documents.
Fixes#279.
Co-authored-by: Kerollmops <clement@meilisearch.com>
284: [http-ui] Introduce the route `die` r=Kerollmops a=irevoire
This route just `exit` the process. This can come in handy when you run `http-ui` inside of another process (a profiler for example), and you don't want to kill everything
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
275: Fix the benchmarks dependencies r=Kerollmops a=irevoire
Import exactly the same dependency as milli instead of a wildcard that can do anything
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
276: Fix the fmt of the auto-generated file r=Kerollmops a=irevoire
The file generated by the `build.rs` file of the benchmark was badly formatted and that was causing an issue with the git pre-commit hook I wrote [earlier](https://github.com/meilisearch/milli/blob/main/script/pre-commit)
Co-authored-by: Tamo <tamo@meilisearch.com>
269: Fix bug when inserting previously deleted documents r=Kerollmops a=Kerollmops
This PR fixes#268.
The issue was in the `ExternalDocumentsIds` implementation in the specific case that an external document id was in the soft map marked as deleted.
The bug was due to a wrong assumption on my side about how the FST unions were returning the `IndexedValue`s, I thought the values returned in an array were in the same order as the FSTs given to the `OpBuilder` but in fact, [the `IndexedValue`'s `index` field was here to indicate from which FST the values were coming from](https://docs.rs/fst/0.4.7/fst/map/struct.IndexedValue.html).
271: Remove the roaring operation functions warnings r=Kerollmops a=Kerollmops
In this PR we are just replacing the usages of the roaring operations function by the new operators. This removes a lot of warnings.
Co-authored-by: Kerollmops <clement@meilisearch.com>
267: Highlighting r=Kerollmops a=irevoire
closes#262
I basically rewrote a part of the damerau-levenshtein function we were using for the highlighting to accept at most two errors from the user and stop on the third mistake.
Also, now it supports utf-8, so it should fix our issue.
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
266: Bump LMDB to the latest version (v0.9.70) r=Kerollmops a=Kerollmops
By bumping to a new version of heed (from git, v0.12.0 unpublished yet), this PR fixes Windows disk reservation problems. This new version of heed changes the `del/put_current`, and `append` iterator methods signature by declaring them unsafe.
This PR also bumps milli itself into v0.7.0 as it is breaking due to the heed/LMDB bump.
This PR must be merged after #264.
Co-authored-by: Clément Renault <clement@meilisearch.com>
It is undefined behavior to keep a reference to the database while
modifying it, we were keeping references in the database and also
feeding the heed put_current methods with keys referenced inside
the database itself.
https://github.com/Kerollmops/heed/pull/108
257: Fix unconditional facet indexing r=Kerollmops a=Kerollmops
We were indexing every searchable field as filterable, this was a mistake.
Co-authored-by: Kerollmops <clement@meilisearch.com>
246: Improve the ci r=Kerollmops a=irevoire
Rewrite the CI entirely:
- run the ci on Linux, macOS and Windows.
- run the ci on rust stable, beta and nightly
- add rustfmt to the CI.
- split the CI into multiple tasks, this way, the ci should be faster to fail
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
255: Fix facet distribution error r=Kerollmops a=Kerollmops
This PR fixes two invalid behaviors and fixes#253:
- We were ignoring the list of fields for which the user wanted a facet distribution.
- We were not raising any error for when a non-filterable field was requested a facet distribution.
~For the latter behavior I need the help of @curquiza to help me choose the right error type.~
Co-authored-by: Kerollmops <clement@meilisearch.com>
