308: Implement a better parallel indexer r=Kerollmops a=ManyTheFish
Rewrite the indexer:
- enhance memory consumption control
- optimize parallelism using rayon and crossbeam channel
- factorize the different parts and make new DB implementation easier
- optimize and fix prefix databases
Co-authored-by: many <maxime@meilisearch.com>
337: Update version for the next release (v0.12.0) r=Kerollmops a=curquiza
Breaking because of the new indexer that implies DB changes #308
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
335: Get sortable_fields from index only if criteria present in query r=Kerollmops a=shekhirin
Seems like we don't need to retrieve `sortable_fields` from the index if there's no any `sort_criteria` in the query.
Small 🤏 optimization opportunity out there.
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
329: Run all benchmarks once every friday r=irevoire a=irevoire
All the benchmarks run every Friday on the `main` branch.
To avoid having pending benchmarks everywhere, we execute one benchmark every 8 hours.
Then the results are uploaded as if it was a normal user-run benchmark.
This PR closes#314 and #321
Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
330: Introduce the reset_sortable_fields Settings method r=irevoire a=Kerollmops
I forgot to add the `reset_sortable_fields` method on the `Settings` builder, it is no big deal as the library user (like MeiliSearch) can always call `set_sortable_fields` with an empty list of fields, it is equivalent.
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
332: Sortable attributes in http-ui r=Kerollmops a=irevoire
- Add a `reset_sortable_attribute` method
- Add the `sortable_attributes` to http-ui
- Fix some broken test in http-ui
Co-authored-by: Tamo <tamo@meilisearch.com>
325: Update milli version to v0.11.0 r=curquiza a=Kerollmops
This PR also clean-up some dependencies in the Cargo.toml.
Co-authored-by: Kerollmops <clement@meilisearch.com>
315: Rewrite the indexing benchmarks r=Kerollmops a=irevoire
There was a panic on the benchmark and while I was trying to understand what was happening I decided to rewrite the way the benchmarks were working.
Before we were creating a database with the good setting, and then for each benchmarks we were:
1. Deleting all documents in the database
2. Indexing a batch of documents
Now for each iteration we recreate entirely a new database from scratch.
Since deleting all the documents in a database may not be the same as starting with a fresh new database I prefer this solution.
Co-authored-by: Irevoire <tamo@meilisearch.com>
320: Sort at query time r=Kerollmops a=Kerollmops
Re-introduce the Sort at the query time (https://github.com/meilisearch/milli/issues/305)
Co-authored-by: Clément Renault <renault.cle@gmail.com>
317: Fix the facet string docids filterable deletion bug r=Kerollmops a=Kerollmops
Fixes a bug where the deletion of documents was returning a decoding error. But only when the settings are set with filterable attributes.
This bug was introduced in #254 in which we made the engine faster in returning the facet distribution. We changed the way we were storing the inverted index, we were no more storing only documents ids with the original values but also groups identified with integers, depending on the facet level we were using. This is similar to how facet numbers are already stored.
⚠️ As `@curquiza` already said, we must first revert #309 before merging this!
Related to https://github.com/meilisearch/MeiliSearch/issues/1601.
Co-authored-by: Clément Renault <clement@meilisearch.com>
318: Revert "Sort at query time" r=Kerollmops a=curquiza
Reverts meilisearch/milli#309
We revert this from `main` not because this leads to a bug, but because we don't want to release it now and we have to merge and release an hotfix on `main`.
Cf:
- https://github.com/meilisearch/milli/issues/316
- https://github.com/meilisearch/milli/pull/317
Once the v0.21.0 is released, we should merge again this awesome addition 👌
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
309: Sort at query time r=Kerollmops a=Kerollmops
This PR:
- Makes the `Asc/Desc` criteria work with strings too, it first returns documents ordered by numbers then by strings, and finally the documents that can't be ordered. Note that it is lexicographically ordered and not ordered by character, which means that it doesn't know about wide and short characters i.e. `a`, `丹`, `▲`.
- Changes the syntax for the `Asc/Desc` criterion by now using a colon to separate the name and the order i.e. `title:asc`, `price:desc`.
- Add the `Sort` criterion at the third position in the ranking rules by default.
- Add the `sort_criteria` method to the `Search` builder struct to let the users define the `Asc/Desc` sortable attributes they want to use at query time. Note that we need to check that the fields are registered in the sortable attributes before performing the search.
- Introduce a new `InvalidSortableAttribute` user error that is raised when the sort criteria declared at query time are not part of the sortable attributes.
- `@ManyTheFish` introduced integration tests for the dynamic Sort criterion.
Fixes#305.
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: many <maxime@meilisearch.com>
