Commit Graph

2439 Commits

Author SHA1 Message Date
Clément Renault
4d92df1b95
Disable sled logging 2024-07-22 14:38:33 +02:00
Clément Renault
ca332883cc
Use an LRU only with 1000 entries 2024-07-22 13:39:20 +02:00
Clément Renault
6264dbf326
Increase the ARCache to 500 entries 2024-07-21 23:08:11 +02:00
Clément Renault
2b7b18fb5f
Fix one and for all the ARC cache 2024-07-21 22:32:14 +02:00
Clément Renault
b03ec3f603
Another algorithm for the ARC cache 2024-07-21 18:56:50 +02:00
Clément Renault
91ec0bdaf4
Replace the SmallVec by a Vec 2024-07-21 18:22:50 +02:00
Clément Renault
56329633d5
Implement a really adaptative cache 2024-07-21 17:46:51 +02:00
Clément Renault
507bce791b
Make sure all keys are prefixed 2024-07-21 15:40:32 +02:00
Clément Renault
7adc715783
Also count the direct_inserts 2024-07-20 17:15:33 +02:00
Clément Renault
f355cf6985
Use sled to count the write insertions 2024-07-20 11:16:57 +02:00
Clément Renault
2603d8d0d0
Add a way to disable the cache 2024-07-20 10:45:16 +02:00
Clément Renault
9b3d303b08
First ArcCache version 2024-07-20 10:45:15 +02:00
Clément Renault
16b4545d23
Prefix Redis keys to avoid false negative excessive writes 2024-07-20 10:45:15 +02:00
Clément Renault
0e08906fcb
Use the sorter cache when collection prefix docids 2024-07-20 10:45:15 +02:00
Clément Renault
a3beaa90c5
Use the sorter cache when extracting the facet number docids 2024-07-20 10:45:15 +02:00
Clément Renault
02fff51902
Use the sorter cache when extracting the facet string docids 2024-07-20 10:45:15 +02:00
Clément Renault
54e2e2aa4a
Use the sorter cache when extracting the word counts 2024-07-20 10:45:14 +02:00
Clément Renault
092a383419
Use the sorter cache when extracting the word pair proximity docids 2024-07-20 10:45:14 +02:00
Clément Renault
98d55e0d4d
Use the sorter cache when extracting the word position docids 2024-07-20 10:45:14 +02:00
Clément Renault
8319552e7d
Use the sorter cache in the word docids extractor 2024-07-20 10:45:14 +02:00
Clément Renault
5d5769fd8a
Introduce a new Sorter Cache for CboRoaringBitmaps 2024-07-20 10:45:14 +02:00
Clément Renault
eafc097a85
Measure much more places where we insert in sorters 2024-07-20 10:45:13 +02:00
Clément Renault
f17cb2ef5b
Use Redis to measure the Sorter insertions 2024-07-20 10:45:13 +02:00
meili-bors[bot]
ea73615abf
Merge #4804
4804: Implements the experimental contains filter operator r=irevoire a=irevoire

# Pull Request
Related PRD: (private link) https://www.notion.so/meilisearch/Contains-Like-Filter-Operator-0d8ad53c6761466f913432eb1d843f1e
Public usage page: https://meilisearch.notion.site/Contains-filter-operator-usage-3e7421b0aacf45f48ab09abe259a1de6

## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/3613

## What does this PR do?
- Extract the contains operator from this PR: https://github.com/meilisearch/meilisearch/pull/3751
- Gate it behind a feature flag
- Add tests


Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 15:47:11 +00:00
Tamo
02c61eabfa fix the range reported when the experimental feature has not been set 2024-07-17 16:54:33 +02:00
Tamo
2af9481804 Implements the experimental contains filter operator« 2024-07-17 11:13:37 +02:00
Louis Dureuil
24240934f9
Improve errors when indexing documents with a user provided embedder 2024-07-16 13:39:01 +02:00
Louis Dureuil
f4c94ac57f
manual embedders: limit max size of errors to 250 2024-07-16 13:39:01 +02:00
Louis Dureuil
4087a88dbe
rest|ollama|openai: increase tries to 10 + randomize retry duration 2024-07-16 13:39:00 +02:00
Louis Dureuil
5adacf2f45
OpenAI: embed only the first MAX_TOKENS tokens 2024-07-16 13:39:00 +02:00
Louis Dureuil
65d0c32aa7
Allow overriding OpenAI's url 2024-07-16 13:39:00 +02:00
Louis Dureuil
82647bcded
When retrieveVectors is true, retrieve _vectors.embedder even if there are no vector for that embedder 2024-07-16 13:39:00 +02:00
Louis Dureuil
e83da00446
Milli changes to match to allow for more flexible lifetimes 2024-07-11 16:29:35 +02:00
Louis Dureuil
7fb3e378ff
Do not fail sort comparisons when the field name or target point are different 2024-07-11 16:28:14 +02:00
meili-bors[bot]
29b44e5541
Merge #4626
4626: Edit Documents with Rhai r=ManyTheFish a=Kerollmops

This PR introduces a first version of [the _Update Documents with Function_ (internal)](https://www.notion.so/meilisearch/Update-Documents-by-Function-45f87b13e61c4435b73943768a490808). It uses [the Rhai programming language](https://rhai.rs/) to let users express the modifications they want apply.

You can read more about the way to use this functions on [the Usage PRD Page](https://meilisearch.notion.site/Edit-Documents-with-Rhai-0cff8fea7655436592e7c8a6de932062?pvs=25). The [prototype is available](https://github.com/meilisearch/meilisearch/actions/runs/9038384483) through Docker by using the following command:

```
docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-edit-documents-with-rhai-3
```

## TODO
 - [x] Support the `DocumentEdition` task in dumps.
 - [x] Remove the unwraps and panics.
 - [x] Improve error codes for the `function` parameter.
 - [x] [Update Rhai to v1.19.0](https://github.com/rhaiscript/rhai/releases/tag/v1.19.0) 🚀
 - [x] Make it an experimental feature (only restrict the HTTP calls).
 - [x] It must be possible not to send a context.
 - [x] Rebase on main.
 - [x] Check that the script cannot do any io.
 - [x] ~Introduce a `Documents.edit` action or~ require the `Documents.all` action.
 - [x] Change the `editionCode` to the clearer `function` field name in the tasks.
 - [x] Support a user provided context and maybe more (but keep function execution isolated for reproducibility).
 - [x] Support deleting documents when the `doc` is `()` (nil, null).
 - [x] Support canceling document edition.
 - [x] Multithread document edition by using rayon (and [rayon-par-bridge](https://docs.rs/rayon-par-bridge/latest/rayon_par_bridge/)).
 - [x] Limit the number of instruction by function execution.
 - [ ] ~Expose the limit of instructions in the settings.~ Not sure, in fact.
 - [x] Ignore unmodified documents in the tasks count.
 - [x] Make the `filter` field optional (not forced to be `null`).

Co-authored-by: Clément Renault <clement@meilisearch.com>
2024-07-11 09:02:55 +00:00
Clément Renault
6e80364c50
Apply review comments 2024-07-11 11:00:27 +02:00
Clément Renault
3bac22fd87
We do not do intersections with the universe when it is related to cache 2024-07-10 16:49:36 +02:00
Clément Renault
ce61cb7fe6
Simplify and speedup an intersection pass 2024-07-10 16:49:36 +02:00
Clément Renault
1693d1a311
Simplify the check to decide to stop a loop 2024-07-10 16:49:36 +02:00
Clément Renault
febea735ca
Remove the unused universe parameter from resolve_negative_phrases 2024-07-10 16:49:36 +02:00
Clément Renault
93ba051094
Remove the invalid get_phrases_docids universe parameter 2024-07-10 16:49:35 +02:00
Clément Renault
cd7a20fa32
Make it work by avoid storing invalid stuff in the cache 2024-07-10 16:49:35 +02:00
Clément Renault
41f51adbec
Do less useless intersections 2024-07-10 16:49:35 +02:00
Clément Renault
0ca1a4e805
Always do the intersections with the universe 2024-07-10 16:49:34 +02:00
Clément Renault
50a7393c55
Modify the compute_query_term_subset_docids function to accept the universe 2024-07-10 16:49:34 +02:00
Clément Renault
837274f853
Restrict even more the Rhai engine 2024-07-10 16:30:18 +02:00
Clément Renault
aace587dd1
Create errors for the internal processing ones 2024-07-10 16:29:18 +02:00
Clément Renault
f35d6710f3
Update rhai to v1.19.0 2024-07-10 16:29:17 +02:00
Clément Renault
81ec0abad1
Use the new rayon-par-bridge library 2024-07-10 16:29:04 +02:00
Clément Renault
b67d385cf0
Parallelize the edition functions 2024-07-10 16:28:54 +02:00