Clément Renault
4d92df1b95
Disable sled logging
2024-07-22 14:38:33 +02:00
Clément Renault
ca332883cc
Use an LRU only with 1000 entries
2024-07-22 13:39:20 +02:00
Clément Renault
6264dbf326
Increase the ARCache to 500 entries
2024-07-21 23:08:11 +02:00
Clément Renault
2b7b18fb5f
Fix one and for all the ARC cache
2024-07-21 22:32:14 +02:00
Clément Renault
b03ec3f603
Another algorithm for the ARC cache
2024-07-21 18:56:50 +02:00
Clément Renault
91ec0bdaf4
Replace the SmallVec by a Vec
2024-07-21 18:22:50 +02:00
Clément Renault
56329633d5
Implement a really adaptative cache
2024-07-21 17:46:51 +02:00
Clément Renault
507bce791b
Make sure all keys are prefixed
2024-07-21 15:40:32 +02:00
Clément Renault
7adc715783
Also count the direct_inserts
2024-07-20 17:15:33 +02:00
Clément Renault
f355cf6985
Use sled to count the write insertions
2024-07-20 11:16:57 +02:00
Clément Renault
2603d8d0d0
Add a way to disable the cache
2024-07-20 10:45:16 +02:00
Clément Renault
9b3d303b08
First ArcCache version
2024-07-20 10:45:15 +02:00
Clément Renault
16b4545d23
Prefix Redis keys to avoid false negative excessive writes
2024-07-20 10:45:15 +02:00
Clément Renault
0e08906fcb
Use the sorter cache when collection prefix docids
2024-07-20 10:45:15 +02:00
Clément Renault
a3beaa90c5
Use the sorter cache when extracting the facet number docids
2024-07-20 10:45:15 +02:00
Clément Renault
02fff51902
Use the sorter cache when extracting the facet string docids
2024-07-20 10:45:15 +02:00
Clément Renault
54e2e2aa4a
Use the sorter cache when extracting the word counts
2024-07-20 10:45:14 +02:00
Clément Renault
092a383419
Use the sorter cache when extracting the word pair proximity docids
2024-07-20 10:45:14 +02:00
Clément Renault
98d55e0d4d
Use the sorter cache when extracting the word position docids
2024-07-20 10:45:14 +02:00
Clément Renault
8319552e7d
Use the sorter cache in the word docids extractor
2024-07-20 10:45:14 +02:00
Clément Renault
5d5769fd8a
Introduce a new Sorter Cache for CboRoaringBitmaps
2024-07-20 10:45:14 +02:00
Clément Renault
eafc097a85
Measure much more places where we insert in sorters
2024-07-20 10:45:13 +02:00
Clément Renault
f17cb2ef5b
Use Redis to measure the Sorter insertions
2024-07-20 10:45:13 +02:00
meili-bors[bot]
48bc797dce
Merge #4812
...
4812: Allow `MEILI_NO_VERGEN` env var to skip vergen r=irevoire a=dureuill
- vergen checks the state of the `.git` directory to embed commit information into the `meilisearch` binary and the `cargo xtask bench` invocations.
- This check unfortunately results in too many recompilation of the `meilisearch` binary.
- This PR allows skipping vergen when the `MEILI_NO_VERGEN` variable is present in the environment
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-18 16:16:01 +00:00
Louis Dureuil
c6b33fd407
Allow MEILI_NO_VERGEN env var to skip vergen
2024-07-18 17:28:01 +02:00
meili-bors[bot]
6e9d0de8b7
Merge #4806
...
4806: Update rustls as much as possible r=Kerollmops a=irevoire
# Pull Request
## Related issue
Part of https://github.com/meilisearch/meilisearch/issues/4753
## What does this PR do?
- Update rustls as much as possible
## What is missing
In rustls-0.22.0 two structures we were using have been removed with no explanation or workaround
<img width="518" alt="image" src="https://github.com/user-attachments/assets/fa112db1-3400-4163-8819-7913f22d6b87 ">
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 17:00:01 +00:00
Tamo
1bfb16386c
Update rustls as much as possible
2024-07-17 18:21:26 +02:00
meili-bors[bot]
ea73615abf
Merge #4804
...
4804: Implements the experimental contains filter operator r=irevoire a=irevoire
# Pull Request
Related PRD: (private link) https://www.notion.so/meilisearch/Contains-Like-Filter-Operator-0d8ad53c6761466f913432eb1d843f1e
Public usage page: https://meilisearch.notion.site/Contains-filter-operator-usage-3e7421b0aacf45f48ab09abe259a1de6
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/3613
## What does this PR do?
- Extract the contains operator from this PR: https://github.com/meilisearch/meilisearch/pull/3751
- Gate it behind a feature flag
- Add tests
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 15:47:11 +00:00
Tamo
02c61eabfa
fix the range reported when the experimental feature has not been set
2024-07-17 16:54:33 +02:00
Tamo
56b60ec7a0
apply review comment
2024-07-17 16:13:40 +02:00
meili-bors[bot]
8f416e8f34
Merge #4805
...
4805: Log the time to index a batch of task r=Kerollmops a=irevoire
This was proposed by `@qdequele` in a private conversation and I think it’s a nice addition.
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-07-17 11:45:39 +00:00
Tamo
cf760cbfb1
Log the time to index a batch of task
2024-07-17 11:56:57 +02:00
Tamo
2af9481804
Implements the experimental contains filter operator«
2024-07-17 11:13:37 +02:00
meili-bors[bot]
7a292b572a
Merge #4801
...
4801: AI quality-of-life improvements r=irevoire a=dureuill
# Pull Request
## Related issue
Fixes #4802
## What does this PR do?
This PR implements several quality-of-life improvements described in the [public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#ece824a1814e47a0a986d786baff1be9 )
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-17 09:00:47 +00:00
Louis Dureuil
8d6ac261ae
Add tests on various failure modes for embedders
2024-07-16 13:39:02 +02:00
Louis Dureuil
b4c8b01c88
Update existing snapshots
2024-07-16 13:39:01 +02:00
Louis Dureuil
24240934f9
Improve errors when indexing documents with a user provided embedder
2024-07-16 13:39:01 +02:00
Louis Dureuil
f4c94ac57f
manual embedders: limit max size of errors to 250
2024-07-16 13:39:01 +02:00
Louis Dureuil
4087a88dbe
rest|ollama|openai: increase tries to 10 + randomize retry duration
2024-07-16 13:39:00 +02:00
Louis Dureuil
5adacf2f45
OpenAI: embed only the first MAX_TOKENS tokens
2024-07-16 13:39:00 +02:00
Louis Dureuil
65d0c32aa7
Allow overriding OpenAI's url
2024-07-16 13:39:00 +02:00
Louis Dureuil
82647bcded
When retrieveVectors is true, retrieve _vectors.embedder even if there are no vector for that embedder
2024-07-16 13:39:00 +02:00
meili-bors[bot]
1582c7e788
Merge #4769
...
4769: Federated search r=ManyTheFish a=dureuill
# Pull Request
## Related issue
Fixes #4747
[Usage](https://meilisearch.notion.site/v1-10-federated-search-698dfe36ab6b4668b044f735fb40f0b2 )
## What does this PR do?
- multi-search now allows a top-level federation object. When not `null`, the results of multi-search are modified to be a single list of results rather than a list of a list of results
- changed lifetimes around tokenizer et al. to be able to make hits one by one rather than using a vector
- adds `roaring` to Meilisearch itself. As the federated search happens at the Meilisearch level (reuses the search functions declared at the Meilisearch level + merge happens after the hits were created), `RoaringBitmap`s are needed to track the candidates: hits that were seen, all candidates.
- Refactor `make_hits` to allow for an individual, optimized `make_hit`
- Score details comparison no longer fail when sorting on different field names or target point (for geo)
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-07-16 08:14:46 +00:00
Louis Dureuil
20094eba06
Apply review comments
2024-07-15 12:43:29 +02:00
Louis Dureuil
c35904d6e8
search::federated::ranking_rules -> search::ranking_rules
2024-07-15 08:43:22 +02:00
Louis Dureuil
2cacc448b6
Rename src/search.rs -> src/search/mod.rs
2024-07-15 08:43:21 +02:00
Louis Dureuil
a61b852695
Add tests
2024-07-15 08:43:21 +02:00
Louis Dureuil
3167411e98
Analytics
2024-07-15 08:43:21 +02:00
Louis Dureuil
83d71662aa
Changes to multi_search route
2024-07-15 08:43:21 +02:00
Louis Dureuil
5c323cecc7
search: introduce federated search
2024-07-15 08:43:21 +02:00
