Clément Renault
1d59c19cd2
Send the WordsFst by using an Mmap
2024-09-04 14:30:09 +02:00
Clément Renault
98e48371c3
Factorize some stuff
2024-09-04 12:17:13 +02:00
Clément Renault
6d74fb0229
Introduce the WordFidWordDocids database
2024-09-04 11:40:55 +02:00
ManyTheFish
1eb75a1040
remove milli/src/update/new/extract/tokenize_document.rs
2024-09-04 11:40:26 +02:00
Clément Renault
3b82d8b5b9
Fix the cache to serialize entries correctly
2024-09-04 10:55:36 +02:00
ManyTheFish
781a186f75
remove milli/src/update/new/extract/extract_word_docids.rs
2024-09-04 10:28:31 +02:00
ManyTheFish
6a399556b5
Implement more searchable extractor
2024-09-04 10:20:18 +02:00
Clément Renault
27b4cab857
Extract and write the documents and words fst in the database
2024-09-04 09:59:19 +02:00
dependabot[bot]
3f3cebf5f9
Bump quinn-proto from 0.11.3 to 0.11.8
...
Bumps [quinn-proto](https://github.com/quinn-rs/quinn ) from 0.11.3 to 0.11.8.
- [Release notes](https://github.com/quinn-rs/quinn/releases )
- [Commits](https://github.com/quinn-rs/quinn/compare/quinn-proto-0.11.3...quinn-proto-0.11.8 )
---
updated-dependencies:
- dependency-name: quinn-proto
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-09-03 20:50:30 +00:00
meili-bors[bot]
b278815617
Merge #4908
...
4908: Bring back changes from release v1.10.1 to main r=dureuill a=irevoire
# Pull Request
Following the [latest release](https://github.com/meilisearch/meilisearch/releases/tag/v1.10.1 ), this PR brings back the changes to main.
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: irevoire <irevoire@users.noreply.github.com>
2024-09-03 14:28:12 +00:00
Clément Renault
52d32b4ee9
Move the channel sender in the closure to stop the merger thread
2024-09-03 16:08:33 +02:00
ManyTheFish
da61408e52
Remove unimplemented from document changes
2024-09-03 15:14:16 +02:00
ManyTheFish
fe69385bd7
Fix tokenizer test
2024-09-03 14:24:37 +02:00
meili-bors[bot]
40e13ceef3
Merge #4892
...
4892: Add a documentTemplateMaxBytes parameter to limit the max length of document templates r=ManyTheFish a=dureuill
# Pull Request
## Related issue
Fixes #4885
See [public usage](https://meilisearch.notion.site/v1-11-AI-search-changes-0e37727193884a70999f254fa953ce6e#a3d63628129e40adba943ae7b8ec06c2 )
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-09-03 11:50:07 +00:00
Louis Dureuil
18a2c13e4e
add analytics
2024-09-03 12:07:59 +02:00
Louis Dureuil
ed19b7c3c3
Only reindex if the size increased
2024-09-03 12:07:59 +02:00
Louis Dureuil
66bda2ce8a
fix tests
2024-09-03 12:07:58 +02:00
Louis Dureuil
1ac008926b
Add maxBytes parameter
2024-09-03 12:07:15 +02:00
Louis Dureuil
c49d892c82
Changes to prompt
2024-09-03 12:07:10 +02:00
Louis Dureuil
de962a26f3
New error type when maxBytes is null
2024-09-03 12:01:04 +02:00
Clément Renault
c1557734dc
Use the GlobalFieldsIdsMap everywhere and write it to disk
...
Co-authored-by: Dureuill <louis@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-09-03 12:01:01 +02:00
Tamo
005204e9e5
make the code of init_web_app in common between most tests
2024-09-03 11:40:05 +02:00
Tamo
1040e5e2b4
spawn on search queue per test
2024-09-03 11:20:25 +02:00
ManyTheFish
c50d3edc4a
Integrate first searchable exctrator
2024-09-03 11:02:39 +02:00
meili-bors[bot]
80408c92dc
Merge #4906
...
4906: Add searchable fields to template r=dureuill a=dureuill
# Pull Request
## Related issue
Fixes #4886
See [public usage](https://meilisearch.notion.site/v1-11-AI-search-changes-0e37727193884a70999f254fa953ce6e#1dd6f0eee5a1422888e1c5d48e107cd1 )
## What does this PR do?
- `Prompt::render` now requires and uses metadata to indicate if the fields are searchable or not
- Changes default template
- Updated tests
- Correctly reindex vectors when the list of searchable fields changes in a settings update.
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-09-03 07:14:58 +00:00
Clément Renault
5369bf4a62
Change some lifetimes
2024-09-02 19:51:22 +02:00
Clément Renault
bcb1aa3d22
Find a temporary solution to par into iter on an HashMap
...
Spoiler: Do not use an HashMap but drain it into a Vec
2024-09-02 19:39:48 +02:00
Tamo
fa1a0beb0c
fix conflicts after rebase
2024-09-02 18:15:42 +02:00
Tamo
5aefe7cd17
add the snapshots
2024-09-02 16:27:51 +02:00
Tamo
e6dd66e4a0
Do not fail the whole batch when a single document deletion by filter fails
2024-09-02 16:27:51 +02:00
Tamo
6e3839d8b6
autobatch document deletion by filter
2024-09-02 16:27:51 +02:00
Tamo
cd271b8762
stop trying to process searches after one minute
2024-09-02 16:27:51 +02:00
Tamo
3ce8500d4c
ensure we never early exit when we have a permit and remove the warning when we implicitely drop a permit
2024-09-02 16:27:51 +02:00
Tamo
588000d398
add a warning to help us find when we forget to drop explicitely drop a permit
2024-09-02 16:27:51 +02:00
Tamo
92b151607c
explicitely drop the search permit
2024-09-02 16:27:51 +02:00
irevoire
42e7499260
Update version for the next release (v1.10.1) in Cargo.toml
2024-09-02 16:27:51 +02:00
Tamo
41aa1e1424
Only spawn one search queue in actix-web
2024-09-02 16:27:50 +02:00
Clément Renault
9b7858fb90
Expose the new indexer
2024-09-02 15:21:59 +02:00
Clément Renault
ab01679a8f
Remove the useless option from the document changes
2024-09-02 15:21:00 +02:00
Clément Renault
521775f788
I push for Many
2024-09-02 15:10:21 +02:00
Clément Renault
72e7b7846e
Renaming the indexers
2024-09-02 14:42:27 +02:00
Clément Renault
6526ce1208
Fix the merging of documents
2024-09-02 14:41:20 +02:00
Louis Dureuil
24ace5c381
Add reindexing test
2024-09-02 13:37:01 +02:00
Louis Dureuil
21296190a3
Reindex embedders
2024-09-02 13:00:53 +02:00
Louis Dureuil
03fda78901
update other tests
2024-09-02 11:31:31 +02:00
Louis Dureuil
30a143f149
Test new facilities
2024-09-02 11:31:23 +02:00
Louis Dureuil
4464d319af
Change default template to use the new facility
2024-09-02 11:30:59 +02:00
Louis Dureuil
580ea2f450
Pass the fields <-> ids map with metadata to render
2024-09-02 11:30:10 +02:00
Louis Dureuil
915cf4bae5
Add field.is_searchable property to fields
2024-09-02 11:28:53 +02:00
Clément Renault
e639ec79d1
Move the indexers into their own modules
2024-09-02 10:42:19 +02:00
