Tamo
867e6a8f1d
rename the send_progress field to progress since it s not sending anything
2024-12-11 16:25:01 +01:00
Tamo
6f4823fc97
make the number of document in the document tasks more incremental
2024-12-11 16:25:01 +01:00
Tamo
df9b68f8ed
inital implementation of the progress
2024-12-11 16:25:01 +01:00
meili-bors[bot]
5bc6391700
Merge #5153
...
5153: Return docid in case of errors while rendering the document template r=Kerollmops a=dureuill
Improves error message:
Before:
```
ERROR index_scheduler: Batch failed Index `mieli`: user error: missing field in document: liquid: Unknown index
with:
variable=doc
requested index=title
available indexes=by, id, kids, parent, text, time, type
```
After:
```
ERROR index_scheduler: Batch failed Index `mieli`: user error: missing field in document `11345147`: liquid: Unknown index
with:
variable=doc
requested index=title
available indexes=by, id, kids, parent, text, time, type
```
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-11 15:01:40 +00:00
Kerollmops
eaa897d983
Avoid compiling when unecessary
2024-12-11 15:57:16 +01:00
Louis Dureuil
bfca54cc2c
Return docid in case of errors while rendering the document template
2024-12-11 15:26:18 +01:00
Kerollmops
04a62d2b97
Compile Meilisearch or run the dedicated binary file
2024-12-11 14:57:07 +01:00
meili-bors[bot]
8c19cb0a0b
Merge #5146
...
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 19s
Test suite / Run tests in debug (push) Failing after 14s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 23s
Test suite / Run Rustfmt (push) Successful in 4m52s
Test suite / Run Clippy (push) Failing after 8m9s
5146: Offline upgrade v1.12 r=irevoire a=ManyTheFish
# Pull Request
## Related issue
Fixes #4978
## What does this PR do?
- add v1_11_to_v1_12 function to upgrade Meilisearch from v1.11 to v1.12
- Convert the update files from OBKV to ndjson format
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Many the fish <many@meilisearch.com>
2024-12-11 13:39:14 +00:00
Many the fish
5c492031d9
Update crates/meilitool/src/upgrade/v1_12.rs
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-11 14:34:18 +01:00
meili-bors[bot]
fb1caa4724
Merge #5148
...
5148: Do not duplicate NDJson data when unecessary r=dureuill a=Kerollmops
This PR improves the NDJSON support. Usually, we save all of the user's document content into a temporary file, validate its content, and then convert everything into NDJSON in the file store (update files in the tasks).
It is a waste of time when users are already sending NDJSON. So, this PR removes the last copy and directly stores the user content in the file store, validating it from the file store. If an issue arises, the file will not persist and will be dropped/deleted instead.
Related to #5078 .
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-11 13:00:50 +00:00
Kerollmops
5622b9607d
Wrap the read NDJSON pass into a tokio blocking
2024-12-11 12:18:36 +01:00
Kerollmops
01bcc601be
Use a nonrandom hasher when decoding JSON
2024-12-11 12:04:29 +01:00
Kerollmops
93fbdc06d3
Use a nonrandom hasher when decoding NDJSON
2024-12-11 12:03:09 +01:00
Kerollmops
69c931334f
Fix the error messages categorization with invalid NDJson
2024-12-11 12:02:48 +01:00
Kerollmops
d683f5980c
Do not duplicate NDJson when unecessary
2024-12-11 12:02:48 +01:00
meili-bors[bot]
f8ba112f66
Merge #5150
...
5150: Reintroduce the Document Addition Logs r=dureuill a=Kerollmops
This PR reintroduces lost tracing logs showing some information about the number of indexed documents.
Related to #5078 . Resolves [this comment](https://github.com/meilisearch/meilisearch/pull/4900/files?show-deleted-files=true&show-viewed-files=true&file-filters%5B%5D=#r1852158338 ) and [this other one](https://github.com/meilisearch/meilisearch/pull/4900/files?show-deleted-files=true&show-viewed-files=true&file-filters%5B%5D=#r1852159073 ).
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-11 10:48:48 +00:00
ManyTheFish
c614d0dd35
Add context when returning an error
2024-12-11 10:55:39 +01:00
ManyTheFish
479607e5dd
Convert update files from OBKV to ndjson
2024-12-11 10:55:39 +01:00
Kerollmops
bb00e70087
Reintroduce the document addition logs
2024-12-11 10:39:04 +01:00
meili-bors[bot]
e974be9518
Merge #5145
...
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 11s
Test suite / Run tests in debug (push) Failing after 9s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 22s
Test suite / Run Rustfmt (push) Successful in 1m18s
Test suite / Run Clippy (push) Successful in 5m32s
5145: Use bumparaw-collections in Meilisearch/milli r=dureuill a=Kerollmops
This PR is related to #5078 . It uses the now published bumparaw-collections and (soon) makes the `RawMap` hasher nonrandom.
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-10 15:51:01 +00:00
Kerollmops
aeb6b74725
Make sure we use an FxHashBuilder on the Value
2024-12-10 15:52:22 +01:00
Kerollmops
a751972c57
Prefer using a stable than a random hash builder
2024-12-10 14:25:53 +01:00
Kerollmops
6b269795d2
Update bumparaw-collections to 0.1.2
2024-12-10 14:25:13 +01:00
Louis Dureuil
d075be798a
Fix tests
2024-12-10 13:39:07 +01:00
Kerollmops
89637bcaaf
Use bumparaw-collections in Meilisearch/milli
2024-12-10 11:52:20 +01:00
Louis Dureuil
866ac91be3
Fix error messages
2024-12-10 11:06:58 +01:00
Louis Dureuil
e610af36aa
User failure for documents with docid of ==512 bytes
2024-12-10 11:06:24 +01:00
Louis Dureuil
7cf6707ed3
Extend test to add the ==512 bytes case
2024-12-10 11:05:42 +01:00
Kushal Kumar
34254b42b6
refactor: use test configuration on import
...
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
2024-12-10 00:00:43 +05:30
meili-bors[bot]
1995040846
Merge #5142
...
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 10s
Test suite / Run tests in debug (push) Failing after 11s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 22s
Test suite / Run Rustfmt (push) Successful in 1m19s
Test suite / Run Clippy (push) Successful in 5m49s
5142: Try merge optimisation r=dureuill a=ManyTheFish
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-12-09 14:48:26 +00:00
ManyTheFish
07f42e8057
Do not index a filed count when no word is counted
2024-12-09 15:45:12 +01:00
ManyTheFish
71f59749dc
Reduce union impact in merging
2024-12-09 15:44:06 +01:00
meili-bors[bot]
3b0b9967f6
Merge #5141
...
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 16s
Test suite / Run tests in debug (push) Failing after 14s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 44s
Test suite / Run Rustfmt (push) Successful in 9m52s
Test suite / Run Clippy (push) Successful in 1h2m24s
5141: Use the right amount of max memory and not impact the settings r=curquiza a=Kerollmops
Fixes #5132 . Related to #5125 .
Co-authored-by: Kerollmops <clement@meilisearch.com>
2024-12-09 10:40:46 +00:00
meili-bors[bot]
123b54a178
Merge #5056
...
5056: Attach index name in error message r=irevoire a=airycanon
# Pull Request
## Related issue
Fixes #4392
## What does this PR do?
- ...
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: airycanon <airycanon@airycanon.me>
2024-12-09 09:59:12 +00:00
Kerollmops
f5dd8dfc3e
Rollback max memory usage changes
2024-12-09 10:26:30 +01:00
Kerollmops
bcfed70888
Revert "Merge #5125 "
...
This reverts commit 9a9383643fcac355bfa7
2024-12-09 10:08:02 +01:00
meili-bors[bot]
503ef3bbc9
Merge #5138
...
5138: Allow xtask bench to proceed without a commit message r=Kerollmops a=dureuill
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-09 09:00:12 +00:00
Louis Dureuil
08f2c696b0
Allow xtask bench to proceed without a commit message
2024-12-09 09:36:59 +01:00
James Hiew
54e34beac6
Check attributes are filterable before evaluating search query
2024-12-07 21:13:13 +00:00
Kushal Kumar
c0aa018c87
tests: split test in separate file
...
Signed-off-by: Kushal Kumar <kushalkumargupta4@gmail.com>
2024-12-08 00:32:32 +05:30
airycanon
b75f1f4c17
fix tests
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/index-scheduler/src/snapshots/lib.rs/fail_in_process_batch_for_document_deletion/after_removing_the_documents.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fifth_task_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_bad_primary_key/fourth_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key/third_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_multiple_primary_key_batch_wrong_key/second_and_third_tasks_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/all_other_tasks_succeeds.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/second_task_fails.snap
# crates/index-scheduler/src/snapshots/lib.rs/test_document_addition_with_set_and_null_primary_key_inference_works/third_task_succeeds.snap
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
# crates/meilisearch/tests/vector/mod.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
2024-12-06 02:03:02 +08:00
airycanon
95ed079761
attach index name in errors
...
# Conflicts:
# crates/index-scheduler/src/batch.rs
# Conflicts:
# crates/index-scheduler/src/batch.rs
# crates/meilisearch/src/search/mod.rs
2024-12-06 01:12:13 +08:00
meili-bors[bot]
4a082683df
Merge #5131
...
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 21s
Test suite / Tests on ubuntu-20.04 (push) Failing after 10s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 10s
Test suite / Run Rustfmt (push) Successful in 1m25s
Test suite / Run Clippy (push) Successful in 5m54s
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Has been cancelled
5131: Ignore documents whose selected fields didn't change r=dureuill a=dureuill
Attempts to improve the new indexer performance by ignoring documents whose selected fields didn't change:
- Add `Update::has_changed_for_fields` function
- Ignore documents whose searchable attributes didn't change for word docids and word pair proximity extraction
- Ignore documents whose faceted attributes didn't change for facet extraction
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2024-12-05 16:04:16 +00:00
meili-bors[bot]
26be5e0733
Merge #5123
...
5123: Fix batch details r=dureuill a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5079
Fixes https://github.com/meilisearch/meilisearch/issues/5112
## What does this PR do?
- Make the processing tasks actually processing in the stats of the batch instead of enqueued
- Stop counting one extra task for all non-prioritized batches in the stats
- Add a test
Co-authored-by: Tamo <tamo@meilisearch.com>
2024-12-05 15:21:55 +00:00
Louis Dureuil
bd5110a2fe
Fix clippy warnings
2024-12-05 16:13:07 +01:00
Louis Dureuil
fa8b9acdf6
Ignore documents that didn't change in facets
2024-12-05 16:12:52 +01:00
Louis Dureuil
2b74d1824b
Ignore documents that didn't change any field in word pair proximity
2024-12-05 15:56:22 +01:00
Louis Dureuil
c77b00d3ac
Don't extract word docids when no searchable changed
2024-12-05 15:51:58 +01:00
Louis Dureuil
c77073efcc
Update::has_changed_for_fields
2024-12-05 15:50:12 +01:00
meili-bors[bot]
1537323eb9
Merge #5119
...
5119: Settings opt out error msg r=Kerollmops a=ManyTheFish
# Pull Request
## Related issue
PRD: https://meilisearch.notion.site/API-usage-Settings-to-opt-out-indexing-features-fff4b06b651f8108ade3f858aeb16b14?pvs=4
## What does this PR do?
Add a new error code and message when the user tries a facet search on an index where the facet search is disabled:
```json
{
"message": "The facet search is disabled for this index",
"code": "facet_search_disabled",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#invalid_facet_search_disabled "
}
```
Co-authored-by: ManyTheFish <many@meilisearch.com>
2024-12-05 13:51:11 +00:00
