manojks1999
528d9d6d8b
Removed CouldNotUpgrade from error file
2025-01-26 21:04:57 +05:30
meili-bors[bot]
022205af90
Merge #5279
...
Test suite / Tests on ubuntu-20.04 (push) Failing after 0s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 2s
Test suite / Tests on windows-2022 (push) Failing after 22s
Test suite / Run Rustfmt (push) Successful in 2m14s
Test suite / Run Clippy (push) Successful in 5m21s
Run the indexing fuzzer / Setup the action (push) Successful in 1h4m54s
Test suite / Tests on macos-13 (push) Has been cancelled
Indexing bench (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of indexing (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Has been cancelled
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Has been cancelled
5279: Bring back changes from v1.12.7 into main r=dureuill a=Kerollmops
This PR brings back v1.12.7 into main.
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
2025-01-24 11:48:46 +00:00
Louis Dureuil
50280bf02b
Support offline upgrade up to v1.12.7
2025-01-24 12:25:33 +01:00
Clément Renault
9b579069df
Comment the max grant of the bbqueue
...
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-24 12:18:32 +01:00
Louis Dureuil
f5a4a1c8b2
Give more RAM to bbqueue.
...
- bbqueue buffers used to have (5% * 2%) / num_threads
- they now have 5% / num_threads
2025-01-24 12:18:32 +01:00
Kerollmops
5ab4cdb1f3
Reduce the maximum grant possible we can store in the BBQueue
2025-01-24 12:18:32 +01:00
meili-bors[bot]
1f54f07f72
Merge #5264
...
Indexing bench (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of indexing (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 12s
Test suite / Run tests in debug (push) Failing after 2s
Test suite / Tests on windows-2022 (push) Failing after 21s
Test suite / Run Rustfmt (push) Failing after 8s
Test suite / Run Clippy (push) Successful in 6m30s
Run the indexing fuzzer / Setup the action (push) Successful in 1h5m21s
Test suite / Tests on macos-13 (push) Has been cancelled
5264: Dumpless upgrade r=dureuill a=irevoire
# Pull Request
Usage: https://meilisearch.notion.site/Dumpless-upgrade-fff4b06b651f81f1acafe24d4687b3f7?pvs=74
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/5162
## What does this PR do?
- Implement the dumpless upgrade with multiple hooks:
- In meilisearch directly before the task queue has been opened
- In the index-scheduler while processing the task
- In milli while upgrading the indexes
- There is no hook at search/query time to handle the old version of a database. That's left to the next person upgrading a database
- A new special type of task (`upgradeDatabase`) that can be retried has been introduced
- A new experimental cli flag has been introduced
- The version has been upgraded to the v1.13.0 in this PR otherwise it was a lot of useless work to test the dumpless upgrade
- Multiple tests have been introduced
## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Update the issue template we use for features, mentioning what we should do in case of a database upgrade
- [ ] The experimental feature discussion should be opened and updated in the PR
- [ ] Update the PRD
- [ ] Add the new error codes
- [ ] Add the task details
- [ ] Add the telemetry
## Notes
The new tests introduced are not _that_ slow
Co-authored-by: Tamo <tamo@meilisearch.com>
2025-01-23 16:22:37 +00:00
Louis Dureuil
73d8a4eace
Remove db.snapshot
2025-01-23 17:21:42 +01:00
Louis Dureuil
c1e5897076
Do not assume v1.12 when there is no index-scheduler version
2025-01-23 17:16:53 +01:00
Louis Dureuil
718a98fbbf
remove : char from filenames
2025-01-23 17:08:35 +01:00
Louis Dureuil
86bf231d29
Change to meilitool after rebase
2025-01-23 16:59:32 +01:00
Louis Dureuil
182c3f4b80
Write assumed version to the index-scheduler version db when it is missing
2025-01-23 16:51:25 +01:00
Tamo
c1eba66443
introduce a corruption in the v1.12 data.ms field distribution
2025-01-23 16:51:24 +01:00
Tamo
7197ced673
fix the bad index version on opening
2025-01-23 16:51:24 +01:00
Tamo
4f21ee6c66
update the data.ms snapshot
2025-01-23 16:51:24 +01:00
Tamo
787472453d
write the version of the index while upgrading it
2025-01-23 16:51:24 +01:00
Tamo
8f65f35de9
rewrite part of the index-scheduler upgrade test
2025-01-23 16:51:23 +01:00
Tamo
c27c923439
introduce a trait to upgrade the indexes
2025-01-23 16:51:23 +01:00
Tamo
fd5649091d
add the upgradeTo field in the details
2025-01-23 16:51:23 +01:00
Tamo
9a57736773
fix the early exit when rewriting a batch
2025-01-23 16:51:23 +01:00
Tamo
7740997ea8
reintroduce the unrecoverable error and use it where its supposed to be used
2025-01-23 16:51:22 +01:00
Tamo
7eb23f73ba
add the version to the index-scheduler snapshots + fix a bug when opening an index scheduler for the first time
2025-01-23 16:51:22 +01:00
Tamo
b9e9fc376a
add the version in the index-scheduler
2025-01-23 16:51:22 +01:00
Tamo
27bf2f1298
remove the empty progress made for the upgrade database
2025-01-23 16:51:22 +01:00
Tamo
d4d82fbd0c
commit the index wtxn before the index-scheduler wtxn
2025-01-23 16:51:21 +01:00
Tamo
eda09a54da
improve the index-scheduler tests
2025-01-23 16:51:21 +01:00
Tamo
b132d70413
fix the details in all cases
2025-01-23 16:51:21 +01:00
Tamo
e41ebd3047
expose the number of database in the index-scheduler and rewrite the lib.rs to use the value provided in the options instead of a magic number
2025-01-23 16:51:21 +01:00
Tamo
705d31e8bd
apply all the comments changes
2025-01-23 16:51:21 +01:00
Tamo
7d95950ce6
fix warning
2025-01-23 16:51:21 +01:00
Tamo
c6b4c21c23
update the snapshots after the rebase
2025-01-23 16:51:20 +01:00
Tamo
bf96fdb858
update the cli url
2025-01-23 16:51:20 +01:00
Tamo
41eeffd88d
fmt
2025-01-23 16:51:20 +01:00
Tamo
1eb9fe8562
remove warnings
2025-01-23 16:51:20 +01:00
Tamo
bac7a1623a
fix the upgrade test
2025-01-23 16:51:19 +01:00
Tamo
5458850d21
write a test ensuring the index-scheduler is effectively down when the upgrade task fail and try to process it when it restarts. There is a bug when deleting this task
2025-01-23 16:51:19 +01:00
Tamo
20ac59c946
fix the field distribution when upgrading from the v1_12
2025-01-23 16:51:19 +01:00
Tamo
cfc1e193b6
update the test with the stats
2025-01-23 16:51:19 +01:00
Tamo
0cc25c7e4c
add a large test importing a data.ms from the v1.12.0
2025-01-23 16:51:18 +01:00
Tamo
102681e384
starts adding tests and fix the starts of meilisearch
2025-01-23 16:51:18 +01:00
Tamo
3ef7a478cd
move the version check to the task queue
2025-01-23 16:48:32 +01:00
Tamo
e70ac35e02
fix bugs after rebase
2025-01-23 16:48:32 +01:00
Tamo
d3654906bf
Add the new tasks with most of the job done
2025-01-23 16:48:32 +01:00
meili-bors[bot]
b15de68831
Merge #5257
...
Indexing bench (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of indexing (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for geo (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for songs (push) / Run and upload benchmarks (push) Waiting to run
Benchmarks of search for Wikipedia articles (push) / Run and upload benchmarks (push) Waiting to run
Test suite / Tests on ubuntu-20.04 (push) Failing after 0s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 11s
Test suite / Run Clippy (push) Failing after 11s
Test suite / Tests on windows-2022 (push) Failing after 43s
Test suite / Run Rustfmt (push) Successful in 2m21s
Run the indexing fuzzer / Setup the action (push) Successful in 1h5m6s
Test suite / Tests on macos-13 (push) Has been cancelled
5257: Fix ollama r=Kerollmops a=dureuill
Fix oversight in ollama embedder
WIP Integration tests are on branch `ollama-integration-test` and will be added as a future PR.
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
2025-01-22 15:36:15 +00:00
meili-bors[bot]
6723700fb9
Merge #5262
...
5262: Bring back changes from v1.12.4, v1.12.5, and v1.12.6 into main r=dureuill a=Kerollmops
This PR follows [this guideline to bring back changes after we worked on v1.12.4, v1.12.5, and v1.12.6](https://github.com/meilisearch/engine-team/blob/main/resources/meilisearch-release.md#after-the-release-bring-back-changes-to-main ).
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Kerollmops <Kerollmops@users.noreply.github.com>
2025-01-22 14:55:02 +00:00
Louis Dureuil
2c099b7c23
Update Cargo.lock again
2025-01-22 15:53:52 +01:00
Louis Dureuil
50fca8fc70
Create update files in new format
2025-01-22 15:51:21 +01:00
Kerollmops
b9d92c481b
Update version for the next release (v1.12.6) in Cargo.toml
2025-01-22 15:51:20 +01:00
Kerollmops
d142c5e432
Do not panic when the facet string is not found
2025-01-22 15:50:43 +01:00
Kerollmops
4d4683adb6
Add a test to check the facet casing is good
2025-01-22 15:50:42 +01:00
