4861: Make sure the index scheduler never stops running r=irevoire a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4748
## What does this PR do?
- Whatever happens, we always try to process tasks once every minute (if no tasks are enqueued that's practically free)
Co-authored-by: Tamo <tamo@meilisearch.com>
4839: In prometheus metrics return the route pattern instead of the real route when returning the HTTP requests total r=irevoire a=irevoire
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/4825
## What does this PR do?
- return the route pattern instead of the real route when returning the HTTP requests total
Co-authored-by: Tamo <tamo@meilisearch.com>
4808: Make the tests run faster r=irevoire a=irevoire
## Index-Scheduler
### Only check the consistency of the index-scheduler on snapshots when running in release mode
This saves 12s on the tests, and since the tests run in release mode in the CI, we don't lose any information.
From 28s to 16s
### We were snapshotting the index for no reason in `advance_till`, I removed this call
This saved an additional 8s on the tests, going from 16s to 8s.
----
After these two optimizations, the test suite as a whole executes 14% quicker
## Meilisearch integration tests
While profiling this test suite, nothing stands out. The only noticeable thing is that we're losing most of our time creating and dropping threads.
I made the theory that by sharing a single common instance between all integrations tests I would gain some time again.
In 355a7acd1c I saved another 15s by only testing this theory on the module that tests the error messages.
But we can do it on many more tests. **We must take care of not making any test flaky, though**.
## Use two indexing threads
By moving from one to two indexing threads, we gain an additional 30% in performance.
# Conclusion
## Before
The execution of the test suite was taking around:
- 4m40s on my computer
- 15 minutes on the debug CI with cache
- 29 minutes on the Windows CI with cache
## After
The execution of the test suite is taking around:
- 2m20 on my computer
- 8 minutes on the debug CI with cache
- 29 minutes on the Windows CI with cache
## This means the test suite should now run ~50% faster on your computer; the CI may report errors twice faster, but we'll still wait for ~the same amount of time to merge a PR
Co-authored-by: Tamo <tamo@meilisearch.com>
4830: Use the dtolnay's Rust Toolchain r=dureuill a=Kerollmops
Fixes the CI by using another rust-toolchain GitHub repo.
Note: the [helix-editor/rust-toolchain repository](https://github.com/helix-editor/rust-toolchain) has been deleted so we moved to the [dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain) one. However, the dtolnay's one doesn't support `rust-toolchain.toml` and the version is directly in the rust-toolchain@version. We keep the `rust-toolchain.toml` for local builds only.
Co-authored-by: Clément Renault <clement@meilisearch.com>
4818: Custom headers and QoL improvements r=ManyTheFish a=dureuill
# Pull Request
## Related issue
Fixes#4734
Depends on #4815
## What does this PR do?
- Adds custom headers for rest embedders ([public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#41354652885242c899def07e36a66d49))
- Quality of life: allow specifying `dimensions` for `ollama` embedders ([public usage](https://meilisearch.notion.site/v1-10-AI-search-changes-737c9d7d010d4dd685582bf5dab579e2#37218531431343dab3d2d3a9a1937e9d)). As for `rest` embedders, specifying `dimensions` disables the "test" embedding when the embedder is spawned.
- Improve error message again when indexing documents that don't have a vector for a user-provided vector
1. Remove the contents of the document
2. Display the docid of the first document that triggered the error
3. Indicate how many documents in that chunk suffered from the same issue for that embedder
Co-authored-by: Louis Dureuil <louis@meilisearch.com>
