10125 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Tamo	71807cac6d	makes clippy happy	2024-11-20 17:40:58 +01:00
Tamo	21a2264782	improve the details and stats of the current batch processing	2024-11-20 17:25:55 +01:00
meili-bors[bot]	d4d8becfa7	Merge #5060 ... 5060: Batch route r=Kerollmops a=irevoire # Pull Request See [usage](https://www.notion.so/meilisearch/Enhance-visibility-on-batched-tasks-1194b06b651f810b8fe0fab5d72846a8). ## Related issue Fixes https://github.com/meilisearch/meilisearch/issues/4977 ## What does this PR do? - For more detailed information, see the PRD. - Added a `batchUid` to the tasks (that's the cause of all the updates of the dumps): - For all enqueued tasks, it's set to `None` - For every other tasks it must be set to something - ⚠️ For all the tasks imported in a dump, the `batchUid` will be set to `None` as well. - Add two new routes: - `GET /batches/:uid` - to query a batch by its id - `GET /batches` - to retrieve a list of batches. It accepts all the same query parameters that are available on the `GET /tasks` route - Adds new databases to query the batches directly: - When doing a query against the batches, the rule of thumb is that we want to return a batch iif **at least one** task in it matches the provided filter. - We don't need a `canceledBy` batch specific database because we can just retrieve the task and if it's a `taskCancelation` retrieve its `batchUid` - The task cancelation has been updated and simplified a bit: - Instead of updating the matching tasks on disk while processing the cancelation task, we instead retrieve the task and let the `tick` function do the work afterward. - In the `tick` function, we now have to take care of not missing any tasks - All the tests applied to the tasks were duplicated and updated to works with the new batches routes - The deletion of batches doesn't contain any tests because it's already tested in the deletion of tasks (and especially highlighted in the snapshots) Currently, one part of the PRD is not implemented: it's the progress. Co-authored-by: Tamo <tamo@meilisearch.com>	2024-11-20 14:07:48 +00:00
Tamo	ec06879d28	apply review changes	2024-11-20 14:40:36 +01:00
Tamo	83d1f858c1	Update crates/index-scheduler/src/lib.rs ... Co-authored-by: Clément Renault <clement@meilisearch.com>	2024-11-20 14:36:05 +01:00
Tamo	a7ac590e9e	implements the reverse query parameter for the batches	2024-11-20 13:29:52 +01:00
Tamo	8ad68dd708	stop leaking the update files of the canceled tasks	2024-11-20 13:17:54 +01:00
Tamo	7e379b3d14	remove useless prints	2024-11-20 12:27:12 +01:00
Tamo	56eacd221f	update the tests after the rebase	2024-11-20 10:54:38 +01:00
Tamo	bdb51a85fe	now that the task cancelation shares their started at with all the tasks of their batch we don't need the trick of retrieving the previous batch anymore	2024-11-20 10:51:07 +01:00
Tamo	b24a34830d	fix the dump test -> the only change is that we now have a null batch_uid in all the tasks	2024-11-20 10:51:06 +01:00
Tamo	e145d71a62	implements the two last TODOs	2024-11-20 10:51:06 +01:00
Tamo	d9a4e69990	push a missing snapshot	2024-11-20 10:51:06 +01:00
Tamo	b906e3ed70	improve the way we access the mutex	2024-11-20 10:51:06 +01:00
Tamo	4abcd9c04e	add some stats on the batches	2024-11-20 10:51:06 +01:00
Tamo	229fa0f902	implements the batch details	2024-11-20 10:51:06 +01:00
Tamo	5d10c2312b	remove unused file	2024-11-20 10:51:06 +01:00
Tamo	f1d38581e5	add the front end tests on the batches routes	2024-11-20 10:51:06 +01:00
Tamo	62646af7b9	implements the automatic batch deletion	2024-11-20 10:51:06 +01:00
Tamo	1fcb9526f5	fix the task cancelation	2024-11-20 10:51:06 +01:00
Tamo	15eefa4fcc	fixes a lot of small issue, the test about the cancellation is still failing	2024-11-20 10:51:05 +01:00
Tamo	ad9763ffcd	copy multiple task query tests to batches. Currently, they fails	2024-11-20 10:49:25 +01:00
Tamo	d489f5635f	add the mapping between the task and batches	2024-11-20 10:49:23 +01:00
Tamo	a1251c3c83	Implements the get all batches route with filters working	2024-11-20 10:42:55 +01:00
Tamo	6062914654	add the batch_id to the tasks	2024-11-20 10:42:54 +01:00
Lukas Kalbertodt	057fcb3993	Add indices field to _matchesPosition to specify where in an array a match comes from (#5005) ... * Remove unreachable code * Add `indices` field to `MatchBounds` For matches inside arrays, this field holds the indices of the array elements that matched. For example, searching for `cat` inside `{ "a": ["dog", "cat", "fox"] }` would return `indices: [1]`. For nested arrays, this contains multiple indices, starting with the one for the top-most array. For matches in fields without arrays, `indices` is not serialized (does not exist) to save space.	2024-11-20 01:00:43 +01:00
meili-bors[bot]	c1d8ee2a8d	Merge #5048 ... 5048: Reverse the order of the task queue r=Kerollmops a=irevoire # Pull Request ## Related issue Fixes https://github.com/meilisearch/meilisearch/issues/5047 ## What does this PR do? - Provide a new parameter to reverse the order of the task queue - Add tests - Remove some unrelated tests that were duplicated in tests/tasks/mod.rs and tests/tasks/error.rs Co-authored-by: Tamo <tamo@meilisearch.com>	2024-11-18 16:24:12 +00:00
meili-bors[bot]	94fb55bb6f	Merge #5049 ... 5049: Fix the path used in the flaky tests CI r=irevoire a=Kerollmops This PR fixes [the flaky tests CI](https://github.com/meilisearch/meilisearch/actions/runs/11741717787) path used. Co-authored-by: Clément Renault <clement@meilisearch.com>	2024-11-13 10:26:50 +00:00
Clément Renault	009709eace	Fix the path used in the flaky tests CI	2024-11-13 09:52:10 +01:00
Tamo	2eb1801e85	reverse the order of the task queue	2024-11-07 19:19:44 +01:00
meili-bors[bot]	a5d7ae23bd	Merge #5044 ... 5044: Adds new metrics to prometheus r=irevoire a=PedroTurik not 100% confident in this solution, especially because i couldn't make the "Search Queue searches waiting" metric give me any value other than 0 with my local testing 😆. But i believe it solves the Issue. # Pull Request ## Related issue Fixes #4998 ## What does this PR do? ### Adds new metrics to prometheus; - SearchQueue size, - SearchQueue searches running, - and Search Queue searches waiting. ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Co-authored-by: Pedro Turik Firmino <pedroturik@gmail.com>	2024-11-07 17:05:43 +00:00
PedroTurik	03886d0012	Applies optimizations to formatted integration tests (#5043)	2024-11-07 15:58:55 +01:00
meili-bors[bot]	b427b9e88f	Merge #5025 ... 5025: test: improve performance of get_documents.rs r=irevoire a=PedroTurik # Pull Request ## Related issue Fixes one item from #4840 ## What does this PR do? - Applies the changes recommended on the issue for `meilisearch/tests/documents/get_documents.rs` ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Thank you so much for contributing to Meilisearch! Co-authored-by: Pedro Turik Firmino <pedroturik@gmail.com>	2024-11-07 09:46:34 +00:00
Pedro Turik Firmino	8b95f5ccc6	Adds new metrics to prometheus: SearchQueue size, SearchQueue searches running, and Search Queue searches waiting.	2024-11-06 15:37:16 -03:00
meili-bors[bot]	6b67f9fc4c	Merge #5030 ... 5030: Bump Swatinem/rust-cache from 2.7.1 to 2.7.5 r=curquiza a=dependabot[bot] Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.7.1 to 2.7.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/swatinem/rust-cache/releases">Swatinem/rust-cache's releases</a>.</em></p> <blockquote> <h2>v2.7.5</h2> <h2>What's Changed</h2> <ul> <li>Upgrade checkout action from version 3 to 4 by <a href="https://github.com/carsten-wenderdel"><code>`@​carsten-wenderdel</code></a>` in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/190">Swatinem/rust-cache#190</a></li> <li>fix: usage of <code>deprecated</code> version of <code>node</code> by <a href="https://github.com/hamirmahal"><code>`@​hamirmahal</code></a>` in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/197">Swatinem/rust-cache#197</a></li> <li>Only run macOsWorkaround() on macOS by <a href="https://github.com/heksesang"><code>`@​heksesang</code></a>` in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/206">Swatinem/rust-cache#206</a></li> <li>Support Cargo.lock format cargo-lock v4 by <a href="https://github.com/NobodyXu"><code>`@​NobodyXu</code></a>` in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/211">Swatinem/rust-cache#211</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/carsten-wenderdel"><code>`@​carsten-wenderdel</code></a>` made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/190">Swatinem/rust-cache#190</a></li> <li><a href="https://github.com/hamirmahal"><code>`@​hamirmahal</code></a>` made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/197">Swatinem/rust-cache#197</a></li> <li><a href="https://github.com/heksesang"><code>`@​heksesang</code></a>` made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/206">Swatinem/rust-cache#206</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/Swatinem/rust-cache/compare/v2.7.3...v2.7.5">https://github.com/Swatinem/rust-cache/compare/v2.7.3...v2.7.5</a></p> <h2>v2.7.3</h2> <ul> <li>Work around upstream problem that causes cache saving to hang for minutes.</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/Swatinem/rust-cache/compare/v2.7.2...v2.7.3">https://github.com/Swatinem/rust-cache/compare/v2.7.2...v2.7.3</a></p> <h2>v2.7.2</h2> <h2>What's Changed</h2> <ul> <li>Update action runtime to <code>node20</code> by <a href="https://github.com/rhysd"><code>`@​rhysd</code></a>` in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/175">Swatinem/rust-cache#175</a></li> <li>Only key by <code>Cargo.toml</code> and <code>Cargo.lock</code> files of workspace members by <a href="https://github.com/max-heller"><code>`@​max-heller</code></a>` in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/180">Swatinem/rust-cache#180</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/rhysd"><code>`@​rhysd</code></a>` made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/175">Swatinem/rust-cache#175</a></li> <li><a href="https://github.com/max-heller"><code>`@​max-heller</code></a>` made their first contribution in <a href="https://redirect.github.com/Swatinem/rust-cache/pull/180">Swatinem/rust-cache#180</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/Swatinem/rust-cache/compare/v2.7.1...v2.7.2">https://github.com/Swatinem/rust-cache/compare/v2.7.1...v2.7.2</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Swatinem/rust-cache/blob/master/CHANGELOG.md">Swatinem/rust-cache's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h2>2.7.3</h2> <ul> <li>Work around upstream problem that causes cache saving to hang for minutes.</li> </ul> <h2>2.7.2</h2> <ul> <li>Only key by <code>Cargo.toml</code> and <code>Cargo.lock</code> files of workspace members.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="82a92a6e8f"><code>82a92a6</code></a> 2.7.5</li> <li><a href="598fe25fa1"><code>598fe25</code></a> update dependencies, rebuild</li> <li><a href="8f842c2d45"><code>8f842c2</code></a> Support Cargo.lock format cargo-lock v4 (<a href="https://redirect.github.com/swatinem/rust-cache/issues/211">#211</a>)</li> <li><a href="96a8d65dba"><code>96a8d65</code></a> Only run macOsWorkaround() on macOS (<a href="https://redirect.github.com/swatinem/rust-cache/issues/206">#206</a>)</li> <li><a href="9bdad043e8"><code>9bdad04</code></a> fix: usage of <code>deprecated</code> version of <code>node</code> (<a href="https://redirect.github.com/swatinem/rust-cache/issues/197">#197</a>)</li> <li><a href="f7a52f6914"><code>f7a52f6</code></a> &quot;add jsonpath test&quot;</li> <li><a href="2bceda3912"><code>2bceda3</code></a> &quot;update dependencies&quot;</li> <li><a href="640a22190e"><code>640a221</code></a> Upgrade checkout action from version 3 to 4 (<a href="https://redirect.github.com/swatinem/rust-cache/issues/190">#190</a>)</li> <li><a href="1582741630"><code>1582741</code></a> update dependencies</li> <li><a href="23bce251a8"><code>23bce25</code></a> 2.7.3</li> <li>Additional commits viewable in <a href="https://github.com/swatinem/rust-cache/compare/v2.7.1...v2.7.5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) You can trigger a rebase of this PR by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-11-06 12:59:36 +00:00
dependabot[bot]	2e4d4b398d	Bump Swatinem/rust-cache from 2.7.1 to 2.7.5 ... Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.7.1 to 2.7.5. - [Release notes](https://github.com/swatinem/rust-cache/releases) - [Changelog](https://github.com/Swatinem/rust-cache/blob/master/CHANGELOG.md) - [Commits](https://github.com/swatinem/rust-cache/compare/v2.7.1...v2.7.5) --- updated-dependencies: - dependency-name: Swatinem/rust-cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2024-11-06 12:57:04 +00:00
Pedro Turik Firmino	da59a043ba	Fixes formatting issues	2024-11-06 09:55:48 -03:00
Pedro Turik Firmino	da4d47b5d0	Fixes formatting issues	2024-11-06 09:54:20 -03:00
meili-bors[bot]	0507f5d99b	Merge #4928 ... 4928: Make matches consider phrases as a single `Match` r=ManyTheFish a=flevi29 # Pull Request ## Related issue Fixes #4732 ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Thank you so much for contributing to Meilisearch! Co-authored-by: F. Levi <55688616+flevi29@users.noreply.github.com>	2024-11-06 08:22:01 +00:00
meili-bors[bot]	be2a7c70f2	Merge #5037 ... 5037: Fix the benchmarks r=Kerollmops a=irevoire # Pull Request ## Related issue https://github.com/meilisearch/meilisearch/pull/5016 broke all benchmarks. This PR fix the benchmarks Co-authored-by: Tamo <tamo@meilisearch.com>	2024-11-05 15:37:55 +00:00
meili-bors[bot]	ede086bc30	Merge #5034 ... 5034: Upgrade from v1 10 to v1 11 r=irevoire a=irevoire # Pull Request ## Related issue Parts of https://github.com/meilisearch/meilisearch/issues/4978 ## What does this PR do? - Move the code around the offline upgrade to its own module with a file per version - Fix the upgrade from v1.9 to v1.10 because I couldn’t make it work anymore. It now uses a specified format instead of relying on cargo to get the right set of feature - ☝️ must be checked against docker - Provide an update path from v1.10 to v1.11. Most of the code is boilerplate in meilitool, the real code is located here: 053807bf38/src/lib.rs (L161-L269) Co-authored-by: Tamo <tamo@meilisearch.com>	2024-11-05 14:49:56 +00:00
Tamo	7415ef7ff5	Update crates/meilitool/src/upgrade/v1_11.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2024-11-05 15:37:59 +01:00
Tamo	a5d138ac34	use a tag while importing arroy instead of a loose branch or rev	2024-11-05 15:24:02 +01:00
Tamo	0f74a93346	Update crates/meilitool/src/upgrade/v1_11.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2024-11-05 15:14:02 +01:00
Tamo	e4993aa705	Update crates/meilitool/src/upgrade/mod.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2024-11-05 15:13:50 +01:00
Tamo	66b7e0824e	Update crates/meilitool/src/upgrade/mod.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2024-11-05 15:13:40 +01:00
Tamo	f193c3a67c	Update crates/meilitool/src/main.rs ... Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2024-11-05 15:13:32 +01:00
Tamo	9799812b27	fix the benchmarks	2024-11-05 15:08:01 +01:00
Tamo	48ab898ca2	fix the datetime of v1.9	2024-11-05 10:30:53 +01:00
F. Levi	a5dc783ffa	Merge with main branch	2024-11-05 10:56:17 +02:00