3036: Bump milli to v0.35.1 r=irevoire a=Kerollmops
This PR bumps milli to v0.35.1 which brings some fixes. You can see [the changelog of milli on the release page](https://github.com/meilisearch/milli/releases/tag/v0.35.1).
Fixes#2905Fixes#3004Fixes#3000Fixes#3021Fixes#2945
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
3023: Update error codes related to tasks cancelation + add canceledBy filter r=Kerollmops a=Kerollmops
<details>This PR changes the error codes [to follow the specification](https://github.com/meilisearch/specifications/pull/195).
- [x] The `missing_filters` error code is renamed `missing_task_filters` to be more accurate and follow the `invalid_task_*` convention.
- [x] The error code `invalid_task_uids_filter` is added.
- [x] The error code `invalid_task_canceled_by_filter` is added.
- [x] The error code `invalid_task_date_filter` is added.
- The error message is the same as for expires_at in the API Key EXCEPT that it does not explicitly mention that a date must be given in the future.
</details>
Edit by `@loiclec` :
I have added a few more changes into this PR. The related issues are:
- Fixes https://github.com/meilisearch/meilisearch/issues/3029
- Implements https://github.com/meilisearch/meilisearch/issues/3026
- Fixes https://github.com/meilisearch/meilisearch/issues/2940
- Fixes https://github.com/meilisearch/meilisearch/issues/2939
Additionally:
- Fixes a bug where global tasks were returned by `GET /tasks` queries even if the user did not have the `index.*` API key action.
- Rename `originalQuery` to `originalFilters`
- Display `error: null` and `canceledBy: null` in the task views
- Allow using the star operator in the task filters in the `DELETE /tasks` and `POST /tasks/cancel` routes
- Make sure that the index scheduler keeps making progress even when a grave error occurs.
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
3032: Fix Index name parsing error message to fit the specification r=Kerollmops a=ManyTheFish
fixes#2924
Co-authored-by: ManyTheFish <many@meilisearch.com>
+ Allow star operator in delete/cancel tasks
+ rename originalQuery to originalFilters
+ Display error/canceled_by in task view even when they are = null
+ Rename task filter fields by using their plural forms
+ Prepare an error code for canceledBy filter
+ Only return global tasks if the API key action `index.*` is there
3022: Store the `started_at` for a task that is canceled when processing r=irevoire a=Kerollmops
This PR changes the current behavior of the displayed tasks. When a processing task is canceled, the `started_at` date time is kept and displayed to the user. Otherwise, if the task was just enqueued, the `started_at` remains `null`. If a task is processing, the engine is ctrl-c, starts again, and the task becomes enqueued again, so if it is canceled, its `started_at` will be `null`.
You can read more [in this discussion](https://github.com/meilisearch/specifications/pull/195/files#r1009602335).
Co-authored-by: Kerollmops <clement@meilisearch.com>
3019: Fix error code of the "duplicate index found" error on the index swap route r=irevoire a=loiclec
According to the spec, the code of the error should be `duplicate_index_found`, but it was `bad_request` instead.
Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
3002: Fix dump import without instance uid r=Kerollmops a=irevoire
When creating a dump without any instance-uid (that can happen if you’ve always run meilisearch with the `--no-analytics` flag), you could get an error when trying to load the dump.
Co-authored-by: Irevoire <tamo@meilisearch.com>
2981: Move index swap error handling from meilisearch-http to index-scheduler r=irevoire a=loiclec
And make index_not_found error asynchronous, since we can't know whether the index will exist by the time the index swap task is processed.
Improve the index-swap test to verify that future tasks are not swapped and to test the new error messages that were introduced.
## Related issue
https://github.com/meilisearch/meilisearch/issues/2973
2996: Get rids of the unecessary tasks when an index_uid is specified r=Kerollmops a=irevoire
Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
2982: Adapt task queries to account for special index swap rules r=irevoire a=loiclec
# Pull Request
## Related issue
Fixes https://github.com/meilisearch/meilisearch/issues/2970
## What does this PR do?
- Replace the `get_tasks` method with a `get_tasks_from_authorized_indexes` which returns the list of tasks matched by the query **from the point of view of the user**. That is, it takes into consideration the list of authorised indexes as well as the special case of `IndexSwap` which should not be returned if an index_uid is specified or if any of its associated indexes are not authorised.
- Adapt the code in other places following this change
- Add some tests
- Also the method `get_task_ids_from_authorized_indexes` now takes a read transaction as argument. This is because we want to make sure that the implementation of `get_tasks_from_authorized_indexes` only uses one read transaction. Otherwise, we could (1) get a list of task ids matching the query, then (2) one of these task ids is deleted by a taskDeletion task, and finally (3) we try to get the `Task`s associated with each returned task ids, and get a `CorruptedTaskQueue` error.
Co-authored-by: Loïc Lecrenier <loic.lecrenier@me.com>
2993: Reconsider the Windows tests r=irevoire a=Kerollmops
This PR removes the `ignore` cfg on top of a lot of our tests. Now that we reworked the index scheduler we can make them pass again!
Fixes#2038, fixes#1966.
Co-authored-by: Clément Renault <clement@meilisearch.com>
2991: Update version for the next release (v0.30.0) in Cargo.toml files r=Kerollmops a=meili-bot
⚠️ This PR is automatically generated. Check the new version is the expected one before merging.
Co-authored-by: curquiza <curquiza@users.noreply.github.com>
2990: isolate the search in another task r=Kerollmops a=irevoire
In case there is a failure on milli's side that should avoid blocking the tokio main thread
Co-authored-by: Irevoire <tamo@meilisearch.com>
And make index_not_found error asynchronous, since we can't know
whether the index will exist by the time the index swap task is
processed.
Improve the index-swap test to verify that future tasks are not swapped
and to test the new error messages that were introduced.
2763: Index scheduler r=Kerollmops a=irevoire
Fix https://github.com/meilisearch/meilisearch/issues/2725
- [x] Durability of the tasks once an answer has been sent to the user.
- [x] Fix the analytics
- [x] Disable the auto-batching system.
- [x] Make sure the task scheduler run if there are tasks to process.
- [x] Auto-batching of enqueued tasks:
- [x] Do not batch operations from two different indexes.
- [x] Document addition.
- [x] Document updates.
- [x] Settings.
- [x] Document deletion.
- [x] Make sure that we only merge batches with the same index-creation rights:
- [x] the batch either starts with a `yes`
- [x] [we only batch `no`s together and stop batching when we encounter a `yes`](https://www.youtube.com/watch?v=O27mdRvR1GY)
- [x] Unify the logic about `false` and `true` index creation rights.
- [ ] Execute all batch kind:
- [x] Import dumps at startup time.
- [x] Export dumps i.e. export the tasks queue.
- [x] Document addition
- [x] Document update
- [x] Document deletion.
- [x] Clear all documents.
- [x] Update the settings of an index.
- [ ] Merge multiple settings into a single one.
- [x] Index update e.g. Create an Index, change an index primary key, delete an index.
- [x] Cancel enqueued or processing tasks (with filters) (don't count tasks from forbidden indexes) (can't cancel a task with a higher or equal task_id than your own).
- [x] Delete processed tasks from the task store (with filters) (don't count tasks from forbidden indexes) (can't flush a task with a higher or equal task_id than your own)
- [x] Document addition + settings
- [x] Document addition + settings + clear all documents
- [x] anything + index deletion
- [x] Snapshot
- [x] Make the `SnapshotCreation` task visible.
- [x] Snapshot tasks are scheduled by a detached thread.
- [x] Only include update files that are useful.
- [x] Check that statuses and details are correctly set. (ie; if you enqueue a `documentAddition`, is the `documentReceived` well set?)
- [x] Prioritize and reorder tasks i.e. Index deletion, Delete all the documents.
- [x] Always accept new tasks without blocking.
- [x] Fairly share the loads over the different indexes e.g. Always process the index queue with the lowest id.
- [x] Easily testable.
- [x] Well tested i.e. tasks reordering, tasks prioritizing, use atomic barriers to block the tasks for tests.
- [x] Dump
- [x] Serialize the uuid as string in the keys
- [x] Create a dump crate with getters and setters
- [x] Serialize the API key in the dump task
- [x] Get the instance-uuid in the dump task
- [x] List and filter tasks:
- [x] Paginate the tasks.
- [x] Filter by index name.
- [x] Filter on the status, the enqueued, processing, and finished tasks.
- [x] Filter on the type of task.
- [x] Check that it works in `meilisearch-http`.
- [x] Think about [the index wrapper](2c4c14caa8/index/src/updates.rs (L269)) and probably move or remove it.
- [x] Reduce the amount of copy/paste for the batched operations by creating a sub-enum for the `Batch` enum.
- [x] Move the `IndexScheduler` in the lib.rs file.
- [x] Think about the `MilliError` type and probably remove it.
- [x] Remove the `index` crate entirely
- [x] Remove the `Kind` type from the `TaskView` and introduce another type, remove the `<Kind as FromStr>`.
- [x] Once the point above is done; remove the unreachable variant from the autobatchingkind
- [x] Rename the `Settings` task `Kind` to `SettingsUpdate`
- [x] Rename the `DumpExport` task `Kind` to `DumpExport`
- [x] Path the error message when deserializing a `Kind` and `Status`.
- [x] Check the version file when starting.
- [x] Copy the version file when creating snapshots.
---------
Once everything above is done;
- [ ] Check what happens with the update files i.e. when are they deleted.
- [ ] When a TaskDeletion occurs
- [ ] When a TaskCancelation
- [ ] When a task is finished
- [ ] When a task fails
- [ ] When importing a dump forward the date to milli
- [ ] Add tests for the snapshots.
- [ ] Look at all the places where we put _TODOs_.
- [ ] Rename a bunch of things, see https://github.com/meilisearch/meilisearch/pull/2917
- [ ] Ensure that when compiling meilisearch-http with `no-default-features` it doesn’t pull lindera etc
- [ ] Run a bunch of operations in a `tokio::spawn_blocking`
- [ ] The search requests
- [ ] Issue to create once this is merged:
- [ ] Realtime progressing status e.g. Websocket events (optional).
- [ ] Implement an `Uuid` codec instead of using a `Bincode<Uuid>`.
- [ ] Handle the dump-v1
- [ ] When importing a dump v1 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v2 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v3 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v4 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v5 we could iterate over the whole task queue to find the creation and last update date
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
