2763: Index scheduler r=Kerollmops a=irevoire
Fix https://github.com/meilisearch/meilisearch/issues/2725
- [x] Durability of the tasks once an answer has been sent to the user.
- [x] Fix the analytics
- [x] Disable the auto-batching system.
- [x] Make sure the task scheduler run if there are tasks to process.
- [x] Auto-batching of enqueued tasks:
- [x] Do not batch operations from two different indexes.
- [x] Document addition.
- [x] Document updates.
- [x] Settings.
- [x] Document deletion.
- [x] Make sure that we only merge batches with the same index-creation rights:
- [x] the batch either starts with a `yes`
- [x] [we only batch `no`s together and stop batching when we encounter a `yes`](https://www.youtube.com/watch?v=O27mdRvR1GY)
- [x] Unify the logic about `false` and `true` index creation rights.
- [ ] Execute all batch kind:
- [x] Import dumps at startup time.
- [x] Export dumps i.e. export the tasks queue.
- [x] Document addition
- [x] Document update
- [x] Document deletion.
- [x] Clear all documents.
- [x] Update the settings of an index.
- [ ] Merge multiple settings into a single one.
- [x] Index update e.g. Create an Index, change an index primary key, delete an index.
- [x] Cancel enqueued or processing tasks (with filters) (don't count tasks from forbidden indexes) (can't cancel a task with a higher or equal task_id than your own).
- [x] Delete processed tasks from the task store (with filters) (don't count tasks from forbidden indexes) (can't flush a task with a higher or equal task_id than your own)
- [x] Document addition + settings
- [x] Document addition + settings + clear all documents
- [x] anything + index deletion
- [x] Snapshot
- [x] Make the `SnapshotCreation` task visible.
- [x] Snapshot tasks are scheduled by a detached thread.
- [x] Only include update files that are useful.
- [x] Check that statuses and details are correctly set. (ie; if you enqueue a `documentAddition`, is the `documentReceived` well set?)
- [x] Prioritize and reorder tasks i.e. Index deletion, Delete all the documents.
- [x] Always accept new tasks without blocking.
- [x] Fairly share the loads over the different indexes e.g. Always process the index queue with the lowest id.
- [x] Easily testable.
- [x] Well tested i.e. tasks reordering, tasks prioritizing, use atomic barriers to block the tasks for tests.
- [x] Dump
- [x] Serialize the uuid as string in the keys
- [x] Create a dump crate with getters and setters
- [x] Serialize the API key in the dump task
- [x] Get the instance-uuid in the dump task
- [x] List and filter tasks:
- [x] Paginate the tasks.
- [x] Filter by index name.
- [x] Filter on the status, the enqueued, processing, and finished tasks.
- [x] Filter on the type of task.
- [x] Check that it works in `meilisearch-http`.
- [x] Think about [the index wrapper](2c4c14caa8/index/src/updates.rs (L269)) and probably move or remove it.
- [x] Reduce the amount of copy/paste for the batched operations by creating a sub-enum for the `Batch` enum.
- [x] Move the `IndexScheduler` in the lib.rs file.
- [x] Think about the `MilliError` type and probably remove it.
- [x] Remove the `index` crate entirely
- [x] Remove the `Kind` type from the `TaskView` and introduce another type, remove the `<Kind as FromStr>`.
- [x] Once the point above is done; remove the unreachable variant from the autobatchingkind
- [x] Rename the `Settings` task `Kind` to `SettingsUpdate`
- [x] Rename the `DumpExport` task `Kind` to `DumpExport`
- [x] Path the error message when deserializing a `Kind` and `Status`.
- [x] Check the version file when starting.
- [x] Copy the version file when creating snapshots.
---------
Once everything above is done;
- [ ] Check what happens with the update files i.e. when are they deleted.
- [ ] When a TaskDeletion occurs
- [ ] When a TaskCancelation
- [ ] When a task is finished
- [ ] When a task fails
- [ ] When importing a dump forward the date to milli
- [ ] Add tests for the snapshots.
- [ ] Look at all the places where we put _TODOs_.
- [ ] Rename a bunch of things, see https://github.com/meilisearch/meilisearch/pull/2917
- [ ] Ensure that when compiling meilisearch-http with `no-default-features` it doesn’t pull lindera etc
- [ ] Run a bunch of operations in a `tokio::spawn_blocking`
- [ ] The search requests
- [ ] Issue to create once this is merged:
- [ ] Realtime progressing status e.g. Websocket events (optional).
- [ ] Implement an `Uuid` codec instead of using a `Bincode<Uuid>`.
- [ ] Handle the dump-v1
- [ ] When importing a dump v1 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v2 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v3 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v4 we could iterate over the whole task queue to find the creation and last update date
- [ ] When importing a dump v5 we could iterate over the whole task queue to find the creation and last update date
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
