Clément Renault
2a7ef3b352
Reapply #2830
2022-10-27 11:34:18 +02:00
Clément Renault
72ec4ce96b
Fix allow_index_creation useless field
2022-10-27 11:34:17 +02:00
Clément Renault
75857bf476
Fix the insta tests
2022-10-27 11:34:17 +02:00
Irevoire
0bbf80186f
push the snapshot files
2022-10-27 11:34:17 +02:00
Irevoire
b6a0abea9f
fix the index deletion when the index doesn’t exists but would be created by one of the autobatched tasks
2022-10-27 11:34:16 +02:00
Irevoire
5303bbffab
fix the last rule about merging the allow_index_creation
2022-10-27 11:34:16 +02:00
Irevoire
fc944c39a5
simplify the code A LOT and create less false positive
2022-10-27 11:34:16 +02:00
Irevoire
a1d4cc673d
add a whole new batch of tests around the index already exists / allow_index_creation
2022-10-27 11:34:16 +02:00
Irevoire
28d9f2c041
fix all the snapshot tests
2022-10-27 11:34:16 +02:00
Irevoire
d9218578e3
it probably works but it's also horrendous
2022-10-27 11:34:16 +02:00
Loïc Lecrenier
11fee30f47
Apply review suggestions and stop using rtxn.commit
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
17cd2a4aa0
Implement POST /indexes-swap
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
28bd8b6c6b
Remove key from index_tasks database when the value is empty
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
169f386418
Add some documentation to the index scheduler
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
bdb17954d2
Fix bug where assert used != instead of ==
...
And update snapshot tests.
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
23b01a58df
cargo fmt
2022-10-27 11:34:14 +02:00
Loïc Lecrenier
ec3391808d
Fix date parsing for task queries
...
Use rfc3339 or YYYY-MM-DD.
Add a day to the parsed date when it is an excluded lower bound
and the YYYY-MM-DD was used.
Also the Query type does not need to be serialisable anymore
2022-10-27 11:34:14 +02:00
Loïc Lecrenier
10a547df4f
Apply suggestions from code review
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
Apply suggestions from code review
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Apply suggestions from code review
Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Apply code review suggestion
Co-authored-by: Clément Renault <clement@meilisearch.com>
2022-10-27 11:34:14 +02:00
Loïc Lecrenier
22cf0559fe
Implement task date filters
...
before/after enqueued/started/finished at
2022-10-27 11:34:14 +02:00
Irevoire
5765883600
fix the auto-generated details
2022-10-27 11:34:14 +02:00
Tamo
cff003c928
remove the unused variants from the autobatcher
2022-10-27 11:34:14 +02:00
Tamo
ab8f1c2865
fix a bunch of snapshot tests
2022-10-27 11:34:13 +02:00
Kerollmops
50b8b9df6a
Delete the tasks content file once the transaction has been successfully committed
2022-10-27 11:34:13 +02:00
Kerollmops
ec0a5a9f01
Remove the useless r#union thing
2022-10-27 11:34:13 +02:00
Kerollmops
6460b78e08
Clean up the delete_persisted_task_data function
2022-10-27 11:34:13 +02:00
Kerollmops
d21651c968
Throw the error if we can't register the tasks in the store
2022-10-27 11:34:13 +02:00
Kerollmops
6e904d0997
Introduce a ProcessingTasks constructor
2022-10-27 11:34:12 +02:00
Kerollmops
b373d19831
Extract the must_stop flag out of the RwLock
2022-10-27 11:34:12 +02:00
Kerollmops
3cbfacb616
Prefer using an u64 instead of a usize in some places
2022-10-27 11:34:12 +02:00
Kerollmops
79c4275bfc
Delete the persisted data when we cancel a task
2022-10-27 11:34:12 +02:00
Kerollmops
c2ec4a089b
Put the original URL query in the tasks details
2022-10-27 11:34:12 +02:00
Kerollmops
751e9bac3b
Add the tasks cancel route to cancel tasks
2022-10-27 11:34:11 +02:00
Kerollmops
290945e258
Update the canceledBy and finishedAt fields
2022-10-27 11:34:11 +02:00
Kerollmops
725158b454
Introduce the core algorithm of task cancelation
2022-10-27 11:34:11 +02:00
Kerollmops
b2c5bc67b7
Add more enum-iterator related stuff
2022-10-27 11:34:11 +02:00
Kerollmops
591527a99d
Prefer using TaskDeletion in the dumps
2022-10-27 11:34:11 +02:00
Kerollmops
1ca9a67c49
Introduce the task cancelation task type
2022-10-27 11:34:11 +02:00
Kerollmops
703ba7a1fb
Introduce the ProcessingTasks struct
2022-10-27 11:34:10 +02:00
Loïc Lecrenier
ea60d35c71
Delete a task's persisted data when appropriate
2022-10-27 11:34:10 +02:00
Tamo
9e85f050b2
fix the tests
2022-10-27 11:34:09 +02:00
Tamo
2f748480a1
share the rtxn between the access to the tasks and to the indexes
2022-10-27 11:34:09 +02:00
Tamo
6bd6321226
dump the content of the dump tasks instead of recreating at import time with wrong API keys
2022-10-27 11:34:08 +02:00
Tamo
655705eb2b
remove useless todo
2022-10-27 11:34:08 +02:00
Tamo
83f3c5ec57
flush the dump-writer only once everything has been inserted
2022-10-27 11:34:08 +02:00
Tamo
78ce29f461
apply most style comments of the review
2022-10-27 11:34:08 +02:00
Tamo
d0e91555d1
rebase on index-scheduler
2022-10-27 11:34:08 +02:00
Tamo
e0221fc0a3
fix a synchronization bug while importing tasks
2022-10-27 11:34:07 +02:00
Tamo
ba150f2127
commit after creating an index
2022-10-27 11:34:07 +02:00
Tamo
e9295c03ce
the index-scheduler needs to wake-up after importing a dump
2022-10-27 11:34:06 +02:00
Tamo
d481669b7e
fix the content_file import
2022-10-27 11:34:06 +02:00
Tamo
dd506e5d87
stop dumping the current dumping task as enqueued so it's not looping for ever
2022-10-27 11:34:06 +02:00
Tamo
d976e680c5
first mostly working version
2022-10-27 11:34:06 +02:00
Tamo
c051166bcc
update the API a little bit
2022-10-27 11:34:05 +02:00
Tamo
72a906ae75
fix the tests
2022-10-27 11:34:05 +02:00
Tamo
b7f9c94f4a
write the dump export
2022-10-27 11:34:05 +02:00
Loïc Lecrenier
8954b1bd1d
Fix number of deleted tasks details after duplicate task deletion
2022-10-27 11:34:05 +02:00
Loïc Lecrenier
8defad6c38
Add task deletion tests where the same task is deleted twice
2022-10-27 11:34:05 +02:00
Loïc Lecrenier
f32b973945
Return an error when calling DELETE /tasks with an empty query
2022-10-27 11:34:04 +02:00
Loïc Lecrenier
fbd2be2ec8
Apply suggested changes from PR review
2022-10-27 11:34:04 +02:00
Loïc Lecrenier
441417447e
Avoid creating two read txn at the same time
2022-10-27 11:34:04 +02:00
Loïc Lecrenier
8c6aeaada5
Update snapshot tests following git rebase that fixes a bug
2022-10-27 11:34:04 +02:00
Loïc Lecrenier
8bb0fcd144
Finish first draft of the DELETE /tasks route
2022-10-27 11:34:04 +02:00
Loïc Lecrenier
9522b75454
Continue implementation of task deletion
...
1. Matched tasks are a roaring bitmap
2. Start implementation in meilisearch-http
3. Snapshots use meili-snap
4. Rename to TaskDeletion
2022-10-27 11:34:03 +02:00
Kerollmops
e4d461ecba
Make sure that we do not batch tasks from different indexes
2022-10-27 11:34:03 +02:00
Kerollmops
b029369653
Add a test to check different indexes autobatching
2022-10-27 11:34:03 +02:00
Kerollmops
408d00136c
Extract index creation rights and simplify the autobatcher rules
2022-10-27 11:34:03 +02:00
Kerollmops
2c24c7d403
Fix invalid import of tasks types
2022-10-27 11:34:03 +02:00
Tamo
7034803712
move the API key in meilisearch_types
2022-10-27 11:34:02 +02:00
Tamo
c192146fbe
remove an unused file
2022-10-27 11:34:02 +02:00
Tamo
2f1eb78b1d
refactor the Task a little bit
2022-10-27 11:34:02 +02:00
Tamo
510ce9fc51
start moving a lot of task types to meilisearch_types
2022-10-27 11:34:01 +02:00
Tamo
0af00f6b32
fix all the import and comment most of the dump v6
2022-10-27 11:34:01 +02:00
Tamo
667c282e19
get rids of the index crate + the document_types crate
2022-10-27 11:34:00 +02:00
Loïc Lecrenier
9a74ea0943
Fix compiler errors related autobatching option of the index scheduler
2022-10-27 11:34:00 +02:00
Loïc Lecrenier
eabac9676b
Fix typo and remove useless code in tests
2022-10-27 11:34:00 +02:00
Loïc Lecrenier
ab4e649221
Apply suggestions from code review
...
Co-authored-by: Tamo <tamo@meilisearch.com>
2022-10-27 11:34:00 +02:00
Loïc Lecrenier
568199fc0d
Add more task deletion tests
2022-10-27 11:33:59 +02:00
Loïc Lecrenier
13a72f8757
Use more complete snapshot tests for the index scheduler
2022-10-27 11:33:59 +02:00
Loïc Lecrenier
4c55c30027
Add a DetailsView type and improve index scheduler snapshots
...
The DetailsView type is necessary because serde incorrectly
deserialises the `Details` type, so the database fails to correctly
decode Tasks
2022-10-27 11:33:59 +02:00
Loïc Lecrenier
dc81992eb2
Implement TaskDeletion in the index scheduler
2022-10-27 11:33:59 +02:00
Kerollmops
fe84f2648b
Allow a user to disable the auto batching system
2022-10-27 11:33:59 +02:00
Kerollmops
e2a766acb5
Add a test to check that it works without autobatching
2022-10-27 11:33:58 +02:00
Kerollmops
db9d1b18ca
Remove the IndexScheduler::notify method
2022-10-27 11:33:58 +02:00
Kerollmops
19c6f8303f
Make sure that the index-scheduler tick loop is rerun after processing
2022-10-27 11:33:58 +02:00
Kerollmops
b311eb3bed
Add a test that verifies that sending multiple tasks works
2022-10-27 11:33:58 +02:00
Tamo
0972587cfc
start writting the compat layer between v5 and v6
2022-10-27 11:33:52 +02:00
Tamo
e845cc2b6f
fix the tests
2022-10-27 11:33:51 +02:00
Clément Renault
123f47dbc4
Create the index only if the task has the rights to do so
2022-10-27 11:33:50 +02:00
Clément Renault
068a4b2884
Correctly batch tasks with different index creation rights
2022-10-27 11:33:50 +02:00
Clément Renault
87212cfd20
Use a ControlFlow in the autobatcher function
2022-10-27 11:33:50 +02:00
Kerollmops
f1b1cfdbcc
IndexDeletion operation have ClearAll details
2022-10-27 11:33:50 +02:00
Kerollmops
a083c9e452
Only mark the first clear document with the amount of cleared documents
2022-10-27 11:33:50 +02:00
Kerollmops
b24b13b036
Let the tick function set the Failed status itself
2022-10-27 11:33:50 +02:00
Kerollmops
566c15fb74
Fill an IndexDeletion task with the number of documents removed
2022-10-27 11:33:49 +02:00
Kerollmops
6b3b05fb73
Panic if we encountered a wring KindWithContent type
2022-10-27 11:33:49 +02:00
Kerollmops
36e5efde0d
Update the tasks statuses
2022-10-27 11:33:49 +02:00
Kerollmops
2fbdd104b8
Implement the IndexDeletion batch operation
2022-10-27 11:33:49 +02:00
Kerollmops
da363a92ac
Implement the IndexUpdate batch operation
2022-10-27 11:33:49 +02:00
Kerollmops
0543cba6eb
Implement the IndexCreate batch operation
2022-10-27 11:33:48 +02:00
Kerollmops
cf6084151b
Make sure that meilisearch-http works without index wrapper
2022-10-27 11:33:48 +02:00
