bors[bot]
dd1011ba76
Merge #2995
...
2995: merge the settings and do one indexation at the end r=irevoire a=irevoire
Co-authored-by: Irevoire <tamo@meilisearch.com>
2022-10-27 21:24:21 +00:00
Tamo
87cac158c4
Update index-scheduler/src/batch.rs
2022-10-27 18:08:21 +02:00
Tamo
c9f89d38e3
Merge branch 'main' into index-swap-error-handling
2022-10-27 18:06:45 +02:00
Irevoire
313f204f39
merge the settings and do one indexation at the end
2022-10-27 16:38:21 +02:00
Loïc Lecrenier
4f4fc20acf
Make clippy happy
2022-10-27 13:00:30 +02:00
Loïc Lecrenier
78ffa00f98
Move index swap error handling from meilisearch-http to index-scheduler
...
And make index_not_found error asynchronous, since we can't know
whether the index will exist by the time the index swap task is
processed.
Improve the index-swap test to verify that future tasks are not swapped
and to test the new error messages that were introduced.
2022-10-27 11:45:38 +02:00
Loïc Lecrenier
7b93ba40bd
Reimplement task queries to account for special index swap rules
2022-10-27 11:44:51 +02:00
Irevoire
8ec3681cf8
fix clippy part1
2022-10-27 11:35:20 +02:00
Kerollmops
2ba5e3b519
Clean up some code
2022-10-27 11:35:20 +02:00
Irevoire
4e1b6b514e
update reviewer change
2022-10-27 11:35:19 +02:00
Loïc Lecrenier
1f75caae88
Fix a few index swap bugs.
...
1. Details of the indexSwap task
2. Query tasks with type=indexUid
3. Synchronous error message for multiple index not found
2022-10-27 11:35:17 +02:00
Kerollmops
035e8eeff5
Clean-up some TODOs
2022-10-27 11:35:15 +02:00
Kerollmops
e35fe33712
Fix some bugs with files
2022-10-27 11:35:15 +02:00
Kerollmops
942b7c338b
Compress the snapshot in a tarball
2022-10-27 11:35:15 +02:00
Kerollmops
4cafc63561
Reintroduce the versioning functions
2022-10-27 11:35:14 +02:00
Kerollmops
89e127e4f4
Declare the auth path in the index scheduler
2022-10-27 11:35:14 +02:00
Kerollmops
eec43ec953
Implement a first version of the snapshots
2022-10-27 11:35:14 +02:00
Kerollmops
e0548e42e7
Rename the Snapshot task into SnapshotCreation
2022-10-27 11:35:14 +02:00
Loïc Lecrenier
d92425658e
Add index scheduler tests for task cancelation
2022-10-27 11:35:12 +02:00
Loïc Lecrenier
16fac10074
Fix crash when batching an index swap task containing 0 swaps
2022-10-27 11:35:12 +02:00
Irevoire
0aca5e84b9
rename received_document_ids to matched_documents in the DocumentDeletion task type (reimplementation of #2826 )
2022-10-27 11:35:12 +02:00
Loïc Lecrenier
4de445d386
Start testing unexpected errors and panics in index scheduler
2022-10-27 11:35:10 +02:00
Irevoire
ecf4e43b3d
rename the dumpExport to dumpCreation
2022-10-27 11:35:10 +02:00
Irevoire
e9055f5572
fix clippy
2022-10-27 11:35:08 +02:00
Irevoire
c8ee453b6c
fix the autobatched document deletion
2022-10-27 11:35:07 +02:00
Irevoire
a8de5368e5
fix the index creation in case an index already exists
2022-10-27 11:35:07 +02:00
Irevoire
9bb2e3c790
fix the failed document addition with a primary key
2022-10-27 11:35:07 +02:00
Irevoire
8d1408c65e
fix the import of the dumpv4&v5 when there is no instance-uid + rename the Kind+KindWithContent+Details variant for the DocumentImport and the Setting
2022-10-27 11:35:05 +02:00
Clément Renault
80b2e70ee7
Introduce a rustfmt file
2022-10-27 11:35:05 +02:00
Clément Renault
72ec4ce96b
Fix allow_index_creation useless field
2022-10-27 11:34:17 +02:00
Irevoire
b6a0abea9f
fix the index deletion when the index doesn’t exists but would be created by one of the autobatched tasks
2022-10-27 11:34:16 +02:00
Irevoire
d9218578e3
it probably works but it's also horrendous
2022-10-27 11:34:16 +02:00
Loïc Lecrenier
11fee30f47
Apply review suggestions and stop using rtxn.commit
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
17cd2a4aa0
Implement POST /indexes-swap
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
169f386418
Add some documentation to the index scheduler
2022-10-27 11:34:15 +02:00
Loïc Lecrenier
22cf0559fe
Implement task date filters
...
before/after enqueued/started/finished at
2022-10-27 11:34:14 +02:00
Irevoire
5765883600
fix the auto-generated details
2022-10-27 11:34:14 +02:00
Tamo
cff003c928
remove the unused variants from the autobatcher
2022-10-27 11:34:14 +02:00
Kerollmops
50b8b9df6a
Delete the tasks content file once the transaction has been successfully committed
2022-10-27 11:34:13 +02:00
Kerollmops
b373d19831
Extract the must_stop flag out of the RwLock
2022-10-27 11:34:12 +02:00
Kerollmops
3cbfacb616
Prefer using an u64 instead of a usize in some places
2022-10-27 11:34:12 +02:00
Kerollmops
79c4275bfc
Delete the persisted data when we cancel a task
2022-10-27 11:34:12 +02:00
Kerollmops
c2ec4a089b
Put the original URL query in the tasks details
2022-10-27 11:34:12 +02:00
Kerollmops
290945e258
Update the canceledBy and finishedAt fields
2022-10-27 11:34:11 +02:00
Kerollmops
725158b454
Introduce the core algorithm of task cancelation
2022-10-27 11:34:11 +02:00
Kerollmops
1ca9a67c49
Introduce the task cancelation task type
2022-10-27 11:34:11 +02:00
Kerollmops
703ba7a1fb
Introduce the ProcessingTasks struct
2022-10-27 11:34:10 +02:00
Loïc Lecrenier
ea60d35c71
Delete a task's persisted data when appropriate
2022-10-27 11:34:10 +02:00
Tamo
2f748480a1
share the rtxn between the access to the tasks and to the indexes
2022-10-27 11:34:09 +02:00
Tamo
83f3c5ec57
flush the dump-writer only once everything has been inserted
2022-10-27 11:34:08 +02:00
