snowykami/meilisearch
1
0
Fork 0
mirror of https://github.com/meilisearch/meilisearch.git synced 2024-11-23 02:27:40 +08:00
Code Issues Actions 9 Packages Projects Releases Wiki Activity
8,085 Commits 165 Branches 386 Tags 80 MiB
c47b8a8bfe
Commit Graph

373 Commits

Author SHA1 Message Date
Clément Renault
75857bf476
Fix the insta tests 2022-10-27 11:34:17 +02:00
Irevoire
0bbf80186f
push the snapshot files 2022-10-27 11:34:17 +02:00
Irevoire
b6a0abea9f
fix the index deletion when the index doesn’t exists but would be created by one of the autobatched tasks 2022-10-27 11:34:16 +02:00
Irevoire
5303bbffab
fix the last rule about merging the allow_index_creation 2022-10-27 11:34:16 +02:00
Irevoire
fc944c39a5
simplify the code A LOT and create less false positive 2022-10-27 11:34:16 +02:00
Irevoire
a1d4cc673d
add a whole new batch of tests around the index already exists / allow_index_creation 2022-10-27 11:34:16 +02:00
Irevoire
28d9f2c041
fix all the snapshot tests 2022-10-27 11:34:16 +02:00
Irevoire
d9218578e3
it probably works but it's also horrendous 2022-10-27 11:34:16 +02:00
Loïc Lecrenier
11fee30f47
Apply review suggestions and stop using rtxn.commit 2022-10-27 11:34:15 +02:00
Loïc Lecrenier
17cd2a4aa0
Implement POST /indexes-swap 2022-10-27 11:34:15 +02:00
Loïc Lecrenier
28bd8b6c6b
Remove key from index_tasks database when the value is empty 2022-10-27 11:34:15 +02:00
Loïc Lecrenier
169f386418
Add some documentation to the index scheduler 2022-10-27 11:34:15 +02:00
Loïc Lecrenier
bdb17954d2
Fix bug where assert used != instead of == 
And update snapshot tests.
 2022-10-27 11:34:15 +02:00
Loïc Lecrenier
23b01a58df
cargo fmt 2022-10-27 11:34:14 +02:00
Loïc Lecrenier
ec3391808d
Fix date parsing for task queries 
Use rfc3339 or YYYY-MM-DD.

Add a day to the parsed date when it is an excluded lower bound
and the YYYY-MM-DD was used.

Also the Query type does not need to be serialisable anymore
 2022-10-27 11:34:14 +02:00
Loïc Lecrenier
10a547df4f
Apply suggestions from code review 
Co-authored-by: Clément Renault <clement@meilisearch.com>

Apply suggestions from code review

Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>

Apply suggestions from code review

Co-authored-by: Clément Renault <clement@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>

Apply code review suggestion

Co-authored-by: Clément Renault <clement@meilisearch.com>
 2022-10-27 11:34:14 +02:00
Loïc Lecrenier
22cf0559fe
Implement task date filters 
before/after enqueued/started/finished at
 2022-10-27 11:34:14 +02:00
Irevoire
5765883600
fix the auto-generated details 2022-10-27 11:34:14 +02:00
Tamo
cff003c928
remove the unused variants from the autobatcher 2022-10-27 11:34:14 +02:00
Tamo
ab8f1c2865
fix a bunch of snapshot tests 2022-10-27 11:34:13 +02:00
Kerollmops
50b8b9df6a
Delete the tasks content file once the transaction has been successfully committed 2022-10-27 11:34:13 +02:00
Kerollmops
ec0a5a9f01
Remove the useless r#union thing 2022-10-27 11:34:13 +02:00
Kerollmops
6460b78e08
Clean up the delete_persisted_task_data function 2022-10-27 11:34:13 +02:00
Kerollmops
d21651c968
Throw the error if we can't register the tasks in the store 2022-10-27 11:34:13 +02:00
Kerollmops
6e904d0997
Introduce a ProcessingTasks constructor 2022-10-27 11:34:12 +02:00
Kerollmops
b373d19831
Extract the must_stop flag out of the RwLock 2022-10-27 11:34:12 +02:00
Kerollmops
3cbfacb616
Prefer using an u64 instead of a usize in some places 2022-10-27 11:34:12 +02:00
Kerollmops
79c4275bfc
Delete the persisted data when we cancel a task 2022-10-27 11:34:12 +02:00
Kerollmops
c2ec4a089b
Put the original URL query in the tasks details 2022-10-27 11:34:12 +02:00
Kerollmops
751e9bac3b
Add the tasks cancel route to cancel tasks 2022-10-27 11:34:11 +02:00
Kerollmops
290945e258
Update the canceledBy and finishedAt fields 2022-10-27 11:34:11 +02:00
Kerollmops
725158b454
Introduce the core algorithm of task cancelation 2022-10-27 11:34:11 +02:00
Kerollmops
b2c5bc67b7
Add more enum-iterator related stuff 2022-10-27 11:34:11 +02:00
Kerollmops
591527a99d
Prefer using TaskDeletion in the dumps 2022-10-27 11:34:11 +02:00
Kerollmops
1ca9a67c49
Introduce the task cancelation task type 2022-10-27 11:34:11 +02:00
Kerollmops
703ba7a1fb
Introduce the ProcessingTasks struct 2022-10-27 11:34:10 +02:00
Loïc Lecrenier
ea60d35c71
Delete a task's persisted data when appropriate 2022-10-27 11:34:10 +02:00
Tamo
9e85f050b2
fix the tests 2022-10-27 11:34:09 +02:00
Tamo
2f748480a1
share the rtxn between the access to the tasks and to the indexes 2022-10-27 11:34:09 +02:00
Tamo
6bd6321226
dump the content of the dump tasks instead of recreating at import time with wrong API keys 2022-10-27 11:34:08 +02:00
Tamo
655705eb2b
remove useless todo 2022-10-27 11:34:08 +02:00
Tamo
83f3c5ec57
flush the dump-writer only once everything has been inserted 2022-10-27 11:34:08 +02:00
Tamo
78ce29f461
apply most style comments of the review 2022-10-27 11:34:08 +02:00
Tamo
d0e91555d1
rebase on index-scheduler 2022-10-27 11:34:08 +02:00
Tamo
e0221fc0a3
fix a synchronization bug while importing tasks 2022-10-27 11:34:07 +02:00
Tamo
ba150f2127
commit after creating an index 2022-10-27 11:34:07 +02:00
Tamo
e9295c03ce
the index-scheduler needs to wake-up after importing a dump 2022-10-27 11:34:06 +02:00
Tamo
d481669b7e
fix the content_file import 2022-10-27 11:34:06 +02:00
Tamo
dd506e5d87
stop dumping the current dumping task as enqueued so it's not looping for ever 2022-10-27 11:34:06 +02:00
Tamo
d976e680c5
first mostly working version 2022-10-27 11:34:06 +02:00
Tamo
c051166bcc
update the API a little bit 2022-10-27 11:34:05 +02:00
Tamo
72a906ae75
fix the tests 2022-10-27 11:34:05 +02:00
Tamo
b7f9c94f4a
write the dump export 2022-10-27 11:34:05 +02:00
Loïc Lecrenier
8954b1bd1d
Fix number of deleted tasks details after duplicate task deletion 2022-10-27 11:34:05 +02:00
Loïc Lecrenier
8defad6c38
Add task deletion tests where the same task is deleted twice 2022-10-27 11:34:05 +02:00
Loïc Lecrenier
f32b973945
Return an error when calling DELETE /tasks with an empty query 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
fbd2be2ec8
Apply suggested changes from PR review 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
441417447e
Avoid creating two read txn at the same time 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
8c6aeaada5
Update snapshot tests following git rebase that fixes a bug 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
8bb0fcd144
Finish first draft of the DELETE /tasks route 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
9522b75454
Continue implementation of task deletion 
1. Matched tasks are a roaring bitmap
2. Start implementation in meilisearch-http
3. Snapshots use meili-snap
4. Rename to TaskDeletion
 2022-10-27 11:34:03 +02:00
Kerollmops
e4d461ecba
Make sure that we do not batch tasks from different indexes 2022-10-27 11:34:03 +02:00
Kerollmops
b029369653
Add a test to check different indexes autobatching 2022-10-27 11:34:03 +02:00
Kerollmops
408d00136c
Extract index creation rights and simplify the autobatcher rules 2022-10-27 11:34:03 +02:00
Kerollmops
2c24c7d403
Fix invalid import of tasks types 2022-10-27 11:34:03 +02:00
Tamo
7034803712
move the API key in meilisearch_types 2022-10-27 11:34:02 +02:00
Tamo
c192146fbe
remove an unused file 2022-10-27 11:34:02 +02:00
Tamo
2f1eb78b1d
refactor the Task a little bit 2022-10-27 11:34:02 +02:00
Tamo
510ce9fc51
start moving a lot of task types to meilisearch_types 2022-10-27 11:34:01 +02:00
Tamo
0af00f6b32
fix all the import and comment most of the dump v6 2022-10-27 11:34:01 +02:00
Tamo
667c282e19
get rids of the index crate + the document_types crate 2022-10-27 11:34:00 +02:00
Loïc Lecrenier
9a74ea0943
Fix compiler errors related autobatching option of the index scheduler 2022-10-27 11:34:00 +02:00
Loïc Lecrenier
eabac9676b
Fix typo and remove useless code in tests 2022-10-27 11:34:00 +02:00
Loïc Lecrenier
ab4e649221
Apply suggestions from code review 
Co-authored-by: Tamo <tamo@meilisearch.com>
 2022-10-27 11:34:00 +02:00
Loïc Lecrenier
568199fc0d
Add more task deletion tests 2022-10-27 11:33:59 +02:00
Loïc Lecrenier
13a72f8757
Use more complete snapshot tests for the index scheduler 2022-10-27 11:33:59 +02:00
Loïc Lecrenier
4c55c30027
Add a DetailsView type and improve index scheduler snapshots 
The DetailsView type is necessary because serde incorrectly
deserialises the `Details` type, so the database fails to correctly
decode Tasks
 2022-10-27 11:33:59 +02:00
Loïc Lecrenier
dc81992eb2
Implement TaskDeletion in the index scheduler 2022-10-27 11:33:59 +02:00
Kerollmops
fe84f2648b
Allow a user to disable the auto batching system 2022-10-27 11:33:59 +02:00
Kerollmops
e2a766acb5
Add a test to check that it works without autobatching 2022-10-27 11:33:58 +02:00
Kerollmops
db9d1b18ca
Remove the IndexScheduler::notify method 2022-10-27 11:33:58 +02:00
Kerollmops
19c6f8303f
Make sure that the index-scheduler tick loop is rerun after processing 2022-10-27 11:33:58 +02:00
Kerollmops
b311eb3bed
Add a test that verifies that sending multiple tasks works 2022-10-27 11:33:58 +02:00
Tamo
0972587cfc
start writting the compat layer between v5 and v6 2022-10-27 11:33:52 +02:00
Tamo
e845cc2b6f
fix the tests 2022-10-27 11:33:51 +02:00
Clément Renault
123f47dbc4
Create the index only if the task has the rights to do so 2022-10-27 11:33:50 +02:00
Clément Renault
068a4b2884
Correctly batch tasks with different index creation rights 2022-10-27 11:33:50 +02:00
Clément Renault
87212cfd20
Use a ControlFlow in the autobatcher function 2022-10-27 11:33:50 +02:00
Kerollmops
f1b1cfdbcc
IndexDeletion operation have ClearAll details 2022-10-27 11:33:50 +02:00
Kerollmops
a083c9e452
Only mark the first clear document with the amount of cleared documents 2022-10-27 11:33:50 +02:00
Kerollmops
b24b13b036
Let the tick function set the Failed status itself 2022-10-27 11:33:50 +02:00
Kerollmops
566c15fb74
Fill an IndexDeletion task with the number of documents removed 2022-10-27 11:33:49 +02:00
Kerollmops
6b3b05fb73
Panic if we encountered a wring KindWithContent type 2022-10-27 11:33:49 +02:00
Kerollmops
36e5efde0d
Update the tasks statuses 2022-10-27 11:33:49 +02:00
Kerollmops
2fbdd104b8
Implement the IndexDeletion batch operation 2022-10-27 11:33:49 +02:00
Kerollmops
da363a92ac
Implement the IndexUpdate batch operation 2022-10-27 11:33:49 +02:00
Kerollmops
0543cba6eb
Implement the IndexCreate batch operation 2022-10-27 11:33:48 +02:00
Kerollmops
cf6084151b
Make sure that meilisearch-http works without index wrapper 2022-10-27 11:33:48 +02:00
Kerollmops
d76634a36c
Remove the Index wrapper and use milli::Index directly 2022-10-27 11:33:48 +02:00
Kerollmops
9e8242c57d
Remove the IndexRename operation 2022-10-27 11:33:48 +02:00
Kerollmops
5fa214abb1
Move the IndexScheduler to the root of the index-scheduler crate 2022-10-27 11:33:47 +02:00
Kerollmops
9a9e98fb77
Add a TODO about the index creation 2022-10-27 11:33:47 +02:00
Kerollmops
5d21c790ef
Make clippy happy 2022-10-27 11:33:47 +02:00
Kerollmops
31de33d5ee
Implement a recursive indexation for the index-related operations 2022-10-27 11:33:47 +02:00
Kerollmops
07286fcc79
Implement the SettingsAndDocumentImport batch operation 2022-10-27 11:33:47 +02:00
Kerollmops
f68906f5dc
Merge both DocumentAddition/Update into one DocumentImport variant 2022-10-27 11:33:46 +02:00
Kerollmops
5174c78f87
Implement the DocumentClear batch operation 2022-10-27 11:33:46 +02:00
Kerollmops
025bb5f616
Implement the DocumentClearAndSettings batch operation 2022-10-27 11:33:46 +02:00
Kerollmops
41ec737e73
Implement the Settings batch operation 2022-10-27 11:33:46 +02:00
Kerollmops
7b4a913704
Implement the DocumentUpdate batch operation 2022-10-27 11:33:46 +02:00
Kerollmops
a6a1043abb
Implement the DocumentDeletion batch operation 2022-10-27 11:33:46 +02:00
Tamo
7a0f17c912
remove an old unworking part of the batch execution 2022-10-27 11:33:45 +02:00
Tamo
c2899fe9b2
bring back the IndexMeta and IndexStats in meilisearch-http 2022-10-27 11:33:45 +02:00
Tamo
2c8f1a43e9
get rids of meilisearch-lib 2022-10-27 11:33:44 +02:00
Tamo
0ba1c46e19
fix a deadlock 2022-10-27 11:33:44 +02:00
Tamo
22bfb5a7a0
remove Clone from the IndexScheduler 2022-10-27 11:33:44 +02:00
Tamo
d8d3499aec
remove a bunch of comments 2022-10-27 11:33:44 +02:00
Tamo
64e132ce53
move as many fields as possible out of the IndexScheduler 2022-10-27 11:33:44 +02:00
Tamo
9e1f38ec7c
move the test function in the test module 2022-10-27 11:33:44 +02:00
Tamo
6f4dcc0c38
start implementing some logic to test the internal states of the scheduler 2022-10-27 11:33:43 +02:00
Tamo
84cd5cef0b
fix the tests 2022-10-27 11:33:43 +02:00
Tamo
ae86a8ccd6
slightly refactor the autobatching tests 2022-10-27 11:33:43 +02:00
Tamo
ce2dfecc03
connect the new scheduler to meilisearch-http officially. 
I can index documents and do search
 2022-10-27 11:33:43 +02:00
Tamo
cb4feabca2
implements the get_tasks 2022-10-27 11:33:43 +02:00
Tamo
19154e48fe
fix all compilation errors 2022-10-27 11:33:42 +02:00
Irevoire
8d51c1f389
wip integrating the scheduler in meilisearch-http 2022-10-27 11:33:42 +02:00
Irevoire
250410495c
start integrating the index-scheduler in meilisearch-lib 2022-10-27 11:33:42 +02:00
Irevoire
8f0fd35358
add insta::json for later 2022-10-27 11:33:42 +02:00
Irevoire
8770e07397
I can index documents without meilisearch 2022-10-27 11:33:42 +02:00
Tamo
edd8344dc9
wip 2022-10-27 11:33:42 +02:00
Tamo
e547552702
create the end Batch type for all Index* operations 2022-10-27 11:33:41 +02:00
Tamo
925971809a
create the end Batch type for all Document* operation 2022-10-27 11:33:41 +02:00
Tamo
1ea9c0b4c0
write most of the run loop 2022-10-27 11:33:41 +02:00
Tamo
9ff0fe952e
split the run function in two 2022-10-27 11:33:41 +02:00
Tamo
a8b18b2c96
fix the register test 2022-10-27 11:33:40 +02:00
Tamo
5436b996ab
reduce the size of the snapshots 2022-10-27 11:33:40 +02:00
Tamo
7d0c8a3379
test the register tasks 2022-10-27 11:33:40 +02:00
Tamo
fc098022c7
start integrating the index-scheduler in the meilisearch codebase 2022-10-27 11:33:40 +02:00
Tamo
b816535e33
greatly reduce the number of warnings 2022-10-27 11:33:40 +02:00
Tamo
38e4ffe73c
fix smol typo 2022-10-27 11:33:40 +02:00
Tamo
366a344474
get rids of the horrendous spinlock in favor of synchronoise 2022-10-27 11:33:39 +02:00
Tamo
7b6673dc1d
implement the index swap in the index mapper 2022-10-27 11:33:39 +02:00
Tamo
03aca2e452
move the index mapping logic in another structure 2022-10-27 11:33:39 +02:00
Tamo
4129783019
migrate the index handling code in a different file + implements the create index 2022-10-27 11:33:39 +02:00
Tamo
1804416afa
reintroduce the uuid mapping for the indexes 2022-10-27 11:33:39 +02:00
Tamo
c97d51a624
add a bunch of tests 2022-10-27 11:33:39 +02:00
Tamo
803f2157af
split the DocumentAdditionOrUpdate in two tasks; DocumentAddition and DocumentUpdate 2022-10-27 11:33:38 +02:00
Tamo
b7c5b71a53
starts importing the real tasks 2022-10-27 11:33:38 +02:00
Tamo
5cc8f96237
get rids of the auto-generated mains 2022-10-27 11:33:38 +02:00
Tamo
94e29a9f5f
extract the index abstraction out of the index-scheduler in its own module 2022-10-27 11:33:38 +02:00
Tamo
48138c21a9
rename the update-file-store to file-store since it can store any kind of file 2022-10-27 11:33:38 +02:00
Tamo
76597fc382
import the update_file_store in the index-scheduler 2022-10-27 11:33:37 +02:00
Tamo
a0588d6b94
finishes the global skelton of the auto-batcher 2022-10-27 11:33:37 +02:00
Tamo
b3c9b128d9
polish the global structure of the batch creation 2022-10-27 11:33:37 +02:00
Irevoire
448f44f631
move the autobatcher logic to another file 2022-10-27 11:33:36 +02:00
Tamo
f638774764
add the document format file 2022-10-27 11:33:36 +02:00
Tamo
516860f342
fix the create_new_batch method 2022-10-27 11:33:36 +02:00
Tamo
6b9689a1c0
fix the whole batchKind thingy 2022-10-27 11:33:36 +02:00
Tamo
af0f5d6c0c
implements most operations 2022-10-27 11:33:36 +02:00
Tamo
5a7fcf2688
fix a few typos 2022-10-27 11:33:35 +02:00
Tamo
30d2b24689
implements the index deletion, creation and swap 2022-10-27 11:33:35 +02:00
Tamo
72b2e68de4
makes the updates getters smoother to uses 2022-10-27 11:33:35 +02:00
Tamo
7879189c6b
make the project compile again 2022-10-27 11:33:35 +02:00
Tamo
46b8ebcab4
fix the file store 2022-10-27 11:33:35 +02:00
Tamo
fa742f60e8
make the file store entirely synchronous, including the file deletion 2022-10-27 11:33:35 +02:00
Tamo
a7aa92df5f
fix most of the index module 2022-10-27 11:33:34 +02:00
Irevoire
d8b8e04ad1
wip porting the index back in the scheduler 2022-10-27 11:33:34 +02:00
Irevoire
fe330e1be9
add a little bit of documentation 2022-10-27 11:33:34 +02:00
Tamo
2c4e5ce8be
implements the filter query 2022-10-27 11:33:34 +02:00
Tamo
705af94fd7
add the task to the index db in the register task 2022-10-27 11:33:34 +02:00
Tamo
ed745591e1
split the scheduler into multiples files 2022-10-27 11:33:34 +02:00
Tamo
22d24dba56
implement the get_batch method 2022-10-27 11:33:33 +02:00
Tamo
1a47949063
START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented 2022-10-27 11:33:33 +02:00