377 Commits

Author	SHA1	Message	Date
Irevoire	4e1b6b514e	update reviewer change	2022-10-27 11:35:19 +02:00
Irevoire	64e55b4db9	fix the index creation. When an index is being created we insert it in the index_map straight away to avoid someone else from trying to re-open it. The definitive fix should be made on milli's side	2022-10-27 11:35:18 +02:00
Loïc Lecrenier	1f75caae88	Fix a few index swap bugs. ... 1. Details of the indexSwap task 2. Query tasks with type=indexUid 3. Synchronous error message for multiple index not found	2022-10-27 11:35:17 +02:00
Irevoire	29bdcb880c	update the snapshot	2022-10-27 11:35:17 +02:00
Irevoire	a3fc0d3bd9	Fix the last regression	2022-10-27 11:35:17 +02:00
Kerollmops	2de8a0711a	Cargo insta test/review	2022-10-27 11:35:16 +02:00
Kerollmops	2f577b6fcd	Patch the IndexScheduler in meilisearch-http to use the options struct	2022-10-27 11:35:16 +02:00
Kerollmops	71b50853dc	Introduce an options struct to create the IndexScheduler	2022-10-27 11:35:16 +02:00
Kerollmops	7074872a78	cargo insta accept	2022-10-27 11:35:15 +02:00
Kerollmops	035e8eeff5	Clean-up some TODOs	2022-10-27 11:35:15 +02:00
Kerollmops	e35fe33712	Fix some bugs with files	2022-10-27 11:35:15 +02:00
Kerollmops	942b7c338b	Compress the snapshot in a tarball	2022-10-27 11:35:15 +02:00
Kerollmops	4cafc63561	Reintroduce the versioning functions	2022-10-27 11:35:14 +02:00
Kerollmops	89e127e4f4	Declare the auth path in the index scheduler	2022-10-27 11:35:14 +02:00
Kerollmops	eec43ec953	Implement a first version of the snapshots	2022-10-27 11:35:14 +02:00
Kerollmops	c063f154fb	Add the snapshots directory path to the IndexScheduler	2022-10-27 11:35:14 +02:00
Kerollmops	e0548e42e7	Rename the Snapshot task into SnapshotCreation	2022-10-27 11:35:14 +02:00
Kerollmops	4d43a9f5b1	Rename the index-scheduler module into insta_snapshot	2022-10-27 11:35:14 +02:00
Kerollmops	901c405919	Fix the inta-snapshot typos in the tests	2022-10-27 11:35:13 +02:00
Loïc Lecrenier	6db90ba6cc	Make sure that we don't delete or cancel future tasks ... This should already have been the case before, but there is no harm in adding another check.	2022-10-27 11:35:13 +02:00
Irevoire	e0821ad4b0	remove an useless dbg	2022-10-27 11:35:13 +02:00
Irevoire	61f0940f8c	fix an issue with the dates	2022-10-27 11:35:13 +02:00
Irevoire	241300d2d8	add more naive tests around the document addition + remove the old unused snapshot files	2022-10-27 11:35:13 +02:00
Irevoire	570b2d1167	add some naive document addition tests	2022-10-27 11:35:12 +02:00
Loïc Lecrenier	d92425658e	Add index scheduler tests for task cancelation	2022-10-27 11:35:12 +02:00
Loïc Lecrenier	16fac10074	Fix crash when batching an index swap task containing 0 swaps	2022-10-27 11:35:12 +02:00
Irevoire	0aca5e84b9	rename received_document_ids to matched_documents in the DocumentDeletion task type (reimplementation of #2826)	2022-10-27 11:35:12 +02:00
Irevoire	7ed3f00b1e	reformat	2022-10-27 11:35:12 +02:00
Irevoire	9c00b159ba	fix clippy	2022-10-27 11:35:11 +02:00
Irevoire	7e52f1effb	remove a lot of unecessary clone and ref	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	4d25c159e6	Apply code review suggestions	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	e9cd6cbbee	Revert implementation of get_status to query only the database	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	424202d773	Pause the index scheduler for one second when a fatal error occurs	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	4a35eb9849	Fix (hopefully) queries that include processing tasks	2022-10-27 11:35:11 +02:00
Loïc Lecrenier	493a8cff31	Adjust task details correctly following index swap	2022-10-27 11:35:10 +02:00
Loïc Lecrenier	4de445d386	Start testing unexpected errors and panics in index scheduler	2022-10-27 11:35:10 +02:00
Loïc Lecrenier	e3848b5f28	Add assert method to verify validity of index scheduler state	2022-10-27 11:35:10 +02:00
Irevoire	ecf4e43b3d	rename the dumpExport to dumpCreation	2022-10-27 11:35:10 +02:00
Irevoire	8a23e707c1	fix the task view and forward the task db size	2022-10-27 11:35:09 +02:00
Irevoire	eb4bdde432	fix clippy	2022-10-27 11:35:09 +02:00
Irevoire	1d04ce611d	remove ununsed function	2022-10-27 11:35:08 +02:00
Irevoire	e9055f5572	fix clippy	2022-10-27 11:35:08 +02:00
Irevoire	874499a2d2	fix all the snapshots	2022-10-27 11:35:08 +02:00
Irevoire	ecdcbf350f	update all the snapshots with the new kind name	2022-10-27 11:35:08 +02:00
Irevoire	c8ee453b6c	fix the autobatched document deletion	2022-10-27 11:35:07 +02:00
Irevoire	a8de5368e5	fix the index creation in case an index already exists	2022-10-27 11:35:07 +02:00
Irevoire	9bb2e3c790	fix the failed document addition with a primary key	2022-10-27 11:35:07 +02:00
Irevoire	8d1408c65e	fix the import of the dumpv4&v5 when there is no instance-uid + rename the Kind+KindWithContent+Details variant for the DocumentImport and the Setting	2022-10-27 11:35:05 +02:00
Irevoire	131fe30934	fix the error messages and the index stats	2022-10-27 11:35:05 +02:00
Irevoire	50386921df	fix the index creation	2022-10-27 11:35:05 +02:00
Clément Renault	32cfac0cfd	Sort the TOML dependencies	2022-10-27 11:35:05 +02:00
Clément Renault	80b2e70ee7	Introduce a rustfmt file	2022-10-27 11:35:05 +02:00
Clément Renault	2a7ef3b352	Reapply #2830	2022-10-27 11:34:18 +02:00
Clément Renault	72ec4ce96b	Fix allow_index_creation useless field	2022-10-27 11:34:17 +02:00
Clément Renault	75857bf476	Fix the insta tests	2022-10-27 11:34:17 +02:00
Irevoire	0bbf80186f	push the snapshot files	2022-10-27 11:34:17 +02:00
Irevoire	b6a0abea9f	fix the index deletion when the index doesn’t exists but would be created by one of the autobatched tasks	2022-10-27 11:34:16 +02:00
Irevoire	5303bbffab	fix the last rule about merging the allow_index_creation	2022-10-27 11:34:16 +02:00
Irevoire	fc944c39a5	simplify the code A LOT and create less false positive	2022-10-27 11:34:16 +02:00
Irevoire	a1d4cc673d	add a whole new batch of tests around the index already exists / allow_index_creation	2022-10-27 11:34:16 +02:00
Irevoire	28d9f2c041	fix all the snapshot tests	2022-10-27 11:34:16 +02:00
Irevoire	d9218578e3	it probably works but it's also horrendous	2022-10-27 11:34:16 +02:00
Loïc Lecrenier	11fee30f47	Apply review suggestions and stop using rtxn.commit	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	17cd2a4aa0	Implement POST /indexes-swap	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	28bd8b6c6b	Remove key from index_tasks database when the value is empty	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	169f386418	Add some documentation to the index scheduler	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	bdb17954d2	Fix bug where assert used != instead of == ... And update snapshot tests.	2022-10-27 11:34:15 +02:00
Loïc Lecrenier	23b01a58df	cargo fmt	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	ec3391808d	Fix date parsing for task queries ... Use rfc3339 or YYYY-MM-DD. Add a day to the parsed date when it is an excluded lower bound and the YYYY-MM-DD was used. Also the Query type does not need to be serialisable anymore	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	10a547df4f	Apply suggestions from code review ... Co-authored-by: Clément Renault <clement@meilisearch.com> Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Tamo <tamo@meilisearch.com> Apply suggestions from code review Co-authored-by: Clément Renault <clement@meilisearch.com> Co-authored-by: Tamo <tamo@meilisearch.com> Apply code review suggestion Co-authored-by: Clément Renault <clement@meilisearch.com>	2022-10-27 11:34:14 +02:00
Loïc Lecrenier	22cf0559fe	Implement task date filters ... before/after enqueued/started/finished at	2022-10-27 11:34:14 +02:00
Irevoire	5765883600	fix the auto-generated details	2022-10-27 11:34:14 +02:00
Tamo	cff003c928	remove the unused variants from the autobatcher	2022-10-27 11:34:14 +02:00
Tamo	ab8f1c2865	fix a bunch of snapshot tests	2022-10-27 11:34:13 +02:00
Kerollmops	50b8b9df6a	Delete the tasks content file once the transaction has been successfully committed	2022-10-27 11:34:13 +02:00
Kerollmops	ec0a5a9f01	Remove the useless r#union thing	2022-10-27 11:34:13 +02:00
Kerollmops	6460b78e08	Clean up the delete_persisted_task_data function	2022-10-27 11:34:13 +02:00
Kerollmops	d21651c968	Throw the error if we can't register the tasks in the store	2022-10-27 11:34:13 +02:00
Kerollmops	6e904d0997	Introduce a ProcessingTasks constructor	2022-10-27 11:34:12 +02:00
Kerollmops	b373d19831	Extract the must_stop flag out of the RwLock	2022-10-27 11:34:12 +02:00
Kerollmops	3cbfacb616	Prefer using an u64 instead of a usize in some places	2022-10-27 11:34:12 +02:00
Kerollmops	79c4275bfc	Delete the persisted data when we cancel a task	2022-10-27 11:34:12 +02:00
Kerollmops	c2ec4a089b	Put the original URL query in the tasks details	2022-10-27 11:34:12 +02:00
Kerollmops	751e9bac3b	Add the tasks cancel route to cancel tasks	2022-10-27 11:34:11 +02:00
Kerollmops	290945e258	Update the canceledBy and finishedAt fields	2022-10-27 11:34:11 +02:00
Kerollmops	725158b454	Introduce the core algorithm of task cancelation	2022-10-27 11:34:11 +02:00
Kerollmops	b2c5bc67b7	Add more enum-iterator related stuff	2022-10-27 11:34:11 +02:00
Kerollmops	591527a99d	Prefer using TaskDeletion in the dumps	2022-10-27 11:34:11 +02:00
Kerollmops	1ca9a67c49	Introduce the task cancelation task type	2022-10-27 11:34:11 +02:00
Kerollmops	703ba7a1fb	Introduce the ProcessingTasks struct	2022-10-27 11:34:10 +02:00
Loïc Lecrenier	ea60d35c71	Delete a task's persisted data when appropriate	2022-10-27 11:34:10 +02:00
Tamo	9e85f050b2	fix the tests	2022-10-27 11:34:09 +02:00
Tamo	2f748480a1	share the rtxn between the access to the tasks and to the indexes	2022-10-27 11:34:09 +02:00
Tamo	6bd6321226	dump the content of the dump tasks instead of recreating at import time with wrong API keys	2022-10-27 11:34:08 +02:00
Tamo	655705eb2b	remove useless todo	2022-10-27 11:34:08 +02:00
Tamo	83f3c5ec57	flush the dump-writer only once everything has been inserted	2022-10-27 11:34:08 +02:00
Tamo	78ce29f461	apply most style comments of the review	2022-10-27 11:34:08 +02:00
Tamo	d0e91555d1	rebase on index-scheduler	2022-10-27 11:34:08 +02:00
Tamo	e0221fc0a3	fix a synchronization bug while importing tasks	2022-10-27 11:34:07 +02:00
Tamo	ba150f2127	commit after creating an index	2022-10-27 11:34:07 +02:00
Tamo	e9295c03ce	the index-scheduler needs to wake-up after importing a dump	2022-10-27 11:34:06 +02:00
Tamo	d481669b7e	fix the content_file import	2022-10-27 11:34:06 +02:00
Tamo	dd506e5d87	stop dumping the current dumping task as enqueued so it's not looping for ever	2022-10-27 11:34:06 +02:00
Tamo	d976e680c5	first mostly working version	2022-10-27 11:34:06 +02:00
Tamo	c051166bcc	update the API a little bit	2022-10-27 11:34:05 +02:00
Tamo	72a906ae75	fix the tests	2022-10-27 11:34:05 +02:00
Tamo	b7f9c94f4a	write the dump export	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	8954b1bd1d	Fix number of deleted tasks details after duplicate task deletion	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	8defad6c38	Add task deletion tests where the same task is deleted twice	2022-10-27 11:34:05 +02:00
Loïc Lecrenier	f32b973945	Return an error when calling DELETE /tasks with an empty query	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	fbd2be2ec8	Apply suggested changes from PR review	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	441417447e	Avoid creating two read txn at the same time	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	8c6aeaada5	Update snapshot tests following git rebase that fixes a bug	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	8bb0fcd144	Finish first draft of the DELETE /tasks route	2022-10-27 11:34:04 +02:00
Loïc Lecrenier	9522b75454	Continue implementation of task deletion ... 1. Matched tasks are a roaring bitmap 2. Start implementation in meilisearch-http 3. Snapshots use meili-snap 4. Rename to TaskDeletion	2022-10-27 11:34:03 +02:00
Kerollmops	e4d461ecba	Make sure that we do not batch tasks from different indexes	2022-10-27 11:34:03 +02:00
Kerollmops	b029369653	Add a test to check different indexes autobatching	2022-10-27 11:34:03 +02:00
Kerollmops	408d00136c	Extract index creation rights and simplify the autobatcher rules	2022-10-27 11:34:03 +02:00
Kerollmops	2c24c7d403	Fix invalid import of tasks types	2022-10-27 11:34:03 +02:00
Tamo	7034803712	move the API key in meilisearch_types	2022-10-27 11:34:02 +02:00
Tamo	c192146fbe	remove an unused file	2022-10-27 11:34:02 +02:00
Tamo	2f1eb78b1d	refactor the Task a little bit	2022-10-27 11:34:02 +02:00
Tamo	510ce9fc51	start moving a lot of task types to meilisearch_types	2022-10-27 11:34:01 +02:00
Tamo	0af00f6b32	fix all the import and comment most of the dump v6	2022-10-27 11:34:01 +02:00
Tamo	667c282e19	get rids of the index crate + the document_types crate	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	9a74ea0943	Fix compiler errors related autobatching option of the index scheduler	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	eabac9676b	Fix typo and remove useless code in tests	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	ab4e649221	Apply suggestions from code review ... Co-authored-by: Tamo <tamo@meilisearch.com>	2022-10-27 11:34:00 +02:00
Loïc Lecrenier	568199fc0d	Add more task deletion tests	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	13a72f8757	Use more complete snapshot tests for the index scheduler	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	4c55c30027	Add a DetailsView type and improve index scheduler snapshots ... The DetailsView type is necessary because serde incorrectly deserialises the `Details` type, so the database fails to correctly decode Tasks	2022-10-27 11:33:59 +02:00
Loïc Lecrenier	dc81992eb2	Implement TaskDeletion in the index scheduler	2022-10-27 11:33:59 +02:00
Kerollmops	fe84f2648b	Allow a user to disable the auto batching system	2022-10-27 11:33:59 +02:00
Kerollmops	e2a766acb5	Add a test to check that it works without autobatching	2022-10-27 11:33:58 +02:00
Kerollmops	db9d1b18ca	Remove the IndexScheduler::notify method	2022-10-27 11:33:58 +02:00
Kerollmops	19c6f8303f	Make sure that the index-scheduler tick loop is rerun after processing	2022-10-27 11:33:58 +02:00
Kerollmops	b311eb3bed	Add a test that verifies that sending multiple tasks works	2022-10-27 11:33:58 +02:00
Tamo	0972587cfc	start writting the compat layer between v5 and v6	2022-10-27 11:33:52 +02:00
Tamo	e845cc2b6f	fix the tests	2022-10-27 11:33:51 +02:00
Clément Renault	123f47dbc4	Create the index only if the task has the rights to do so	2022-10-27 11:33:50 +02:00
Clément Renault	068a4b2884	Correctly batch tasks with different index creation rights	2022-10-27 11:33:50 +02:00
Clément Renault	87212cfd20	Use a ControlFlow in the autobatcher function	2022-10-27 11:33:50 +02:00
Kerollmops	f1b1cfdbcc	IndexDeletion operation have ClearAll details	2022-10-27 11:33:50 +02:00
Kerollmops	a083c9e452	Only mark the first clear document with the amount of cleared documents	2022-10-27 11:33:50 +02:00
Kerollmops	b24b13b036	Let the tick function set the Failed status itself	2022-10-27 11:33:50 +02:00
Kerollmops	566c15fb74	Fill an IndexDeletion task with the number of documents removed	2022-10-27 11:33:49 +02:00
Kerollmops	6b3b05fb73	Panic if we encountered a wring KindWithContent type	2022-10-27 11:33:49 +02:00
Kerollmops	36e5efde0d	Update the tasks statuses	2022-10-27 11:33:49 +02:00
Kerollmops	2fbdd104b8	Implement the IndexDeletion batch operation	2022-10-27 11:33:49 +02:00
Kerollmops	da363a92ac	Implement the IndexUpdate batch operation	2022-10-27 11:33:49 +02:00
Kerollmops	0543cba6eb	Implement the IndexCreate batch operation	2022-10-27 11:33:48 +02:00
Kerollmops	cf6084151b	Make sure that meilisearch-http works without index wrapper	2022-10-27 11:33:48 +02:00
Kerollmops	d76634a36c	Remove the Index wrapper and use milli::Index directly	2022-10-27 11:33:48 +02:00
Kerollmops	9e8242c57d	Remove the IndexRename operation	2022-10-27 11:33:48 +02:00
Kerollmops	5fa214abb1	Move the IndexScheduler to the root of the index-scheduler crate	2022-10-27 11:33:47 +02:00
Kerollmops	9a9e98fb77	Add a TODO about the index creation	2022-10-27 11:33:47 +02:00
Kerollmops	5d21c790ef	Make clippy happy	2022-10-27 11:33:47 +02:00
Kerollmops	31de33d5ee	Implement a recursive indexation for the index-related operations	2022-10-27 11:33:47 +02:00
Kerollmops	07286fcc79	Implement the SettingsAndDocumentImport batch operation	2022-10-27 11:33:47 +02:00
Kerollmops	f68906f5dc	Merge both DocumentAddition/Update into one DocumentImport variant	2022-10-27 11:33:46 +02:00
Kerollmops	5174c78f87	Implement the DocumentClear batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	025bb5f616	Implement the DocumentClearAndSettings batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	41ec737e73	Implement the Settings batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	7b4a913704	Implement the DocumentUpdate batch operation	2022-10-27 11:33:46 +02:00
Kerollmops	a6a1043abb	Implement the DocumentDeletion batch operation	2022-10-27 11:33:46 +02:00
Tamo	7a0f17c912	remove an old unworking part of the batch execution	2022-10-27 11:33:45 +02:00
Tamo	c2899fe9b2	bring back the IndexMeta and IndexStats in meilisearch-http	2022-10-27 11:33:45 +02:00
Tamo	2c8f1a43e9	get rids of meilisearch-lib	2022-10-27 11:33:44 +02:00
Tamo	0ba1c46e19	fix a deadlock	2022-10-27 11:33:44 +02:00
Tamo	22bfb5a7a0	remove Clone from the IndexScheduler	2022-10-27 11:33:44 +02:00
Tamo	d8d3499aec	remove a bunch of comments	2022-10-27 11:33:44 +02:00
Tamo	64e132ce53	move as many fields as possible out of the IndexScheduler	2022-10-27 11:33:44 +02:00
Tamo	9e1f38ec7c	move the test function in the test module	2022-10-27 11:33:44 +02:00
Tamo	6f4dcc0c38	start implementing some logic to test the internal states of the scheduler	2022-10-27 11:33:43 +02:00
Tamo	84cd5cef0b	fix the tests	2022-10-27 11:33:43 +02:00
Tamo	ae86a8ccd6	slightly refactor the autobatching tests	2022-10-27 11:33:43 +02:00
Tamo	ce2dfecc03	connect the new scheduler to meilisearch-http officially. ... I can index documents and do search	2022-10-27 11:33:43 +02:00
Tamo	cb4feabca2	implements the get_tasks	2022-10-27 11:33:43 +02:00
Tamo	19154e48fe	fix all compilation errors	2022-10-27 11:33:42 +02:00
Irevoire	8d51c1f389	wip integrating the scheduler in meilisearch-http	2022-10-27 11:33:42 +02:00
Irevoire	250410495c	start integrating the index-scheduler in meilisearch-lib	2022-10-27 11:33:42 +02:00
Irevoire	8f0fd35358	add insta::json for later	2022-10-27 11:33:42 +02:00
Irevoire	8770e07397	I can index documents without meilisearch	2022-10-27 11:33:42 +02:00
Tamo	edd8344dc9	wip	2022-10-27 11:33:42 +02:00
Tamo	e547552702	create the end Batch type for all Index* operations	2022-10-27 11:33:41 +02:00
Tamo	925971809a	create the end Batch type for all Document* operation	2022-10-27 11:33:41 +02:00
Tamo	1ea9c0b4c0	write most of the run loop	2022-10-27 11:33:41 +02:00
Tamo	9ff0fe952e	split the run function in two	2022-10-27 11:33:41 +02:00
Tamo	a8b18b2c96	fix the register test	2022-10-27 11:33:40 +02:00
Tamo	5436b996ab	reduce the size of the snapshots	2022-10-27 11:33:40 +02:00
Tamo	7d0c8a3379	test the register tasks	2022-10-27 11:33:40 +02:00
Tamo	fc098022c7	start integrating the index-scheduler in the meilisearch codebase	2022-10-27 11:33:40 +02:00
Tamo	b816535e33	greatly reduce the number of warnings	2022-10-27 11:33:40 +02:00
Tamo	38e4ffe73c	fix smol typo	2022-10-27 11:33:40 +02:00
Tamo	366a344474	get rids of the horrendous spinlock in favor of synchronoise	2022-10-27 11:33:39 +02:00
Tamo	7b6673dc1d	implement the index swap in the index mapper	2022-10-27 11:33:39 +02:00
Tamo	03aca2e452	move the index mapping logic in another structure	2022-10-27 11:33:39 +02:00
Tamo	4129783019	migrate the index handling code in a different file + implements the create index	2022-10-27 11:33:39 +02:00
Tamo	1804416afa	reintroduce the uuid mapping for the indexes	2022-10-27 11:33:39 +02:00
Tamo	c97d51a624	add a bunch of tests	2022-10-27 11:33:39 +02:00