137 Commits

Author	SHA1	Message	Date
Clément Renault	134634d99a	Correctly batch tasks with different index creation rights	2022-10-20 15:44:35 +02:00
Clément Renault	cea19f5af1	Use a ControlFlow in the autobatcher function	2022-10-20 15:44:34 +02:00
Kerollmops	2a45d24143	IndexDeletion operation have ClearAll details	2022-10-20 15:44:34 +02:00
Kerollmops	be192f281d	Only mark the first clear document with the amount of cleared documents	2022-10-20 15:44:34 +02:00
Kerollmops	99d1e9a870	Let the tick function set the Failed status itself	2022-10-20 15:44:34 +02:00
Kerollmops	945cb00fc3	Fill an IndexDeletion task with the number of documents removed	2022-10-20 15:44:34 +02:00
Kerollmops	b7ad050737	Panic if we encountered a wring KindWithContent type	2022-10-20 15:44:34 +02:00
Kerollmops	f3f6bbf624	Update the tasks statuses	2022-10-20 15:44:33 +02:00
Kerollmops	88a0ed9736	Implement the IndexDeletion batch operation	2022-10-20 15:44:33 +02:00
Kerollmops	4a53cb1c50	Implement the IndexUpdate batch operation	2022-10-20 15:44:33 +02:00
Kerollmops	617994a9d7	Implement the IndexCreate batch operation	2022-10-20 15:44:33 +02:00
Kerollmops	4871509507	Make sure that meilisearch-http works without index wrapper	2022-10-20 15:44:33 +02:00
Kerollmops	478e3f0f6b	Remove the Index wrapper and use milli::Index directly	2022-10-20 15:43:03 +02:00
Kerollmops	8bffe41886	Remove the IndexRename operation	2022-10-20 15:43:02 +02:00
Kerollmops	257651c615	Move the IndexScheduler to the root of the index-scheduler crate	2022-10-20 15:43:02 +02:00
Kerollmops	78e730ca37	Add a TODO about the index creation	2022-10-20 15:43:02 +02:00
Kerollmops	373995dca9	Make clippy happy	2022-10-20 15:43:02 +02:00
Kerollmops	3fda1942c0	Implement a recursive indexation for the index-related operations	2022-10-20 15:43:02 +02:00
Kerollmops	e7ca5efe1e	Implement the SettingsAndDocumentImport batch operation	2022-10-20 15:43:01 +02:00
Kerollmops	e24715abe5	Merge both DocumentAddition/Update into one DocumentImport variant	2022-10-20 15:43:01 +02:00
Kerollmops	9e1d7fbfe5	Implement the DocumentClear batch operation	2022-10-20 15:43:01 +02:00
Kerollmops	eb8dda1f81	Implement the DocumentClearAndSettings batch operation	2022-10-20 15:43:01 +02:00
Kerollmops	ad0d285f12	Implement the Settings batch operation	2022-10-20 15:43:01 +02:00
Kerollmops	e84119fc75	Implement the DocumentUpdate batch operation	2022-10-20 15:43:01 +02:00
Kerollmops	78922a7c67	Implement the DocumentDeletion batch operation	2022-10-20 15:43:00 +02:00
Tamo	5cd1fc0b7b	remove an old unworking part of the batch execution	2022-10-20 15:42:45 +02:00
Tamo	0d0b2a9ac1	bring back the IndexMeta and IndexStats in meilisearch-http	2022-10-20 15:42:45 +02:00
Tamo	2d31cff082	get rids of meilisearch-lib	2022-10-20 15:42:44 +02:00
Tamo	2de8f08517	fix a deadlock	2022-10-20 15:36:10 +02:00
Tamo	972dd6bcef	remove Clone from the IndexScheduler	2022-10-20 15:36:10 +02:00
Tamo	7f1a85d443	remove a bunch of comments	2022-10-20 15:36:10 +02:00
Tamo	e43d5e9abe	move as many fields as possible out of the IndexScheduler	2022-10-20 15:36:09 +02:00
Tamo	a43fbc9367	move the test function in the test module	2022-10-20 15:36:09 +02:00
Tamo	03de8669a2	start implementing some logic to test the internal states of the scheduler	2022-10-20 15:36:09 +02:00
Tamo	2d1cc1a12d	fix the tests	2022-10-20 15:36:09 +02:00
Tamo	f58d969899	slightly refactor the autobatching tests	2022-10-20 15:34:55 +02:00
Tamo	42f5c1fc3f	connect the new scheduler to meilisearch-http officially. ... I can index documents and do search	2022-10-20 15:34:55 +02:00
Tamo	f84cbee170	implements the get_tasks	2022-10-20 15:34:55 +02:00
Tamo	5b282acb7b	fix all compilation errors	2022-10-20 15:34:55 +02:00
Irevoire	60ee1f5e64	wip integrating the scheduler in meilisearch-http	2022-10-20 15:34:55 +02:00
Irevoire	acc6d3a82b	start integrating the index-scheduler in meilisearch-lib	2022-10-20 15:30:31 +02:00
Irevoire	01847a14bb	add insta::json for later	2022-10-20 15:30:31 +02:00
Irevoire	16eab7b337	I can index documents without meilisearch	2022-10-20 15:30:20 +02:00
Tamo	7bbd75e1fb	wip	2022-10-20 15:29:48 +02:00
Tamo	a4bf859779	create the end Batch type for all Index* operations	2022-10-20 15:29:48 +02:00
Tamo	72496122c3	create the end Batch type for all Document* operation	2022-10-20 15:29:48 +02:00
Tamo	d7146d7b21	write most of the run loop	2022-10-20 15:29:48 +02:00
Tamo	0acc40b222	split the run function in two	2022-10-20 15:28:51 +02:00
Tamo	9069439a14	fix the register test	2022-10-20 15:28:51 +02:00
Tamo	e7c552a0d0	reduce the size of the snapshots	2022-10-20 15:28:50 +02:00
Tamo	cfc01e4372	test the register tasks	2022-10-20 15:28:50 +02:00
Tamo	f84ced7e38	start integrating the index-scheduler in the meilisearch codebase	2022-10-20 15:28:50 +02:00
Tamo	9882b7fa57	greatly reduce the number of warnings	2022-10-20 15:26:57 +02:00
Tamo	1a4461cd3a	fix smol typo	2022-10-20 15:26:57 +02:00
Tamo	ce0ebf9381	get rids of the horrendous spinlock in favor of synchronoise	2022-10-20 15:26:57 +02:00
Tamo	3001ed3a5f	implement the index swap in the index mapper	2022-10-20 15:26:46 +02:00
Tamo	c326613e06	move the index mapping logic in another structure	2022-10-20 15:26:45 +02:00
Tamo	48e10176b2	migrate the index handling code in a different file + implements the create index	2022-10-20 15:26:45 +02:00
Tamo	f7357fc504	reintroduce the uuid mapping for the indexes	2022-10-20 15:26:45 +02:00
Tamo	5b485e309c	add a bunch of tests	2022-10-20 15:26:45 +02:00
Tamo	3120057039	split the DocumentAdditionOrUpdate in two tasks; DocumentAddition and DocumentUpdate	2022-10-20 15:26:33 +02:00
Tamo	00f13f45b6	starts importing the real tasks	2022-10-20 15:26:32 +02:00
Tamo	bbb50d1b96	get rids of the auto-generated mains	2022-10-20 15:26:32 +02:00
Tamo	cf8c4310ab	extract the index abstraction out of the index-scheduler in its own module	2022-10-20 15:26:32 +02:00
Tamo	5abb79f149	rename the update-file-store to file-store since it can store any kind of file	2022-10-20 15:26:22 +02:00
Tamo	5dcd67c597	import the update_file_store in the index-scheduler	2022-10-20 15:26:22 +02:00
Tamo	98bb742d77	finishes the global skelton of the auto-batcher	2022-10-20 15:26:06 +02:00
Tamo	1efccb1bc4	polish the global structure of the batch creation	2022-10-20 15:26:06 +02:00
Irevoire	78ca4bf608	move the autobatcher logic to another file	2022-10-20 15:26:06 +02:00
Tamo	66397bd0de	add the document format file	2022-10-20 15:26:06 +02:00
Tamo	0790a3cbbf	fix the create_new_batch method	2022-10-20 15:26:06 +02:00
Tamo	074dee59ec	fix the whole batchKind thingy	2022-10-20 15:26:05 +02:00
Tamo	ef4594c078	implements most operations	2022-10-20 15:26:05 +02:00
Tamo	9aa31cd391	fix a few typos	2022-10-20 15:26:05 +02:00
Tamo	b1f0431ab4	implements the index deletion, creation and swap	2022-10-20 15:26:05 +02:00
Tamo	fa27485070	makes the updates getters smoother to uses	2022-10-20 15:26:05 +02:00
Tamo	7c56fcfef3	make the project compile again	2022-10-20 15:26:05 +02:00
Tamo	d87b769469	fix the file store	2022-10-20 15:26:04 +02:00
Tamo	faac665d58	make the file store entirely synchronous, including the file deletion	2022-10-20 15:26:04 +02:00
Tamo	fda5ca60bd	fix most of the index module	2022-10-20 15:26:04 +02:00
Irevoire	e7b2b9306a	wip porting the index back in the scheduler	2022-10-20 15:25:55 +02:00
Irevoire	a2587149f4	add a little bit of documentation	2022-10-20 15:25:42 +02:00
Tamo	f359fcc220	implements the filter query	2022-10-20 15:25:41 +02:00
Tamo	c1eda13e64	add the task to the index db in the register task	2022-10-20 15:25:41 +02:00
Tamo	2b77d54412	split the scheduler into multiples files	2022-10-20 15:25:41 +02:00
Tamo	ff5d6d1703	implement the get_batch method	2022-10-20 15:25:41 +02:00
Tamo	5a9ac4ceb3	START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented	2022-10-20 15:25:41 +02:00