432 Commits

Author	SHA1	Message	Date
Tamo	a6863a7d6e	use the zookeeper-client-sync from meilisearch instead of tamo	2023-11-08 16:21:53 +01:00
Tamo	1585b3ed2f	re-implement the snapshot import at startup	2023-11-07 14:15:57 +01:00
Tamo	c48f72e6b8	fix a few warnings	2023-11-06 16:32:49 +01:00
Tamo	41178e59fc	fix the leader election	2023-11-06 15:00:14 +01:00
Tamo	d1bc7ec58a	fix all the bugs on the snapshots export/import pipeline	2023-11-06 12:34:48 +01:00
Tamo	c5ec817f52	WIP fix a lot of bugs: The follower do not wake up when a new snapshot is available	2023-11-02 18:21:17 +01:00
Tamo	0c18962b13	it compiles, now tokio is complaining that we block the main thread (and is right), I need to patch the zookeeper client to not rely on tokio anymore	2023-11-02 12:55:16 +01:00
Tamo	366144146b	WIT: it compiles but the processing of tasks and loading of snapshots is still not implemented	2023-11-02 10:47:54 +01:00
Tamo	03b510945b	ICE: use a git version of my crate so rust team can pull the repo	2023-10-31 17:59:59 +01:00
Tamo	c3a8d4b7fb	ICE	2023-10-31 17:39:10 +01:00
Tamo	c573261ac4	WIP: start updating the zookeeper client => leader election is missing	2023-10-31 13:04:32 +01:00
Tamo	dfb84f80da	bump strois version	2023-10-10 19:25:12 +02:00
Tamo	98b67f217a	move to our new S3 lib	2023-09-28 11:24:18 +02:00
Tamo	ecd36b15f0	exposes all the s3 arguments	2023-09-13 18:17:56 +02:00
Clément Renault	8a2e8a887f	Load the latest snapshot when we start the engine	2023-09-12 18:08:24 +02:00
Clément Renault	309c33a418	Fix again the dots	2023-09-12 17:55:01 +02:00
Clément Renault	9b01506cee	Move the load snapshot step into a function	2023-09-12 16:05:02 +02:00
Clément Renault	f37fdceb15	Use slashes instead of dots for the s3 paths separators	2023-09-12 15:46:15 +02:00
Clément Renault	f544cfa444	Remove tasks and content file on the s3	2023-09-12 15:19:45 +02:00
Kerollmops	a53a0fdb77	Store content files into the S3	2023-09-11 18:17:22 +02:00
Clément Renault	719fdd701b	Fix and crash when the tasks path is unknown	2023-09-07 11:31:18 +02:00
Kerollmops	01c13c98ac	Mastering minio	2023-09-06 17:54:21 +02:00
Tamo	5b89276fcc	starts using s3	2023-09-05 19:25:09 +02:00
Kerollmops	41697c4d65	Introduce the zk-tasks folder	2023-09-04 18:24:34 +02:00
Kerollmops	7d85753573	Make the snapshot download work	2023-09-04 17:38:56 +02:00
Kerollmops	76657af1f9	Add the options into the IndexScheduler	2023-09-04 16:38:05 +02:00
Tamo	966cbdab69	make the tests compile again	2023-09-04 15:39:54 +02:00
Clément Renault	0c68b9ed4c	WIP making the final snapshot swap	2023-08-31 15:56:42 +02:00
Clément Renault	d7233ecdb8	Make things to compile again	2023-08-31 14:55:14 +02:00
Clément Renault	95a011af13	Wrap the IndexScheduler fields into an inner struct	2023-08-31 10:36:33 +02:00
Clément Renault	e257710961	WIP fix the tests	2023-08-30 18:03:24 +02:00
Clément Renault	8c3ad57ef9	React to changes towards the cluster members	2023-08-30 17:40:12 +02:00
Clément Renault	2d1434da81	Keep the ZK flow when enqueuing tasks	2023-08-30 17:15:15 +02:00
Clément Renault	c488a4a351	Fixup a lot of small issues on the ZK config	2023-08-30 16:42:55 +02:00
Kerollmops	0c7d7c68bc	WIP moving to the sync zookeeper API	2023-08-30 15:06:12 +02:00
Tamo	854745c670	wip: starts working on importing the snapshots	2023-08-16 18:41:05 +02:00
Tamo	777eebb759	starts creating snapshot, the import is still missing	2023-08-10 15:00:25 +02:00
Tamo	61ccfaf9bc	wake up after registering a task	2023-08-10 09:39:39 +02:00
Tamo	f0c4d36ff7	implement the deletion of tasks after processing a batch ... add a lot of comments and logs	2023-08-10 09:36:43 +02:00
Tamo	8c20d6e2fe	fix the leader election	2023-08-09 17:23:13 +02:00
ManyTheFish	8e437ed76c	Start leader election and task processing (WIP)	2023-08-09 16:52:38 +02:00
Tamo	1191ec5939	fix the register task watcher	2023-08-08 13:18:55 +02:00
Tamo	0d20d08daf	fix a few warnings	2023-08-08 11:39:48 +02:00
ManyTheFish	b66bf049b5	Create a task on zookeeper side when a task is created locally	2023-08-07 17:02:51 +02:00
ManyTheFish	b45c36cd71	Merge branch 'main' into tmp-release-v1.3.0	2023-08-01 15:05:17 +02:00
Kerollmops	eef95de30e	First iteration on exposing puffin profiling	2023-07-18 17:38:13 +02:00
Clément Renault	22762808ab	Fix the tests	2023-07-06 12:13:29 +02:00
Clément Renault	86b834c9e4	Display the total number of tasks in the tasks route	2023-07-06 10:05:18 +02:00
meili-bors[bot]	aae099e330	Merge #3851 ... 3851: Expose lastUpdate and isIndexing in /stats endpoint r=dureuill a=gentcys # Pull Request ## Related issue Fixes #3843 ## What does this PR do? - expose lastUpdate in `/stats` endpoint - expose isIndex in `stats` endpoint - add a method `is_task_processing` in index-scheduler/src/lib.rs. ## PR checklist Please check if your PR fulfills the following requirements: - [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)? - [x] Have you read the contributing guidelines? - [x] Have you made sure that the title is accurate and descriptive of the changes? Thank you so much for contributing to Meilisearch! Co-authored-by: Cong Chen <cong.chen@ocrlabs.com> Co-authored-by: ManyTheFish <many@meilisearch.com> Co-authored-by: Louis Dureuil <louis@meilisearch.com>	2023-07-03 13:41:04 +00:00
ManyTheFish	71500a4e15	Update tests	2023-07-03 11:20:43 +02:00