81 Commits

Author	SHA1	Message	Date
Tamo	bdb51a85fe	now that the task cancelation shares their started at with all the tasks of their batch we don't need the trick of retrieving the previous batch anymore	2024-11-20 10:51:07 +01:00
Tamo	e145d71a62	implements the two last TODOs	2024-11-20 10:51:06 +01:00
Tamo	d9a4e69990	push a missing snapshot	2024-11-20 10:51:06 +01:00
Tamo	b906e3ed70	improve the way we access the mutex	2024-11-20 10:51:06 +01:00
Tamo	4abcd9c04e	add some stats on the batches	2024-11-20 10:51:06 +01:00
Tamo	229fa0f902	implements the batch details	2024-11-20 10:51:06 +01:00
Tamo	62646af7b9	implements the automatic batch deletion	2024-11-20 10:51:06 +01:00
Tamo	1fcb9526f5	fix the task cancelation	2024-11-20 10:51:06 +01:00
Tamo	15eefa4fcc	fixes a lot of small issue, the test about the cancellation is still failing	2024-11-20 10:51:05 +01:00
Tamo	ad9763ffcd	copy multiple task query tests to batches. Currently, they fails	2024-11-20 10:49:25 +01:00
Tamo	d489f5635f	add the mapping between the task and batches	2024-11-20 10:49:23 +01:00
Tamo	a1251c3c83	Implements the get all batches route with filters working	2024-11-20 10:42:55 +01:00
Tamo	6062914654	add the batch_id to the tasks	2024-11-20 10:42:54 +01:00
Louis Dureuil	bfefaf71c2	Progress displayed in logs	2024-11-19 09:32:52 +01:00
Louis Dureuil	5f93651cef	fixes	2024-11-18 16:23:11 +01:00
Louis Dureuil	1f8b01a598	Fix snap since _vectors is no longer part of the field distributions	2024-11-18 12:50:59 +01:00
Louis Dureuil	e9d17136b2	Add deadline of 3 seconds to embedding requests made in the context of hybrid search	2024-11-18 12:15:11 +01:00
Clément Renault	5b4c06c24c	Plug the grenad max memory parameter	2024-11-18 11:28:04 +01:00
Louis Dureuil	c202f3dbe2	fix tests and revert change in behavior when primary_key_from_op != primary_key_from_db && index.is_empty()	2024-11-18 10:59:05 +01:00
Clément Renault	677d7293f5	Fix a lot of primary key related tests	2024-11-18 10:59:05 +01:00
Clément Renault	bd31ea2174	Check for at least one valid task after setting their statuses	2024-11-18 10:59:05 +01:00
Clément Renault	83865d2ebd	Expose intermediate errors when processing batches	2024-11-18 10:59:05 +01:00
Clément Renault	9e8367f1e6	Move the rayon thread pool outside the extract method	2024-11-14 10:40:32 +01:00
Louis Dureuil	1fcd5f091e	Remove progress from task	2024-11-12 12:23:13 +01:00
Louis Dureuil	e32677999f	Adapt some snapshots	2024-11-08 00:06:33 +01:00
Louis Dureuil	8a314ab81d	Fix primary key fid order	2024-11-08 00:05:12 +01:00
Tamo	2eb1801e85	reverse the order of the task queue	2024-11-07 19:19:44 +01:00
Louis Dureuil	ee03743355	Merge branch 'indexer-edition-2024' into indexer-edition-2024-doc-chunks	2024-11-06 15:50:53 +01:00
ManyTheFish	10feeb88f2	Merge branch 'main' into indexer-edition-2024	2024-11-06 15:19:18 +01:00
Tamo	cf6ad1ae5e	Merge branch 'main' into tmp-release-v1.11.0	2024-11-04 16:14:44 +01:00
Clément Renault	9c1e54a2c8	Move crates under a sub folder to clean up the code	2024-10-21 08:18:43 +02:00