Commit Graph

636 Commits

Author SHA1 Message Date
bors[bot]
89e05fc6c5
Merge #113
113: snapshots r=MarinPostma a=MarinPostma

 This pr adds support for snapshoting.

The snapshoting process for an index requires that no other update is processing at the same time. A mutex lock has been added to prevent a snapshot from occuring at the same time as an update, while still premitting updates to be pushed.

The list of the indexes to snapshot is first retrieved from the `UuidResolver` which also performs its snapshot.

This list is passed to the update store, which attempts to acquire a lock on the update store while it snaphots itself and it's associated index store.

 This means that a snapshot can only be completed once all indexes have finished their ongoing update.

This pr also adds refactoring of the code to allow unit testing and mocking, and unit test the snapshot creation.

Co-authored-by: mpostma <postma.marin@protonmail.com>
Co-authored-by: tamo <irevoire@protonmail.ch>
Co-authored-by: marin <postma.marin@protonmail.com>
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2021-04-01 13:16:00 +00:00
Marin Postma
248e9b3808
Merge remote-tracking branch 'origin/main' into snapshots 2021-04-01 15:10:33 +02:00
tamo
79c63049d7
update the settings routes 2021-04-01 11:52:26 +02:00
Irevoire
96cffeab1e
update all the response format to be ISO with meilisearch, see #64 2021-04-01 11:43:03 +02:00
Clément Renault
6e1ddfea5a
Merge pull request #129 from shekhirin/fix-docker-commit-sha
fix(ci, http): commit_sha and commit_date in docker builds
2021-03-31 21:46:17 +02:00
Marin Postma
d8af4a7202
ignore snapshot test (#130) 2021-03-31 20:07:52 +02:00
Alexey Shekhirin
3d51db5929 fix(ci, http): commit_sha and commit_date in docker builds
chore(ci): cache dependencies in Docker build
2021-03-31 13:56:28 +03:00
Clément Renault
b0956c09c1
Merge pull request #127 from shekhirin/docker-deps-cache
chore(ci): cache dependencies in Docker build
2021-03-31 12:48:57 +02:00
Alexey Shekhirin
5bc464dc53 chore(ci): cache dependencies in Docker build 2021-03-31 11:23:09 +03:00
marin
7050236a93
Merge pull request #123 from irevoire/snapshots
remove the now useless dead_code flags
2021-03-26 17:54:38 +01:00
tamo
0f2143e7fd
remove the now useless dead_code flags 2021-03-26 14:15:12 +01:00
Clémentine Urquizar
b9f79c8df0
Update display 2021-03-26 12:12:55 +01:00
mpostma
d7c077cffb
atomic snapshot import 2021-03-25 14:48:51 +01:00
mpostma
7d6ec7f3d3
resolve merge 2021-03-25 14:21:05 +01:00
mpostma
f3dc853be3
Merge remote-tracking branch 'origin/main' into snapshots 2021-03-25 13:45:07 +01:00
mpostma
48507460b2
add snapshot tests 2021-03-25 12:02:10 +01:00
mpostma
d029464de8
fix snapshot path 2021-03-25 10:23:31 +01:00
mpostma
79d09705d8
perform snapshot on startup 2021-03-25 09:35:15 +01:00
bors[bot]
868658f3d8
Merge #109
109: Make updates atomic r=curquiza a=MarinPostma

Until now, the index_uid->uuid mapping was done before the update was written to disk in the case of automatic index creation. This was an issue when the update failed, and the index would still exists in the uuid resolver.

This is fixed by this pr, by first creating the update with an uuid if the index does not exist, and then register this uuid to the uuid resolver.

This is preliminary work to the implementation of snapshots (#19).

This pr also changes the `resolve` method on the `UuidResolver` to `get` to make it clearer.


The `create_uuid` method may be bound to disappear when the index name resolution is handled by a remote machine.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-24 12:24:32 +00:00
bors[bot]
fe87477238
Merge #115
115: Add the exhaustiveNbHits in search response body (returns always false) r=curquiza a=irevoire

closes #103 

Co-authored-by: tamo <irevoire@protonmail.ch>
Co-authored-by: Irevoire <irevoire@protonmail.ch>
2021-03-24 12:16:53 +00:00
mpostma
d892a2643e
fix clippy 2021-03-24 12:38:59 +01:00
mpostma
83ffdc888a
remove bad file name test 2021-03-24 12:38:59 +01:00
mpostma
4041d9dc48
format code 2021-03-24 12:38:59 +01:00
mpostma
1f16c8d224
integration test snapshot 2021-03-24 12:38:59 +01:00
mpostma
06f9dae0f3
remove prints 2021-03-24 12:38:59 +01:00
mpostma
48d5f88c1a
fix snapshot dir already exists 2021-03-24 12:38:59 +01:00
mpostma
eb53ed4cc1
load snapshot 2021-03-24 12:38:59 +01:00
mpostma
46293546f3
add tests and mocks 2021-03-24 12:38:59 +01:00
mpostma
3cc3637e2d
refactor for tests 2021-03-24 12:38:56 +01:00
mpostma
1f51fc8baf
create indexes snapshots concurrently 2021-03-24 12:38:12 +01:00
mpostma
e9da191b7d
fix snapshot bugs 2021-03-24 12:38:12 +01:00
mpostma
d73fbdef2e
remove from snapshot 2021-03-24 12:38:12 +01:00
mpostma
44dcfe29aa
clean snapshot creation 2021-03-24 12:38:12 +01:00
mpostma
a85e7abb0c
fix snapshot creation 2021-03-24 12:38:12 +01:00
mpostma
4847884165
restore snapshots 2021-03-24 12:38:12 +01:00
mpostma
7f6a54cb12
add lock to prevent snapshot during update 2021-03-24 12:38:12 +01:00
mpostma
520f7c09ba
sequential index snapshot 2021-03-24 12:38:12 +01:00
mpostma
35a7b800eb
snapshot indexes 2021-03-24 12:38:12 +01:00
mpostma
c966b1dd94
use options to schedule snapshot 2021-03-24 12:38:11 +01:00
mpostma
ee838be41b
implement snapshot scheduler 2021-03-24 12:38:11 +01:00
Irevoire
127e944866
Update meilisearch-http/src/index/search.rs
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 19:13:22 +01:00
Irevoire
cc81aca6a4
Update meilisearch-http/src/index/search.rs
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-03-23 10:47:19 +01:00
Irevoire
46d7cedb18
Update meilisearch-http/src/index/search.rs
Co-authored-by: marin <postma.marin@protonmail.com>
2021-03-23 10:46:59 +01:00
mpostma
5f33672f0e
change payload send to use stream methods 2021-03-22 19:49:21 +01:00
mpostma
b690f1103a
fix typos 2021-03-22 19:25:56 +01:00
tamo
91089db444
add the exhaustive nb hits to be ISO, currently it's always set to false 2021-03-22 18:41:33 +01:00
bors[bot]
2f418ee767
Merge #108
108: use write senders for updates r=MarinPostma a=MarinPostma

 Use write senders to send updates to the `IndexActor`, so updates are performed sequentially on all indexes.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-03-22 14:18:43 +00:00
bors[bot]
2ecde74fa4
Merge #112
112: fix root route r=MarinPostma a=irevoire

closes #93

Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-03-22 14:08:59 +00:00
Irevoire
7ecefe37da
fix root route 2021-03-19 11:34:54 +01:00
mpostma
147756750b
create uuid on successful update addition
also change resolve to get in uuid resolver
2021-03-18 09:09:26 +01:00