This commit is a fix to issue #1750.
As a part of the changes to solve this issue, the following changes have
been made -
1. Route registration for static assets has been modified
2. the `mut` keyword on the `scope` has been removed.
1711: MeiliSearch refactor introducing OBKV format r=MarinPostma a=MarinPostma
This PR refactor some multiple components of meilisearch, and introduce the obkv document format to meilisearch
- [x] Split meilisearch-http and meilisearch-lib
- [x] Replace `IndexActor` and `UuidResolver` with `IndexResolver`
- [x] Remove mentions to Actor
- [x] Remove Actor traits to simplify code
- [x] Integrate obkv document format
- [x] Remove `Data`
- [x] Restore all route
- [x] Replace `Box<dyn error>` with `anyhow::Error`
- [x] Introduce update file store
- [x] Update file store error handling
- [x] Fix dumps
- [x] Fix snapshots
- [x] Fix tests
- [x] Update module documentation
- [x] add csv suppport (feat `@ManyTheFish` #1729 )
- [x] add jsonl support
- [x] integrate geosearch (feat `@irevoire` #1725)
partially implements #1691 and #1690. The error handling is very basic now, I will finish it in the next pr.
Some unit tests have been disabled, I will re-enable them ASAP, but they need a bit more work.
close#1531
P.S: sorry for this monstrous PR :'(
Co-authored-by: mpostma <postma.marin@protonmail.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: many <maxime@meilisearch.com>
`jemallocator` has been abandoned for nearly two years, and `rustc`
itself moved to use `tikv-jemallocator` instead:
3965773ae7
Let's switch to a better maintained version.
1651: Use reset_sortable_fields r=Kerollmops a=shekhirin
Resolves https://github.com/meilisearch/MeiliSearch/issues/1635
1676: Add curl binary to final stage image r=curquiza a=ook
Reference: #1673
Changes: * add `curl` binary to final docker Melisearch image.
For metrics, docker funny layer management makes this add a shrink from 319MB to 315MB:
```
☁ MeiliSearch [feature/1673-add-curl-to-docker-image] ⚡ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
getmeili/meilisearch 0.22.0_ook_1673 938e239ad989 2 hours ago 315MB
getmeili/meilisearch latest 258fa3aa1230 6 days ago 319MB
```
1684: bump dependencies r=MarinPostma a=MarinPostma
Bump meilisearch dependencies.
We still depend on custom patch that have been upgraded along the way.
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Thomas Lecavelier <thomas@followanalytics.com>
Co-authored-by: mpostma <postma.marin@protonmail.com>
1682: Change the format of custom ranking rules when importing old dumps r=curquiza a=Kerollmops
This PR changes the format of the custom ranking rules from `asc(price)` to `title:asc` as the format changed between v0.21 and v0.22. The dumps are now correctly importing the custom ranking rules.
This PR also change the previous default ranking rules (without sort) to the new default ranking rules (with the new sort).
Co-authored-by: Kerollmops <clement@meilisearch.com>
1652: Remove dependabot r=MarinPostma a=curquiza
Fixes#1649
Dependabot for vulnerability and security updates is still activated.
1654: Add Script for Windows r=MarinPostma a=singh08prashant
fixes#1570
changes:
1. added script for detecting windows os running git bash
2. appended `.exe` to `$release_file` for windows as listed [here](https://github.com/meilisearch/MeiliSearch/releases/)
3. removed global `$BINARY_NAME='meilisearch'` as windows require `.exe` file
1657: Bring vergen hotfix from `stable` to `main` r=MarinPostma a=curquiza
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: singh08prashant <singh08prashant@gmail.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
1540: Update milli to version 0.8.1 r=curquiza a=curquiza
Integrates this fix into MeiliSearch https://github.com/meilisearch/milli/pull/296
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
1539: Use serdeval for validating json format. r=curquiza a=MarinPostma
uses [serdeval](https://github.com/MarinPostma/serdeval) to validate that the json payload is valid json, and in the correct format.
fix#1535
Co-authored-by: mpostma <postma.marin@protonmail.com>
1498: Show the filterable and not the faceted attributes in the settings r=Kerollmops a=Kerollmops
Fixes#1497
Co-authored-by: Clément Renault <clement@meilisearch.com>
1478: refactor routes r=irevoire a=MarinPostma
refactor the route directory, so the module tree follows the route structure
Co-authored-by: mpostma <postma.marin@protonmail.com>
1457: Hotfix highlight on emojis panic r=Kerollmops a=ManyTheFish
When the highlight bound is in the middle of a character
or if we are out of bounds, we highlight the complete matching word.
note: we should enhance the tokenizer and the Highlighter to match char indices.
Fix#1368
Co-authored-by: many <maxime@meilisearch.com>
When the highlight bound is in the middle of a character
or if we are out of bounds, we highlight the complete matching word.
note: we should enhance the tokenizer and the Highlighter to match char indices.
Fix#1368
259: Run rustfmt one the whole project and add it to the CI r=curquiza a=irevoire
Since there is currently no other PR modifying the code, I think it's a good time to reformat everything and add rustfmt to the ci.
Co-authored-by: Tamo <tamo@meilisearch.com>
258: Use rustls instead of openssl r=curquiza a=irevoire
I also removed all the `default-features` of reqwest since we are only using the JSON one.
Fix#255
Co-authored-by: Tamo <tamo@meilisearch.com>
246: Stop logging the no space left on device error r=curquiza a=irevoire
closes#208
@qdequele what do you think of that?
Are there any other errors we need to ignore?
As you can see in the code, once we are in `Sentry` the error has already been converted to a `String` so the only thing we can do to see if we need to send the error or not is to match the `String` against our error message.
If we have a lot of other logs we want to ignore I would suggest prefixing all the logs with something like:
```
User error: No space left on device
```
So in Sentry, we could just check if the log start by `User error:` and ignore all these errors at once
Co-authored-by: Tamo <tamo@meilisearch.com>
249: Use half of the computer threads for the indexing process by default r=Kerollmops a=irevoire
closes#241
By default, we use only half of the CPU threads when indexing documents; this allows the user to use the search while indexing. Also, the machine will not appear unresponsive when indexing a large batch of documents.
On the special case where a user only has one core, we use it entirely 😄
Co-authored-by: Tamo <tamo@meilisearch.com>
240: Rework error messages r=irevoire a=MarinPostma
Simplify the error messages, and make them more compliant with legacy Meilisearch.
Basically, stop composing the messages, and simply forward the message of inner errors.
Co-authored-by: marin postma <postma.marin@protonmail.com>
232: Fix payload size limit r=MarinPostma a=MarinPostma
Fix#223
This was due to the fact that Payload ignores the limit payload size limit. I fixed it by implementing my own `Payload` extractor that checks that the size of the payload is not too large.
I also refactored the `create_app` a bit.
Co-authored-by: marin postma <postma.marin@protonmail.com>
227: improve mini dashboard routing r=MarinPostma a=MarinPostma
The dependency we use to statically serve the mini-dashboard used globing to serve the mini-dashboard files. This caused all unfound routes to be caught by the "/" serving the dashboard assets. This fix makes it so that the assets have a dedicated route, and any unfound route is caught by the default service and return a 404.
Co-authored-by: marin postma <postma.marin@protonmail.com>
226: Make facetsDistribution name iso r=MarinPostma a=curquiza
Even if there is an English mistake in `facets_distribution` (because of the `s`) @gmourier asked me to keep the typo: the name of `facetsDistribution` might change completely in the future, he wants to avoid two breakings.
@gmourier can you confirm before we merge this PR?
Sorry I left this update in the code (I'm confused because no issues was open to update `facetsDistribution`), there might have been a confusion with `fieldsDistribution` that has been renamed into `fieldDistribution`. Sorry!
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
213: Implement all the CLI options r=MarinPostma a=irevoire
closes#206
And I looked into #204, I fixed some default values and tried to test as many options as possible, and I think the cli is already mostly working.
If someone knows any issues about it, I would like to hear more 🙂
Co-authored-by: Tamo <tamo@meilisearch.com>
211: fix index deletion race condition r=MarinPostma a=MarinPostma
Make update store block if the currently processing update is from an index we are trying to delete. This ensure that no write to the index can occur after it has been deleted.
218: Update milli version to v0.5.0 r=MarinPostma a=curquiza
Co-authored-by: marin postma <postma.marin@protonmail.com>
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
210: Error handling r=MarinPostma a=MarinPostma
This pr implements the error handling for meilisearch.
Rather than grouping errors by types, this implementation groups them by scope, each scope enclosing errors from a scope further down, or new errors within this scope. This makes the tracking of the origins of errors easier , and error handling easier at the module level.
All errors that are eventually returned to the user implement the `Into<ResponseError>` trait. `ReponseError` in turn implements the `ErrorCode` trait from `meilisearch-error`.
Some new errors have been introduced with the new engine for which we haven't defined error codes yet. It has been decided with @gmourier that those would return the `internal-error` code until the correct error code is specified.
Co-authored-by: marin postma <postma.marin@protonmail.com>
209: Integrate amplitude r=MarinPostma a=irevoire
And merge the sentry and amplitude usage under one “Enable analytics” flag
closes#180
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
197: Update milli (v0.3.1) with filterable attributes r=MarinPostma a=curquiza
Fixes#187 and #70
Also fixes#195
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
144: Concurrent update run loop (refactor) r=MarinPostma a=MarinPostma
This PR allows multiple request to the update store to be performed concurently (i.e, one can list updates while an updates in being written to the update store).
173: Convert UpdateStatus to legacy meilisearch format r=MarinPostma a=MarinPostma
Returns the update statuses with the same format as legacy meilisearch.
The number of documents in a document addition/deletion is not known before processing, so it is only returned when the update is `processed`.
close#78
associated milli PR: https://github.com/meilisearch/milli/pull/178
Co-authored-by: marin postma <postma.marin@protonmail.com>
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
179: Enable filter paramater during search r=MarinPostma a=MarinPostma
This pr makes the necessary changes to transplant in accordance with the specification on filters.
More precisely, it:
- Removes the `filters` parameter
- Renames `facetFilters` to `filter`
- Allows either a string or an array to be passed to the filter param.
It doesn't allow the mixed syntax, that needs to be handled by milli.
close#81close#140
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
172: Fix cors authentication issue r=MarinPostma a=MarinPostma
The error was due to the middleware returning an error, instead of a response containing the error.
close#110
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
143: Shared update store r=irevoire a=MarinPostma
This PR changes the updates process so that only one instance of an update store is shared among indexes.
This allows updates to always be processed sequentially without additional synchronization, and fixes the bug where all the first pending update for each index were reported as processing whereas only one was.
EDIT:
I ended having to rewrite the whole `UpdateStore` to allow updates being really queued and processed sequentially in the ordered they were added. For that purpose I created a `pending_queue` that orders the updates by a global update id.
To find the next `update_id` to use, both globally and for each index, I have created another database that contains the next id to use.
Finally, all updates that have been processed (with success or otherwise) are all stores in an `updates` database.
The layout for the keys of these databases are such that it is easy to iterate over the elements for a particular index, and greatly reduces the amount of code to do so, compared to the former implementation.
I have also simplified the locking mechanism for the update store, thanks to the StateLock data structure, that allow both an arbitrary number of readers and a single writer to concurrently access the state. The current state can be either Idle, Processing, or Snapshotting. When an update or snapshotting is ongoing, the process holds the state lock until it is done processing its task. When it is done, it sets bask the state to Idle.
I have made other small improvements here and there, and have let some other for work, such as:
- When creating an update file to hold a request's content, it would be preferable to first create a temporary file, and then atomically persist it when we have written to it. This would simplify the case when there is no data to be written to the file, since we wouldn't have to take care about cleaning after ourselves.
- The logic for content validation must be factored.
- Some more tests related to error handling in the process_pending_update function.
- The issue #159close#114
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
