2410: Make dump a task r=Kerollmops a=MarinPostma
This PR transforms the dump task into a proper task.
The `GET /dumps/:dump_uid` is removed.
Some changes were made to make this work, and a bit a refactoring was necessary.
- The `dump_actor` module has been renamed do `dumps` and moved to the root
- There isn't a `DumpActor` anymore, and the dump process is handled by the `DumpHandler`.
- The `TaskPerformer` is renamed to `BatchHandler`
- The `BatchHandler` trait no longer has a `perform_job` method, but instead has a `accept` method returning whether a handler can proccess a batch
- The scheduler now accept a list of `BatchHandler`, and iterates trhough them until it finds one to accept the current batch.
- `Job` doesn't exist anymore, and everything in now inside of the `BatchContent` enum.
- The `Vec<TaskId>` from `Batch` is replaced with a `BatchContent` enum which hints at the content.
- The Scheduler is slightly modified to accept batch, and prioritize them before regular tasks.
- The `TaskList` are not identified by a `String` representing the index uid anymore, but by a `TaskListIdentifier` which also works for dumps which are not targeting any specific indexes.
- The `GET /dump/:dump_id` no longer exists
- `DumpActorError` is renamed to `DumpError`
close#2410
Co-authored-by: ad hoc <postma.marin@protonmail.com>
2357: chore(dump): add dump tests r=Kerollmops a=irevoire
Add tests on the import of dump v1, v2, v3 and v4.
Since the dumps are slow to decompress, I made the `flate2` crate always compile in optimized.
And since they're also slow to index, I also made the `milli` crate always compile in optimized. What do you think of this `@MarinPostma?`
Should we keep milli unoptimized in case it could help us debug some things? 👀
Co-authored-by: Tamo <tamo@meilisearch.com>
2316: Add version flag r=Kerollmops a=sanders41
# Pull Request
## What does this PR do?
Fixes#2315
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Paul Sanders <psanders1@gmail.com>
2313: fix(search): remove the back and forth between the IndexMap and the serde_json::Map r=irevoire a=irevoire
This is ok because we're using the preserve_order feature in serde_json which is already internally using an IndexMap.
See https://github.com/meilisearch/meilisearch/pull/2298#discussion_r845228412_
Co-authored-by: Tamo <tamo@meilisearch.com>
2297: Feat(Search): Enhance formating search results r=ManyTheFish a=ManyTheFish
Add new settings and change crop_len behavior to count words instead of characters.
- [x] `highlightPreTag`
- [x] `highlightPostTag`
- [x] `cropMarker`
- [x] `cropLength` count word instead of chars
- [x] `cropLength` 0 is now considered as no `cropLength`
- [ ] ~smart crop finding the best matches interval~ (postponed)
Partially fixes #2214. (no smart crop)
Co-authored-by: ManyTheFish <many@meilisearch.com>
2277: fix(http): fix panic when sending document update without content type header r=MarinPostma a=MarinPostma
I found a panic when pushing documents without a content-type. This fixes is by returning unknown instead of crashing.
Co-authored-by: ad hoc <postma.marin@protonmail.com>
2207: Fix: avoid embedding the user input into the error response. r=Kerollmops a=CNLHC
# Pull Request
## What does this PR do?
Fix#2107.
The problem is meilisearch embeds the user input to the error message.
The reason for this problem is `milli` throws a `serde_json: Error` whose `Display` implementation will do this embedding.
I tried to solve this problem in this PR by manually implementing the `Display` trait for `DocumentFormatError` instead of deriving automatically.
<!-- Please link the issue you're trying to fix with this PR, if none then please create an issue first. -->
## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to Meilisearch!
Co-authored-by: Liu Hancheng <cn_lhc@qq.com>
Co-authored-by: LiuHanCheng <2463765697@qq.com>
2281: Hard limit the number of results returned by a search r=Kerollmops a=Kerollmops
This PR fixes#2133 by hard-limiting the number of results that a search request can return at any time. I would like the guidance of `@MarinPostma` to test that, should I use a mocking test here? Or should I do anything else?
I talked about touching the _nb_hits_ value with `@qdequele` and we concluded that it was not correct to do so.
Could you please confirm that it is the right place to change that?
Co-authored-by: Kerollmops <clement@meilisearch.com>
2253: refactor authentication key extraction r=ManyTheFish a=MarinPostma
I am concerned that the part of the code that performs the key prefix extraction from the jwt token migh be misused in the future. Since this is a critical part of the code, I moved it into it's own function. Since we deserialized the payload twice anyway, I reordered the verifications, and we now use the data from the validated token.
Co-authored-by: ad hoc <postma.marin@protonmail.com>
2245: Add test to validate cli r=irevoire a=MarinPostma
followup on #2242 and #2243
Add a test to make sure the cli is valid, and add a CI task to run the tests in debug to make sure we hit debug assertions.
FYI `@curquiza,` because of CI changes
Co-authored-by: ad hoc <postma.marin@protonmail.com>
2173: chore(all): replace chrono with time r=irevoire a=irevoire
Chrono has been unmaintained for a few month now and there is a CVE on it.
Also I updated all the error messages related to the API key as you can see here: https://github.com/meilisearch/specifications/pull/114fix#2172
Co-authored-by: Irevoire <tamo@meilisearch.com>
2098: feat(dump): Provide the same cli options as the snapshots r=MarinPostma a=irevoire
Add two cli options for the dump:
- `--ignore-missing-dump`
- `--ignore-dump-if-db-exists`
Fix#2087
Co-authored-by: Tamo <tamo@meilisearch.com>
2099: feat(analytics): Set the timestamp of the aggregated event as the first aggregate r=MarinPostma a=irevoire
2108: meta(auth): Enhance tests on authorization r=MarinPostma a=ManyTheFish
Enhance auth tests in order to be able to add new actions without changing tests.
Helping #2080
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
2101: chore(all): update actix-web dependency to 4.0.0-beta.21 r=MarinPostma a=robjtede
# Pull Request
## What does this PR do?
I don't expect any more breaking changes to Actix Web that will affect Meilisearch so bump to latest beta.
Fixes #N/A?
<!-- Please link the issue you're trying to fix with this PR, if none then please create an issue first. -->
## PR checklist
Please check if your PR fulfills the following requirements:
- [ ] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?
Thank you so much for contributing to MeiliSearch!
Co-authored-by: Rob Ede <robjtede@icloud.com>
2075: Allow payloads with no documents r=irevoire a=MarinPostma
accept addition with 0 documents.
0 bytes payload are still refused, since they are not valid json/jsonlines/csv anyways...
close#1987
Co-authored-by: mpostma <postma.marin@protonmail.com>
2068: chore(http): migrate from structopt to clap3 r=Kerollmops a=MarinPostma
migrate from structopt to clap3
This fix the long lasting issue with flags require a value, such as `--no-analytics` or `--schedule-snapshot`.
All flag arguments now take NO argument, i.e:
`meilisearch --schedule-snapshot true` becomes `meilisearch --schedule-snapshot`
as per https://docs.rs/clap/latest/clap/struct.Arg.html#method.env, the env variable is defines as:
> A false literal is n, no, f, false, off or 0. An absent environment variable will also be considered as false. Anything else will considered as true.
`@gmourier`
`@curquiza`
`@meilisearch/docs-team`
Co-authored-by: mpostma <postma.marin@protonmail.com>
2066: bug(http): fix task duration r=MarinPostma a=MarinPostma
`@gmourier` found that the duration in the task view was not computed correctly, this pr fixes it.
`@curquiza,` I let you decide if we need to make a hotfix out of this or wait for the next release. This is not breaking.
Co-authored-by: mpostma <postma.marin@protonmail.com>
Since the auths tests fail sporadically on the windows CI but we can't
reproduce these failures with a real windows machine we are going to
ignore theses one.
But we still ensure they compile.
2008: bug(lib): fix get dumps bad error code r=curquiza a=MarinPostma
fix bad error code being returned whet getting a dump status, and add a test
close#1994
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
2006: chore(http): rename task types r=curquiza a=MarinPostma
Rename
- documentsAddition into documentAddition
- documentsPartial into documentPartial
- documentsDeletion into documentDeletion
close#1999
2007: bug(lib): ignore primary if already set on document addition r=curquiza a=MarinPostma
Ignore the primary key if it is already set on documents updates. Add a test for verify behaviour.
close#2002
Co-authored-by: Marin Postma <postma.marin@protonmail.com>
1989: Extend API keys r=curquiza a=ManyTheFish
# Pull Request
## What does this PR do?
- Add API keys in snapshots
- Add API keys in dumps
- fix QA #1979fix#1979fix#1995fix#2001fix#2003
related to #1890
Co-authored-by: many <maxime@meilisearch.com>
- Add API keys in snapshots
- Add API keys in dumps
- Rename action indexes.add to indexes.create
- fix QA #1979fix#1979fix#1995fix#2001fix#2003
related to #1890
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0085-api-keys.md
- Add tests on API keys management route (meilisearch-http/tests/auth/api_keys.rs)
- Add tests checking authorizations on each meilisearch routes (meilisearch-http/tests/auth/authorization.rs)
- Implement API keys management routes (meilisearch-http/src/routes/api_key.rs)
- Create module to manage API keys and authorizations (meilisearch-auth)
- Reimplement GuardedData to extend authorizations (meilisearch-http/src/extractors/authentication/mod.rs)
- Change X-MEILI-API-KEY by Authorization Bearer (meilisearch-http/src/extractors/authentication/mod.rs)
- Change meilisearch routes to fit to the new authorization feature (meilisearch-http/src/routes/)
- close#1867
1875: Fix search post event and disk size analytics r=irevoire a=gmourier
- Branch POST search on the post_search aggregator
- Use largest disk `total_space` instead of `available_space`
1876: Update SEGMENT_API_KEY r=irevoire a=gmourier
Branch it on our Segment production stack
Co-authored-by: Guillaume Mourier <guillaume@meilisearch.com>
- remove the payload_error_handler in favor of a PayloadError::from
- merge the two match branch into one
- makes the accepted content type a const instead of recalculating it for every error
1768: Fix auth error r=irevoire a=MarinPostma
fix a small auth error, that set the invalid token error token to "hello". This was invilisble to the user because the invalid token is not returned.
thank you hawk-eye `@irevoire`
Co-authored-by: mpostma <postma.marin@protonmail.com>
This commit is a fix to issue #1750.
As a part of the changes to solve this issue, the following changes have
been made -
1. Route registration for static assets has been modified
2. the `mut` keyword on the `scope` has been removed.
1711: MeiliSearch refactor introducing OBKV format r=MarinPostma a=MarinPostma
This PR refactor some multiple components of meilisearch, and introduce the obkv document format to meilisearch
- [x] Split meilisearch-http and meilisearch-lib
- [x] Replace `IndexActor` and `UuidResolver` with `IndexResolver`
- [x] Remove mentions to Actor
- [x] Remove Actor traits to simplify code
- [x] Integrate obkv document format
- [x] Remove `Data`
- [x] Restore all route
- [x] Replace `Box<dyn error>` with `anyhow::Error`
- [x] Introduce update file store
- [x] Update file store error handling
- [x] Fix dumps
- [x] Fix snapshots
- [x] Fix tests
- [x] Update module documentation
- [x] add csv suppport (feat `@ManyTheFish` #1729 )
- [x] add jsonl support
- [x] integrate geosearch (feat `@irevoire` #1725)
partially implements #1691 and #1690. The error handling is very basic now, I will finish it in the next pr.
Some unit tests have been disabled, I will re-enable them ASAP, but they need a bit more work.
close#1531
P.S: sorry for this monstrous PR :'(
Co-authored-by: mpostma <postma.marin@protonmail.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: many <maxime@meilisearch.com>
`jemallocator` has been abandoned for nearly two years, and `rustc`
itself moved to use `tikv-jemallocator` instead:
3965773ae7
Let's switch to a better maintained version.
1651: Use reset_sortable_fields r=Kerollmops a=shekhirin
Resolves https://github.com/meilisearch/MeiliSearch/issues/1635
1676: Add curl binary to final stage image r=curquiza a=ook
Reference: #1673
Changes: * add `curl` binary to final docker Melisearch image.
For metrics, docker funny layer management makes this add a shrink from 319MB to 315MB:
```
☁ MeiliSearch [feature/1673-add-curl-to-docker-image] ⚡ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
getmeili/meilisearch 0.22.0_ook_1673 938e239ad989 2 hours ago 315MB
getmeili/meilisearch latest 258fa3aa1230 6 days ago 319MB
```
1684: bump dependencies r=MarinPostma a=MarinPostma
Bump meilisearch dependencies.
We still depend on custom patch that have been upgraded along the way.
Co-authored-by: Alexey Shekhirin <a.shekhirin@gmail.com>
Co-authored-by: Thomas Lecavelier <thomas@followanalytics.com>
Co-authored-by: mpostma <postma.marin@protonmail.com>
1682: Change the format of custom ranking rules when importing old dumps r=curquiza a=Kerollmops
This PR changes the format of the custom ranking rules from `asc(price)` to `title:asc` as the format changed between v0.21 and v0.22. The dumps are now correctly importing the custom ranking rules.
This PR also change the previous default ranking rules (without sort) to the new default ranking rules (with the new sort).
Co-authored-by: Kerollmops <clement@meilisearch.com>
1652: Remove dependabot r=MarinPostma a=curquiza
Fixes#1649
Dependabot for vulnerability and security updates is still activated.
1654: Add Script for Windows r=MarinPostma a=singh08prashant
fixes#1570
changes:
1. added script for detecting windows os running git bash
2. appended `.exe` to `$release_file` for windows as listed [here](https://github.com/meilisearch/MeiliSearch/releases/)
3. removed global `$BINARY_NAME='meilisearch'` as windows require `.exe` file
1657: Bring vergen hotfix from `stable` to `main` r=MarinPostma a=curquiza
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
Co-authored-by: singh08prashant <singh08prashant@gmail.com>
Co-authored-by: Kerollmops <clement@meilisearch.com>
Co-authored-by: bors[bot] <26634292+bors[bot]@users.noreply.github.com>
1540: Update milli to version 0.8.1 r=curquiza a=curquiza
Integrates this fix into MeiliSearch https://github.com/meilisearch/milli/pull/296
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
1539: Use serdeval for validating json format. r=curquiza a=MarinPostma
uses [serdeval](https://github.com/MarinPostma/serdeval) to validate that the json payload is valid json, and in the correct format.
fix#1535
Co-authored-by: mpostma <postma.marin@protonmail.com>
1498: Show the filterable and not the faceted attributes in the settings r=Kerollmops a=Kerollmops
Fixes#1497
Co-authored-by: Clément Renault <clement@meilisearch.com>
1478: refactor routes r=irevoire a=MarinPostma
refactor the route directory, so the module tree follows the route structure
Co-authored-by: mpostma <postma.marin@protonmail.com>
1457: Hotfix highlight on emojis panic r=Kerollmops a=ManyTheFish
When the highlight bound is in the middle of a character
or if we are out of bounds, we highlight the complete matching word.
note: we should enhance the tokenizer and the Highlighter to match char indices.
Fix#1368
Co-authored-by: many <maxime@meilisearch.com>
When the highlight bound is in the middle of a character
or if we are out of bounds, we highlight the complete matching word.
note: we should enhance the tokenizer and the Highlighter to match char indices.
Fix#1368
259: Run rustfmt one the whole project and add it to the CI r=curquiza a=irevoire
Since there is currently no other PR modifying the code, I think it's a good time to reformat everything and add rustfmt to the ci.
Co-authored-by: Tamo <tamo@meilisearch.com>
258: Use rustls instead of openssl r=curquiza a=irevoire
I also removed all the `default-features` of reqwest since we are only using the JSON one.
Fix#255
Co-authored-by: Tamo <tamo@meilisearch.com>
246: Stop logging the no space left on device error r=curquiza a=irevoire
closes#208
@qdequele what do you think of that?
Are there any other errors we need to ignore?
As you can see in the code, once we are in `Sentry` the error has already been converted to a `String` so the only thing we can do to see if we need to send the error or not is to match the `String` against our error message.
If we have a lot of other logs we want to ignore I would suggest prefixing all the logs with something like:
```
User error: No space left on device
```
So in Sentry, we could just check if the log start by `User error:` and ignore all these errors at once
Co-authored-by: Tamo <tamo@meilisearch.com>
249: Use half of the computer threads for the indexing process by default r=Kerollmops a=irevoire
closes#241
By default, we use only half of the CPU threads when indexing documents; this allows the user to use the search while indexing. Also, the machine will not appear unresponsive when indexing a large batch of documents.
On the special case where a user only has one core, we use it entirely 😄
Co-authored-by: Tamo <tamo@meilisearch.com>
240: Rework error messages r=irevoire a=MarinPostma
Simplify the error messages, and make them more compliant with legacy Meilisearch.
Basically, stop composing the messages, and simply forward the message of inner errors.
Co-authored-by: marin postma <postma.marin@protonmail.com>
232: Fix payload size limit r=MarinPostma a=MarinPostma
Fix#223
This was due to the fact that Payload ignores the limit payload size limit. I fixed it by implementing my own `Payload` extractor that checks that the size of the payload is not too large.
I also refactored the `create_app` a bit.
Co-authored-by: marin postma <postma.marin@protonmail.com>
227: improve mini dashboard routing r=MarinPostma a=MarinPostma
The dependency we use to statically serve the mini-dashboard used globing to serve the mini-dashboard files. This caused all unfound routes to be caught by the "/" serving the dashboard assets. This fix makes it so that the assets have a dedicated route, and any unfound route is caught by the default service and return a 404.
Co-authored-by: marin postma <postma.marin@protonmail.com>
226: Make facetsDistribution name iso r=MarinPostma a=curquiza
Even if there is an English mistake in `facets_distribution` (because of the `s`) @gmourier asked me to keep the typo: the name of `facetsDistribution` might change completely in the future, he wants to avoid two breakings.
@gmourier can you confirm before we merge this PR?
Sorry I left this update in the code (I'm confused because no issues was open to update `facetsDistribution`), there might have been a confusion with `fieldsDistribution` that has been renamed into `fieldDistribution`. Sorry!
Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
