Commit Graph

8897 Commits

Author SHA1 Message Date
mpostma
99889a0ed0 add obkv document serialization test 2021-10-11 15:13:17 +02:00
mpostma
799f3d43c8 fix serialization to obkv format 2021-10-11 15:04:47 +02:00
Tamo
ed7fd855af
fix the wiki indexing benchmark 2021-10-11 14:26:36 +02:00
bors[bot]
b969f34317
Merge #1793
1793: Remove memmap dependency r=curquiza a=palfrey

Fixes #1792. I was going to replace with [memmap2](https://github.com/RazrFalcon/memmap2-rs) which should be a drop-in replacement, but I couldn't actually find anything that actually directly used it. It ends up being a dependency in [milli](https://github.com/meilisearch/milli) so I'm going to go there next and fix that.

Co-authored-by: Tom Parker-Shemilt <palfrey@tevp.net>
2021-10-11 08:29:40 +00:00
Tom Parker-Shemilt
2dfe24f067 memmap -> memmap2 2021-10-10 22:47:12 +01:00
Tom Parker-Shemilt
6c46fbbc57 Remove memmap dependency 2021-10-10 22:33:40 +01:00
Patrick Dung
87115b02d9 Fixing the passing of environment variables 2021-10-10 03:27:51 +08:00
Patrick Dung
c614520405 Cross build with action-rs 2021-10-10 02:21:30 +08:00
John Braun
3756f5a0ca Add test for highlighting numbers 2021-10-08 15:07:45 +02:00
John Braun
5b4e4bb858 Highlight numbers (int) as string in formatted JSON 2021-10-08 15:07:15 +02:00
bors[bot]
a2743baaa3
Merge #383
383: Add check on latitude and longitude r=irevoire a=irevoire

Latitudes are not supposed to go beyond 90 degrees or below -90.
The same goes for longitudes with 180 or -180.

This was badly implemented in the filters, and was not implemented for the `AscDesc` rules.

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-10-08 10:15:25 +00:00
Irevoire
b65aa7b5ac
Apply suggestions from code review
Co-authored-by: Clément Renault <clement@meilisearch.com>
2021-10-07 17:51:52 +02:00
bors[bot]
dffd90b966
Merge #1783
1783: Fix too many open file error r=ManyTheFish a=ManyTheFish

- prepare_for_closing() function wasn't called when an index is deleted, we are now calling it
- Index wasn't deleted in the case where we couldn't insert `uid` in `index_uuid_store`, we are now cleaning it

Fix #1736

Co-authored-by: many <maxime@meilisearch.com>
2021-10-07 15:48:07 +00:00
many
a92a0c3ed3
Log the error instead of returning it when deletion fails 2021-10-07 17:38:22 +02:00
many
0774b1efa5
Close index's heed environment when index is deleted 2021-10-07 17:09:41 +02:00
many
7fc7eb7457
Make sure to remove newly created index if uid is already taken 2021-10-07 16:49:21 +02:00
Tamo
11dfe38761
Update the check on the latitude and longitude
Latitude are not supposed to go beyound 90 degrees or below -90.
The same goes for longitude with 180 or -180.

This was badly implemented in the filters, and was not implemented for the AscDesc rules.
2021-10-07 16:10:43 +02:00
bors[bot]
602a327aa8
Merge #1781
1781: Optimize build size r=irevoire a=MarinPostma

Remove debug symbols from the release build, and strip the binaries.

We used to need to debug symbols for sentry, but since it was removed with #1616, we don't need them anymore.

Shrinks the binary size from ~300MB to ~50MB on linux.

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-07 13:31:03 +00:00
mpostma
14c6ae4735 disable stripping 2021-10-07 12:10:36 +02:00
mpostma
493a0e377d optimize build size 2021-10-07 11:49:52 +02:00
bors[bot]
5e3e108143
Merge #1769
1769: Enforce `Content-Type` header for routes requiring a body payload r=MarinPostma a=irevoire

closes #1665 

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-10-06 15:43:50 +00:00
Tamo
66dbd3cd34
makes clippy happy 2021-10-06 17:39:04 +02:00
Tamo
9a1e44dc78
Apply suggestion
- remove the payload_error_handler in favor of a PayloadError::from

- merge the two match branch into one

- makes the accepted content type a const instead of recalculating it for every error
2021-10-06 17:15:47 +02:00
Tamo
37b267ffb3
duplicate the post document tests with the put verb 2021-10-06 17:15:47 +02:00
Tamo
dfa199f98f
add content-type tests
fix typo

Co-authored-by: Clémentine Urquizar <clementine@meilisearch.com>
2021-10-06 17:15:47 +02:00
Tamo
c6d107a05f
makes the content-type mandatory for every routes 2021-10-06 17:15:47 +02:00
bors[bot]
ddbcf449da
Merge #1763
1763: Index tests r=MarinPostma a=MarinPostma

This pr aims to test more thorougly the usage on index in the meilisearch database, by writing unit tests.

work included:
- [x] Create index mock and stub methods
- [x] Test snapshot creation
- [x] Test Dumps
- [x] Test search

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-06 14:39:53 +00:00
mpostma
9fa61439b1 fix clippy warning & unsafety 2021-10-06 14:51:46 +02:00
Clémentine Urquizar
02dd1ea29d
Merge pull request #1771 from ferdi05/ferdi05-patch-contributing
Update CONTRIBUTING.md
2021-10-06 14:40:38 +02:00
mpostma
a38215de98 edit documentation 2021-10-06 14:35:18 +02:00
mpostma
85b5260d9d simple search unit test 2021-10-06 14:20:05 +02:00
mpostma
4b4ebad9a9 test dumps 2021-10-06 14:10:26 +02:00
mpostma
ece4c739f4 update store tests 2021-10-06 14:10:26 +02:00
mpostma
85ae34cf9f test snapshots 2021-10-06 14:10:23 +02:00
mpostma
0448f0ce56 handle panic in stubs 2021-10-06 14:09:04 +02:00
mpostma
4835d82a0b implement index mock 2021-10-06 14:09:01 +02:00
bors[bot]
dde1da1c0e
Merge #382
382: Refactor attribute criterion r=Kerollmops a=ManyTheFish

### Re-implement set based algorithm for attribute criterion
#### Levels
Instead of doing level iteration and digging in the interesting level, we only iterate over the lowest level.

#### crossword iteration VS minimal position iteration
Instead of crossing word position in order to iterate strictly over the position that gives the best rank in good order; we iterate word by word starting with the word that increases the rank the little as possible.
This new method is a bit less precise but way simpler.

### Simplify word-level-position database
We don't use levels anymore in the attribute criterion, and so we removed the level complexity of the database making a word-position-docids database.

### Benchmarks on search on big datasets

#### songs main VS refactor-attribute-criterion
```diff
  group                                                   search_songsmain_31c18f09               search_songsrefactor-attribute-criterion_1bd15d84
  -----                                                   -------------------------               -------------------------------------------------
- smol-songs.csv: basic filter: <=/Notstandskomitee       1.00     84.8±0.58µs        ? ?/sec     1.09     92.2±8.98µs        ? ?/sec
+ smol-songs.csv: basic filter: TO/Notstandskomitee       1.18     98.0±6.30µs        ? ?/sec     1.00     83.2±0.97µs        ? ?/sec
+ smol-songs.csv: basic with quote/"david" "bowie"        114.68    76.0±0.20ms        ? ?/sec    1.00    662.5±5.03µs        ? ?/sec
- smol-songs.csv: basic with quote/"john"                 1.00    197.4±1.06µs        ? ?/sec     1.05    208.1±1.53µs        ? ?/sec
+ smol-songs.csv: basic with quote/"michael" "jackson"    2.75      2.0±0.01ms        ? ?/sec     1.00    738.9±3.91µs        ? ?/sec
+ smol-songs.csv: basic without quote/david bowie         297.42  1499.3±0.86ms        ? ?/sec    1.00      5.0±0.02ms        ? ?/sec
+ smol-songs.csv: basic without quote/michael jackson     2.55      8.9±0.02ms        ? ?/sec     1.00      3.5±0.01ms        ? ?/sec
+ smol-songs.csv: big filter/john                         1.08    473.6±2.25µs        ? ?/sec     1.00    438.1±2.59µs        ? ?/sec
- smol-songs.csv: prefix search/a                         1.00    446.9±1.81µs        ? ?/sec     1.79    800.5±4.45µs        ? ?/sec
- smol-songs.csv: prefix search/b                         1.00    398.5±2.74µs        ? ?/sec     1.81    723.1±5.46µs        ? ?/sec
- smol-songs.csv: prefix search/i                         1.00    486.3±1.99µs        ? ?/sec     1.69    823.6±9.42µs        ? ?/sec
- smol-songs.csv: prefix search/s                         1.00    229.6±3.29µs        ? ?/sec     2.59    594.4±2.22µs        ? ?/sec
- smol-songs.csv: prefix search/x                         1.00    150.2±0.76µs        ? ?/sec     1.11    166.0±0.87µs        ? ?/sec
```

On songs, the new algorithm gives a big improvement on slow queries, and is slower on one char prefix search (fast queries <1ms).

#### wiki main VS refactor-attribute-criterion
```diff
  group                                                           search_wikimain_31c18f09               search_wikirefactor-attribute-criterion_1bd15d84
  -----                                                           ------------------------               ------------------------------------------------
- smol-wiki-articles.csv: basic with quote/"rock" "and" "roll"    1.00      3.2±0.01ms        ? ?/sec    1.15      3.7±0.01ms        ? ?/sec
- smol-wiki-articles.csv: basic without quote/film                1.00    351.5±2.47µs        ? ?/sec    1.13    396.8±1.63µs        ? ?/sec
+ smol-wiki-articles.csv: basic without quote/rock and roll       1.10      9.4±0.02ms        ? ?/sec    1.00      8.6±0.04ms        ? ?/sec
- smol-wiki-articles.csv: basic without quote/spain               1.00    446.0±3.23µs        ? ?/sec    1.11    496.6±7.75µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/c                         1.00    115.6±0.61µs        ? ?/sec    2.22    256.7±1.24µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/g                         1.00    189.7±2.03µs        ? ?/sec    1.57    297.0±1.35µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/j                         1.00    209.2±1.11µs        ? ?/sec    1.40    293.0±2.09µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/q                         1.00     79.0±0.44µs        ? ?/sec    1.10     87.2±0.69µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/t                         1.00    270.1±1.15µs        ? ?/sec    1.55    419.9±5.16µs        ? ?/sec
- smol-wiki-articles.csv: prefix search/x                         1.00    244.9±1.33µs        ? ?/sec    1.07    260.9±1.95µs        ? ?/sec
- smol-wiki-articles.csv: words/Abraham machin                    1.00      8.1±0.03ms        ? ?/sec    1.17      9.4±0.02ms        ? ?/sec
- smol-wiki-articles.csv: words/Idaho Bellevue pizza              1.00     19.3±0.07ms        ? ?/sec    1.07     20.6±0.05ms        ? ?/sec
```
On wiki we have some regressions `+17%` and `+15%` on request `>1ms`.

Co-authored-by: many <maxime@meilisearch.com>
2021-10-06 09:19:33 +00:00
many
085bc6440c
Apply PR comments 2021-10-06 11:12:26 +02:00
many
1bd15d849b
Reduce candidates threshold 2021-10-05 18:52:14 +02:00
Ferdinand Boas
eaddee9fe2
Update CONTRIBUTING.md
typo + text improvement
all credits go to @guimachiavelli
2021-10-05 18:07:59 +02:00
many
ea4bd29d14
Apply PR comments 2021-10-05 17:35:07 +02:00
bors[bot]
2190764162
Merge #1768
1768: Fix auth error r=irevoire a=MarinPostma

fix a small auth error, that set the invalid token error token to "hello". This was invilisble to the user because the invalid token is not returned.

thank you hawk-eye `@irevoire` 

Co-authored-by: mpostma <postma.marin@protonmail.com>
2021-10-05 15:16:14 +00:00
many
5ed75de0db
Update infos crate 2021-10-05 13:56:12 +02:00
many
3296bb243c
Simplify word level position DB into a word position DB 2021-10-05 12:15:02 +02:00
many
75d341d928
Re-implement set based algorithm for attribute criterion 2021-10-05 12:14:50 +02:00
mpostma
3b91764587 fix auth error 2021-10-05 09:09:40 +02:00
bors[bot]
0c3ec549f8
Merge #1764
1764: Bump Milli to improve geosearch errors r=curquiza a=irevoire

closes #1734 

`@curquiza,` your two examples still don't work because a filter must be composed of multiples operations; look at my screenshot to see what works and what doesn't.
Is this ok? 🤔 

`@gmourier,`
What do you think?

![image](https://user-images.githubusercontent.com/7032172/135846911-588f652d-16db-4d88-89fd-148640bac0f7.png)


And here is a screenshot with all the new errors that have been implemented

![image](https://user-images.githubusercontent.com/7032172/135854851-da469fef-0dd0-4ff1-b15e-89934ed8fb6f.png)

Co-authored-by: Tamo <tamo@meilisearch.com>
2021-10-04 17:30:22 +00:00
Tamo
fca686e7f8
bump meilisearch 2021-10-04 13:52:37 +02:00
bors[bot]
607e28749a
Merge #1755
1755: Fix mini dashboard r=curquiza a=anirudhRowjee

This commit is a fix to issue #1750.
As a part of the changes to solve this issue, the following changes have
been made -
1. Route registration for static assets has been modified
2. the `mut` keyword on the `scope` has been removed.

Co-authored-by: Anirudh Rowjee <ani.rowjee@gmail.com>
2021-10-04 09:56:21 +00:00
Anirudh Rowjee
bffab21b10 Changes
1. Removed redundant scope registration
2021-10-04 14:47:05 +05:30