Commit Graph

789 Commits

Author SHA1 Message Date
Tamo
6bd6321226
dump the content of the dump tasks instead of recreating at import time with wrong API keys 2022-10-27 11:34:08 +02:00
Tamo
9fe24fbff2
get rids of the useless Seek before creating a grenad reader 2022-10-27 11:34:08 +02:00
Tamo
78ce29f461
apply most style comments of the review 2022-10-27 11:34:08 +02:00
Tamo
e0221fc0a3
fix a synchronization bug while importing tasks 2022-10-27 11:34:07 +02:00
Tamo
554600dfd8
fix the deletion of the data.ms in case of errors 2022-10-27 11:34:07 +02:00
Tamo
d976e680c5
first mostly working version 2022-10-27 11:34:06 +02:00
Tamo
b7f9c94f4a
write the dump export 2022-10-27 11:34:05 +02:00
Loïc Lecrenier
f32b973945
Return an error when calling DELETE /tasks with an empty query 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
fbd2be2ec8
Apply suggested changes from PR review 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
8bb0fcd144
Finish first draft of the DELETE /tasks route 2022-10-27 11:34:04 +02:00
Loïc Lecrenier
9522b75454
Continue implementation of task deletion
1. Matched tasks are a roaring bitmap
2. Start implementation in meilisearch-http
3. Snapshots use meili-snap
4. Rename to TaskDeletion
2022-10-27 11:34:03 +02:00
Tamo
7034803712
move the API key in meilisearch_types 2022-10-27 11:34:02 +02:00
Tamo
2f1eb78b1d
refactor the Task a little bit 2022-10-27 11:34:02 +02:00
Tamo
510ce9fc51
start moving a lot of task types to meilisearch_types 2022-10-27 11:34:01 +02:00
Loïc Lecrenier
3e4337c91f
Add meili-snap crate to make writing snapshot tests easier 2022-10-27 11:34:01 +02:00
Tamo
667c282e19
get rids of the index crate + the document_types crate 2022-10-27 11:34:00 +02:00
Kerollmops
36e5efde0d
Update the tasks statuses 2022-10-27 11:33:49 +02:00
Kerollmops
cf6084151b
Make sure that meilisearch-http works without index wrapper 2022-10-27 11:33:48 +02:00
Kerollmops
c70f375669
Implement ErrorCode on the heed Error 2022-10-27 11:33:48 +02:00
Kerollmops
91e13c2824
Implement ErrorCode on the milli::Error type 2022-10-27 11:33:48 +02:00
Kerollmops
5d21c790ef
Make clippy happy 2022-10-27 11:33:47 +02:00
Kerollmops
3b343a930e
Fix meilisearch-http to use the new DocumentImport batch operation 2022-10-27 11:33:47 +02:00
Tamo
c2899fe9b2
bring back the IndexMeta and IndexStats in meilisearch-http 2022-10-27 11:33:45 +02:00
Tamo
c759fd6924
fix import bug 2022-10-27 11:33:45 +02:00
Tamo
fba9aa214a
remove the create_app macro 2022-10-27 11:33:45 +02:00
Tamo
2c8f1a43e9
get rids of meilisearch-lib 2022-10-27 11:33:44 +02:00
Tamo
ce2dfecc03
connect the new scheduler to meilisearch-http officially.
I can index documents and do search
2022-10-27 11:33:43 +02:00
Tamo
cb4feabca2
implements the get_tasks 2022-10-27 11:33:43 +02:00
Tamo
19154e48fe
fix all compilation errors 2022-10-27 11:33:42 +02:00
Irevoire
8d51c1f389
wip integrating the scheduler in meilisearch-http 2022-10-27 11:33:42 +02:00
bors[bot]
ab1800551f
Merge #2922
2922: Add new error when using /keys without masterkey set r=ManyTheFish a=vishalsodani

# Pull Request

## Related issue
Fixes #2918 


Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?




Co-authored-by: vishalsodani <vishalsodani@rediffmail.com>
2022-10-27 09:13:11 +00:00
vishalsodani
689bef7ad2 fmt the code 2022-10-27 14:09:38 +05:30
vishalsodani
89c40c83c3 refactor code to avoid cloning 2022-10-27 14:08:29 +05:30
vishalsodani
9cf3ff72a3 fix checking of master key as per review comment 2022-10-27 12:56:18 +05:30
bors[bot]
25ec51e783
Merge #2601
2601: Ease search result pagination r=Kerollmops a=ManyTheFish

# Summary
This PR is a prototype enhancing search results pagination (#2577)

# Todo

- [x] Update the API to return the number of pages and allow users to directly choose a page instead of computing an offset
- [x] Change computation of `total_pages` in order to have an exact count
  - [x] compute query tree exhaustively
  - [x] compute distinct exhaustively

# Small Documentation

## Default search query

**request**:
```sh
curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "botman" }'
```

**result**:
```json
{
  "hits":[...],
  "query":"botman",
  "processingTimeMs":5,
  "hitsPerPage":20,
  "page":1,
  "totalPages":4,
  "totalHits":66
}
```

## Search query with offset parameter

**request**:
```sh
curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "botman", "offset": 0 }'
```

**result**:
```json
{
  "hits":[...],
  "query":"botman",
  "processingTimeMs":3,
  "limit":20,
  "offset":0,
  "estimatedTotalHits":66
}
```

## Search query selecting page with page parameter

**request**:
```sh
curl \
  -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{ "q": "botman", "page": 2 }'
```

**result**:
```json
{
  "hits":[...],
  "query":"botman",
  "processingTimeMs":5,
  "hitsPerPage":20,
  "page":2,
  "totalPages":4,
  "totalHits":66
}
```

# Related

fixes #2577

## In charge of the feature

Core: `@ManyTheFish` 
Docs: `@guimachiavelli` 
Integration: `@bidoubiwa` 


Co-authored-by: ManyTheFish <many@meilisearch.com>
2022-10-26 16:10:58 +00:00
ManyTheFish
f4021273b8 Add is_finite_pagination method to SearchQuery 2022-10-26 18:08:29 +02:00
vishalsodani
f0ecacb58d add implementation for no master key set and fix tests 2022-10-25 22:41:48 +05:30
LunarMarathon
bc2a161f62 Change err mess for config 2022-10-25 16:16:34 +05:30
ManyTheFish
a2314cf436 Update analytics 2022-10-24 13:56:26 +02:00
ManyTheFish
0578aff8c9 Fix the tests 2022-10-20 17:41:13 +02:00
ManyTheFish
506d08a9f4 Update analytics version 2022-10-19 14:05:42 +02:00
ManyTheFish
b423ef72be PROTO: hardcode version and interval for prototype analytics 2022-10-19 14:05:42 +02:00
ManyTheFish
77e718214f Fix pagination analytics 2022-10-19 14:05:42 +02:00
ManyTheFish
b1bf6722e8 Update API to fit the proto needs 2022-10-19 13:58:03 +02:00
vishalsodani
1cf6efa740 Add new error when using /keys without masterkey set 2022-10-18 10:48:45 +05:30
bors[bot]
b0749407f3
Merge #2804
2804: Add environement variable `MEILI_CONFIG_FILE_PATH` to define the config file path r=Kerollmops a=choznerol

# Pull Request

## What does this PR do?
Fixes #2800

~This is a draft PR base on the code in #2745. I will `rebase` and mark it ready for review only after #2745 merge.~ Done rebase


## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?


## Demo

With `config.toml`, `config_copy1.toml` and `config_copy2.toml` present:
> <img width="692" alt="image" src="https://user-images.githubusercontent.com/12410942/192566891-6b3c9d26-736f-4e23-a09b-687fca1cb50d.png">

`MEILI_CONFIG_FILE_PATH` works:
> <img width="773" alt="image" src="https://user-images.githubusercontent.com/12410942/192567023-f751536e-992a-4e90-a176-cb19122248be.png">

`--config-file-path` still works:
> <img width="768" alt="image" src="https://user-images.githubusercontent.com/12410942/192567318-88c80b24-7873-4cec-8d08-16fe4d228055.png">

When both present, `--config-file-path` taks precedence:
> <img width="1214" alt="image" src="https://user-images.githubusercontent.com/12410942/192567477-8a7cffe1-96f0-42a9-a348-6dbec20dc1e7.png">



Co-authored-by: Lawrence Chou <choznerol@protonmail.com>
2022-10-17 10:40:05 +00:00
bors[bot]
5d895dd7da
Merge #2851
2851: Upgrade clap to 4.0 r=loiclec a=choznerol

# Pull Request

## Related issue
Fixes #2846 

This PR is draft based on #2847 to avoid conflict. I will rebase and mark as 'Ready for review' after #2847 is merged.

## What does this PR do?
1. Upgrade clap to the latest version or 4.0 (4.0.9 as of today) by following the [migrating instruction](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating) from [4.0 changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#migrating)
2. Fix an `ArgGroup` typo that can only be caught after upgrading to 4.0 in 20a715e29ed17c5a76229c98fb31504ada873597

## Notable changes

### The `--help` message

The format, ordering and indentation of `--help` message was changed in 4.0. I recorded the output of `cargo run -- --help` before and after upgrade to 4.0 for reference.

<details>
<summary>diff</summary>

Output of `diff --ignore-all-space --text --unified --new-file help-message-before.txt help-message-after.txt`:

```diff
--- help-message-before.txt	2022-10-14 16:45:36.000000000 +0800
+++ help-message-after.txt	2022-10-14 16:36:53.000000000 +0800
`@@` -1,12 +1,8 `@@`
-meilisearch-http 0.29.1
+Usage: meilisearch [OPTIONS]
 
-USAGE:
-    meilisearch [OPTIONS]
-
-OPTIONS:
+Options:
         --config-file-path <CONFIG_FILE_PATH>
-            Set the path to a configuration file that should be used to setup the engine. Format
-            must be TOML
+          Set the path to a configuration file that should be used to setup the engine. Format must be TOML
 
         --db-path <DB_PATH>
             Designates the location where database files will be created and retrieved
`@@` -26,15 +22,14 `@@`
             [default: dumps/]
 
         --env <ENV>
-            Configures the instance's environment. Value must be either `production` or
-            `development`
+          Configures the instance's environment. Value must be either `production` or `development`
             
             [env: MEILI_ENV=]
             [default: development]
             [possible values: development, production]
 
     -h, --help
-            Print help information
+          Print help information (use `-h` for a summary)
 
         --http-addr <HTTP_ADDR>
             Sets the HTTP address and port Meilisearch will use
`@@` -43,63 +38,53 `@@`
             [default: 127.0.0.1:7700]
 
         --http-payload-size-limit <HTTP_PAYLOAD_SIZE_LIMIT>
-            Sets the maximum size of accepted payloads. Value must be given in bytes or explicitly
-            stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+          Sets the maximum size of accepted payloads. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
             
             [env: MEILI_HTTP_PAYLOAD_SIZE_LIMIT=]
             [default: 100000000]
 
         --ignore-dump-if-db-exists
-            Prevents a Meilisearch instance with an existing database from throwing an error when
-            using `--import-dump`. Instead, the dump will be ignored and Meilisearch will launch
-            using the existing database.
+          Prevents a Meilisearch instance with an existing database from throwing an error when using `--import-dump`. Instead, the dump will be ignored and Meilisearch will launch using the existing database.
             
             This option will trigger an error if `--import-dump` is not defined.
             
             [env: MEILI_IGNORE_DUMP_IF_DB_EXISTS=]
 
         --ignore-missing-dump
-            Prevents Meilisearch from throwing an error when `--import-dump` does not point to a
-            valid dump file. Instead, Meilisearch will start normally without importing any dump.
+          Prevents Meilisearch from throwing an error when `--import-dump` does not point to a valid dump file. Instead, Meilisearch will start normally without importing any dump.
             
             This option will trigger an error if `--import-dump` is not defined.
             
             [env: MEILI_IGNORE_MISSING_DUMP=]
 
         --ignore-missing-snapshot
-            Prevents a Meilisearch instance from throwing an error when `--import-snapshot` does not
-            point to a valid snapshot file.
+          Prevents a Meilisearch instance from throwing an error when `--import-snapshot` does not point to a valid snapshot file.
             
             This command will throw an error if `--import-snapshot` is not defined.
             
             [env: MEILI_IGNORE_MISSING_SNAPSHOT=]
 
         --ignore-snapshot-if-db-exists
-            Prevents a Meilisearch instance with an existing database from throwing an error when
-            using `--import-snapshot`. Instead, the snapshot will be ignored and Meilisearch will
-            launch using the existing database.
+          Prevents a Meilisearch instance with an existing database from throwing an error when using `--import-snapshot`. Instead, the snapshot will be ignored and Meilisearch will launch using the existing database.
             
             This command will throw an error if `--import-snapshot` is not defined.
             
             [env: MEILI_IGNORE_SNAPSHOT_IF_DB_EXISTS=]
 
         --import-dump <IMPORT_DUMP>
-            Imports the dump file located at the specified path. Path must point to a `.dump` file.
-            If a database already exists, Meilisearch will throw an error and abort launch
+          Imports the dump file located at the specified path. Path must point to a `.dump` file. If a database already exists, Meilisearch will throw an error and abort launch
             
             [env: MEILI_IMPORT_DUMP=]
 
         --import-snapshot <IMPORT_SNAPSHOT>
-            Launches Meilisearch after importing a previously-generated snapshot at the given
-            filepath
+          Launches Meilisearch after importing a previously-generated snapshot at the given filepath
             
             [env: MEILI_IMPORT_SNAPSHOT=]
 
         --log-level <LOG_LEVEL>
             Defines how much detail should be present in Meilisearch's logs.
             
-            Meilisearch currently supports five log levels, listed in order of increasing verbosity:
-            ERROR, WARN, INFO, DEBUG, TRACE.
+          Meilisearch currently supports five log levels, listed in order of increasing verbosity: ERROR, WARN, INFO, DEBUG, TRACE.
             
             [env: MEILI_LOG_LEVEL=]
             [default: INFO]
`@@` -110,31 +95,25 `@@`
             [env: MEILI_MASTER_KEY=]
 
         --max-index-size <MAX_INDEX_SIZE>
-            Sets the maximum size of the index. Value must be given in bytes or explicitly stating a
-            base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+          Sets the maximum size of the index. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
             
             [env: MEILI_MAX_INDEX_SIZE=]
             [default: 107374182400]
 
         --max-indexing-memory <MAX_INDEXING_MEMORY>
-            Sets the maximum amount of RAM Meilisearch can use when indexing. By default,
-            Meilisearch uses no more than two thirds of available memory
+          Sets the maximum amount of RAM Meilisearch can use when indexing. By default, Meilisearch uses no more than two thirds of available memory
             
             [env: MEILI_MAX_INDEXING_MEMORY=]
             [default: "21.33 TiB"]
 
         --max-indexing-threads <MAX_INDEXING_THREADS>
-            Sets the maximum number of threads Meilisearch can use during indexation. By default,
-            the indexer avoids using more than half of a machine's total processing units. This
-            ensures Meilisearch is always ready to perform searches, even while you are updating an
-            index
+          Sets the maximum number of threads Meilisearch can use during indexation. By default, the indexer avoids using more than half of a machine's total processing units. This ensures Meilisearch is always ready to perform searches, even while you are updating an index
             
             [env: MEILI_MAX_INDEXING_THREADS=]
             [default: 5]
 
         --max-task-db-size <MAX_TASK_DB_SIZE>
-            Sets the maximum size of the task database. Value must be given in bytes or explicitly
-            stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
+          Sets the maximum size of the task database. Value must be given in bytes or explicitly stating a base unit (for instance: 107374182400, '107.7Gb', or '107374 Mb')
             
             [env: MEILI_MAX_TASK_DB_SIZE=]
             [default: 107374182400]
```

- ~[help-message-before.txt](https://github.com/meilisearch/meilisearch/files/9715683/help-message-before.txt)~ [help-message-before.txt](https://github.com/meilisearch/meilisearch/files/9784156/help-message-before-2.txt)
- ~[help-message-after.txt](https://github.com/meilisearch/meilisearch/files/9715682/help-message-after.txt)~ [help-message-after.txt](https://github.com/meilisearch/meilisearch/files/9784091/help-message-after.txt)

</details>


## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Lawrence Chou <choznerol@protonmail.com>
2022-10-17 10:08:27 +00:00
Lawrence Chou
5dafdd9a23
Preserve --help output ordering after upgrade Clap to 4.0
From the [4.0 breaking change][1]:

  ...
  * (help) Make DeriveDisplayOrder the default and removed the setting. To sort help, set next_display_order(None) (#2808)
  ...

[1]: https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#breaking-changes
2022-10-14 16:43:03 +08:00
Lawrence Chou
53e5229b4a
Assert error message for Windows besides *nix
The 'Tests on windows-latest' now failed with error message below

---- option::test::test_meilli_config_file_path_invalid stdout ----
thread 'option::test::test_meilli_config_file_path_invalid' panicked at 'assertion failed:
  left: `"unable to open or read the \"../configgg.toml\" configuration file: The system cannot find the file specified. (os error 2)."`,
 right: `"unable to open or read the \"../configgg.toml\" configuration file: No such file or directory (os error 2)."`', meilisearch-http\src\option.rs:555:17

https://github.com/meilisearch/meilisearch/actions/runs/3231941308/jobs/5291998750
2022-10-14 14:49:40 +08:00
Lawrence Chou
9ebc73e6ac
Comply with Clippy rule single-match 2022-10-14 14:16:10 +08:00