A lightning-fast search API that fits effortlessly into your apps, websites, and workflow
Go to file
bors[bot] c32f13a909
Merge #1800
1800: Analytics r=irevoire a=irevoire

Closes #1784
Implements [this spec](https://github.com/meilisearch/specifications/blob/update-analytics-specs/text/0034-telemetry-policies.md) 

# Anonymous Analytics Policy

## 1. Functional Specification

### I. Summary

This specification describes an exhaustive list of anonymous metrics collected by the MeiliSearch binary. It also describes the tools we use for this collection and how we identify a Meilisearch instance.

### II. Motivation

At MeiliSearch, our vision is to provide an easy-to-use search solution that meets the essential needs of our users. At all times, we strive to understand our users better and meet their expectations in the best possible way.

Although we can gather needs and understand our users through several channels such as Github, Slack, surveys, interviews or roadmap votes, we realize that this is not enough to have a complete view of MeiliSearch usage and features adoption. By cross-referencing our product discovery phases with aggregated quantitative data, we want to make the product much better than what it is today. Our decision-making will be taken a step further to make a product that users love.

### III. Explanation

#### General Data Protection Regulation (GDPR)

The metrics collected are non-sensitive, non-personal and do not identify an individual or a group of individuals using MeiliSearch. The data collected is secured and anonymized. We do not collect any data from the values stored in the documents.

We, the MeiliSearch team, provide an email address so that users can request the removal of their data: privacy@meilisearch.com.<br>
Thanks to the unique identifier generated for their MeiliSearch installation (`Instance uuid` when launching MeiliSearch), we can remove the corresponding data from all the tools we describe below. Any questions regarding the management of the data collected can be sent to the email address as well.

#### Tools

##### Segment

The collected data is sent to [Segment](https://segment.com/). Segment is a platform for data collection and provides data management tools.

##### Amplitude

[Amplitude](https://amplitude.com/) is a tool for graphing and highlighting collected data. Segment feeds Amplitude so that we can build visualizations according to our needs.

-----------
# The `identify` call we send every hour:

## System Configuration `system`

This property allows us to gather essential information to better understand on which type of machine MeiliSearch is used. This allows us to better advise users on the machines to choose according to their data volume and their use-cases.

 - [x] `system` => Never changes but still sent every hours
     - [x] distribution | On which distribution MeiliSearch is launched, eg: Arch Linux
     - [x] kernel_version | On which kernel version MeiliSearch is launched, eg: 5.14.10-arch1-1
     - [x] cores | How many cores does the machine have, eg: 24
     - [x] ram_size | Total capacity of the machine's ram. Expressed in `Kb`, eg: 33604210
     - [x] disk_size | Total capacity of the biggest disk. Expressed in `Kb`, eg: 336042103
     - [x] server_provider | Users can tell us on which provider MeiliSearch is hosted by filling the `MEILI_SERVER_PROVIDER` env var. This is also filled by our providers deploy scripts. e.g. GCP [cloud-config.yaml](56a7c2630c/scripts/providers/gcp/cloud-config.yaml (L33)), eg: gcp

## MeiliSearch Configuration

- [x] `context.app.version`: MeiliSearch version, eg: 0.23.0
- [x] `env`: `production` / `development`, eg: `production`
- [x] `has_snapshot`: Does the MeiliSearch instance has snapshot activated, eg: `true`

## MeiliSearch Statistics `stats`

 - [x] `stats`
     - [x] `database_size`: Size of indexed data. Expressed in `Kb`, eg: 180230
     - [x] `indexes_number`: Number of indexes, eg: 2
     - [x] `documents_number`: Number of indexed documents, eg: 165847
     - [x] `start_since_days`: How many days ago was the instance launched?, eg: 328

---------

- [x] Launched | This is the first event sent to mark that MeiliSearch is launched a first time

---------

- [x] `Documents Searched POST`: The Documents Searched event is sent once an hour. The event's properties are averaged over all search operations during that time so as not to track everything and generate unnecessary noise.
  - [x] `user-agent`: Represents all the user-agents encountered on this endpoint during one hour, eg: `["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]`
  - [x] `requests`
      - [x] `99th_response_time`: The maximum latency, in ms, for the fastest 99% of requests, eg: `57ms`
      - [x] `total_suceeded`: The total number of succeeded search requests, eg: `3456`
      - [x] `total_failed`: The total number of failed search requests, eg: `24`
      - [x] `total_received`: The total number of received search requests, eg: `3480`
  - [x] `sort`
      - [x] `with_geoPoint`: Does the built-in sort rule _geoPoint rule has been used?, eg: `true` /`false`
      - [x] `avg_criteria_number`: The average number of sort criteria among all the requests containing the sort parameter. "sort": [] equals to 0 while not sending sort does not influence the average, eg: `2`
  - [x] `filter`
      - [x] `with_geoRadius`: Does the built-in filter rule _geoRadius has been used?, eg: `true` /`false`
      - [x] `avg_criteria_number`: The average number of filter criteria among all the requests containing the filter parameter. "filter": [] equals to 0 while not sending filter does not influence the average, eg: `4`
      - [x] `most_used_syntax`: The most used filter syntax among all the requests containing the requests containing the filter parameter. `string` / `array` / `mixed`, `mixed`
  - [x] `q`
      - [x] `avg_terms_number`: The average number of terms for the `q` parameter among all requests, eg: `5`
  - [x] `pagination`:
      - [x] `max_limit`: The maximum limit encountered among all requests, eg: `20`
      - [x] `max_offset`: The maxium offset encountered among all requests, eg: `1000` 

---

- [x] `Documents Searched GET`: The Documents Searched event is sent once an hour. The event's properties are averaged over all search operations during that time so as not to track everything and generate unnecessary noise.
  - [x] `user-agent`: Represents all the user-agents encountered on this endpoint during one hour, eg: `["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]`
  - [x] `requests`
      - [x] `99th_response_time`: The maximum latency, in ms, for the fastest 99% of requests, eg: `57ms`
      - [x] `total_suceeded`: The total number of succeeded search requests, eg: `3456`
      - [x] `total_failed`: The total number of failed search requests, eg: `24`
      - [x] `total_received`: The total number of received search requests, eg: `3480`
  - [x] `sort`
      - [x] `with_geoPoint`: Does the built-in sort rule _geoPoint rule has been used?, eg: `true` /`false`
      - [x] `avg_criteria_number`: The average number of sort criteria among all the requests containing the sort parameter. "sort": [] equals to 0 while not sending sort does not influence the average, eg: `2`
  - [x] `filter`
      - [x] `with_geoRadius`: Does the built-in filter rule _geoRadius has been used?, eg: `true` /`false`
      - [x] `avg_criteria_number`: The average number of filter criteria among all the requests containing the filter parameter. "filter": [] equals to 0 while not sending filter does not influence the average, eg: `4`
      - [x] `most_used_syntax`: The most used filter syntax among all the requests containing the requests containing the filter parameter. `string` / `array` / `mixed`, `mixed`
  - [x] `q`
      - [x] `avg_terms_number`: The average number of terms for the `q` parameter among all requests, eg: `5`
  - [x] `pagination`:
      - [x] `max_limit`: The maximum limit encountered among all requests, eg: `20`
      - [x] `max_offset`: The maxium offset encountered among all requests, eg: `1000` 

---

- [x] `Index Created`
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `primary_key`: The name of the field used as primary key if set, otherwise `null`, eg: `id`

---

- [x] `Index Updated`
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `primary_key`: The name of the field used as primary key if set, otherwise `null`, eg: `id`

---

- [x] `Documents Added`: The Documents Added event is sent once an hour. The event's properties are averaged over all POST /documents additions operations during that time to not track everything and generate unnecessary noise.
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `payload_type`: Represents all the `payload_type` encountered on this endpoint during one hour, eg: [`text/csv`]
  - [x] `primary_key`: The name of the field used as primary key if set, otherwise `null`, eg: `id`
  - [x] `index_creation`: Does an index creation happened, eg: `false`

---

- [x] `Documents Updated`: The Documents Added event is sent once an hour. The event's properties are averaged over all PUT /documents additions operations during that time to not track everything and generate unnecessary noise.
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `payload_type`: Represents all the `payload_type` encountered on this endpoint during one hour, eg: [`application/json`]
  - [x] `primary_key`: The name of the field used as primary key if set, otherwise `null`, eg: `id`
  - [x] `index_creation`: Does an index creation happened, eg: `false`

---

- [x] Settings Updated
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `ranking_rules`
      - [x] `sort_position`: Position of the `sort` ranking rule if any, otherwise `null`, eg: `5`
  - [x] `sortable_attributes`
      - [x] `total`: Number of sortable attributes, eg: `3`
      - [x] `has_geo`: Indicate if `_geo` is set as a sortable attribute, eg: `false`
  - [x] `filterable_attributes`
      - [x] `total`: Number of filterable attributes, eg: `3`
      - [x] `has_geo`: Indicate if `_geo` is set as a filterable attribute, eg: `false`

---

- [x] `RankingRules Updated`
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `sort_position`: Position of the `sort` ranking rule if any, otherwise `null`, eg: `5`

---

- [x] `SortableAttributes Updated`
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `total`: Number of sortable attributes, eg: `3`
  - [x] `has_geo`: Indicate if `_geo` is set as a sortable attribute, eg: `false`

---

- [x] `FilterableAttributes Updated`
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]
  - [x] `total`: Number of filterable attributes, eg: `3`
  - [x] `has_geo`: Indicate if `_geo` is set as a filterable attribute, eg: `false`

---

- [x] Dump Created
  - [x] `user-agent`: Represents the user-agent encountered for this API call, eg: ["MeiliSearch Ruby (2.1)", "Ruby (3.0)"]

---

Ensure the user-id file is well saved and loaded with:
- [x] the dumps
- [x]  the snapshots



- [x] Ensure the CLI uuid only show if analytics are activate at launch **or already exists** (=even if meilisearch was launched without analytics)

Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: Irevoire <tamo@meilisearch.com>
2021-10-29 16:11:03 +00:00
.github Update .github/workflows/publish-crossbuild.yml 2021-10-24 11:31:04 +00:00
assets Remove old gif 2021-08-26 17:42:56 +02:00
meilisearch-error Change authentication error type to be 2021-10-28 16:57:48 +02:00
meilisearch-http fix bad rebase 2021-10-29 17:32:49 +02:00
meilisearch-lib rename user-id to instance-uid 2021-10-29 17:25:52 +02:00
.dockerignore import .git to docker to fix vergen 2021-07-28 19:12:40 +02:00
.gitignore ignore the snapshots and dumps in the gitignore (#1449) 2021-07-01 14:41:53 +02:00
bors.toml Use pr_status isntead of status 2021-10-14 14:21:42 +02:00
Cargo.lock bump segment to be able to display a user 2021-10-29 17:25:53 +02:00
Cargo.toml remove the debug mode in release 2021-10-29 17:25:51 +02:00
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md 2020-04-30 20:16:02 +02:00
CONTRIBUTING.md Update CONTRIBUTING.md 2021-10-05 18:07:59 +02:00
Cross.toml Cross build with action-rs 2021-10-10 02:21:30 +08:00
Dockerfile Add MEILI_SERVER_PROVIDER to dockerfile 2021-10-20 17:53:43 +02:00
download-latest.sh Replace double quotes by single ones 2021-10-18 23:39:07 +02:00
LICENSE Update LICENSE 2021-01-19 00:18:52 +01:00
README.md Update README.md 2021-10-12 13:49:31 +02:00
SECURITY.md Create SECURITY.md 2021-10-27 17:49:00 +02:00

MeiliSearch

MeiliSearch

Website | Roadmap | Blog | LinkedIn | Twitter | Documentation | FAQ

Build Status Dependency status License Slack Bors enabled

Lightning Fast, Ultra Relevant, and Typo-Tolerant Search Engine 🔍

MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box. For more information about features go to our documentation.

Web interface gif

Features

  • Search-as-you-type experience (answers < 50 milliseconds)
  • Full-text search
  • Typo tolerant (understands typos and misspelling)
  • Faceted search and filters
  • Supports hanzi (Chinese characters)
  • Supports synonyms
  • Easy to install, deploy, and maintain
  • Whole documents are returned
  • Highly customizable
  • RESTful API

Getting started

Deploy the Server

Homebrew (Mac OS)

brew update && brew install meilisearch
meilisearch

Docker

docker run -p 7700:7700 -v "$(pwd)/data.ms:/data.ms" getmeili/meilisearch

Announcing a cloud-hosted MeiliSearch

Join the closed beta by filling out this form.

Try MeiliSearch in our Sandbox

Create a MeiliSearch instance in MeiliSearch Sandbox. This instance is free, and will be active for 48 hours.

Run on Digital Ocean

DigitalOcean Marketplace

Deploy on Platform.sh

Deploy on Platform.sh

APT (Debian & Ubuntu)

echo "deb [trusted=yes] https://apt.fury.io/meilisearch/ /" > /etc/apt/sources.list.d/fury.list
apt update && apt install meilisearch-http
meilisearch

Download the binary (Linux & Mac OS)

curl -L https://install.meilisearch.com | sh
./meilisearch

Compile and run it from sources

If you have the latest stable Rust toolchain installed on your local system, clone the repository and change it to your working directory.

git clone https://github.com/meilisearch/MeiliSearch.git
cd MeiliSearch
cargo run --release

Create an Index and Upload Some Documents

Let's create an index! If you need a sample dataset, use this movie database. You can also find it in the datasets/ directory.

curl -L 'https://bit.ly/2PAcw9l' -o movies.json

Now, you're ready to index some data.

curl -i -X POST 'http://127.0.0.1:7700/indexes/movies/documents' \
  --header 'content-type: application/json' \
  --data-binary @movies.json

Search for Documents

In command line

The search engine is now aware of your documents and can serve those via an HTTP server.

The jq command-line tool can greatly help you read the server responses.

curl 'http://127.0.0.1:7700/indexes/movies/search?q=botman+robin&limit=2' | jq
{
  "hits": [
    {
      "id": "415",
      "title": "Batman & Robin",
      "poster": "https://image.tmdb.org/t/p/w1280/79AYCcxw3kSKbhGpx1LiqaCAbwo.jpg",
      "overview": "Along with crime-fighting partner Robin and new recruit Batgirl, Batman battles the dual threat of frosty genius Mr. Freeze and homicidal horticulturalist Poison Ivy. Freeze plans to put Gotham City on ice, while Ivy tries to drive a wedge between the dynamic duo.",
      "release_date": 866768400
    },
    {
      "id": "411736",
      "title": "Batman: Return of the Caped Crusaders",
      "poster": "https://image.tmdb.org/t/p/w1280/GW3IyMW5Xgl0cgCN8wu96IlNpD.jpg",
      "overview": "Adam West and Burt Ward returns to their iconic roles of Batman and Robin. Featuring the voices of Adam West, Burt Ward, and Julie Newmar, the film sees the superheroes going up against classic villains like The Joker, The Riddler, The Penguin and Catwoman, both in Gotham City… and in space.",
      "release_date": 1475888400
    }
  ],
  "nbHits": 8,
  "exhaustiveNbHits": false,
  "query": "botman robin",
  "limit": 2,
  "offset": 0,
  "processingTimeMs": 2
}

Use the Web Interface

We also deliver an out-of-the-box web interface in which you can test MeiliSearch interactively.

You can access the web interface in your web browser at the root of the server. The default URL is http://127.0.0.1:7700. All you need to do is open your web browser and enter MeiliSearchs address to visit it. This will lead you to a web page with a search bar that will allow you to search in the selected index.

| See the gif above

Documentation

Now that your MeiliSearch server is up and running, you can learn more about how to tune your search engine in the documentation.

Contributing

Hey! We're glad you're thinking about contributing to MeiliSearch! Feel free to pick an issue labeled as good first issue, and to ask any question you need. Some points might not be clear and we are available to help you!

Also, we recommend following the CONTRIBUTING to create your PR.

Core engine and tokenizer

The code in this repository is only concerned with managing multiple indexes, handling the update store, and exposing an HTTP API.

Search and indexation are the domain of our core engine, milli, while tokenization is handled by our tokenizer library.

Telemetry

MeiliSearch collects anonymous data regarding general usage. This helps us better understand developers' usage of MeiliSearch features.

To find out more on what information we're retrieving, please see our documentation on Telemetry.

This program is optional, you can disable these analytics by using the MEILI_NO_ANALYTICS env variable.

Feature request

The feature requests are not managed in this repository. Please visit our dedicated repository to see our work about the MeiliSearch product.

If you have a feature request or any feedback about an existing feature, please open a discussion. Also, feel free to participate in the current discussions, we are looking forward to reading your comments.

💌 Contact

Please visit this page.

MeiliSearch is developed by Meili, a young company. To know more about us, you can read our blog. Any suggestion or feedback is highly appreciated. Thank you for your support!