snowykami/meilisearch
1
0
Fork 0
mirror of https://github.com/meilisearch/meilisearch.git synced 2024-11-23 18:45:06 +08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Merge #359

Browse Source 
359: Improve the benchmark comparison script r=irevoire a=irevoire

This modification allow us to compare more than 2 benchmarks or to only print the results of one benchmark



Co-authored-by: Irevoire <tamo@meilisearch.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
This commit is contained in:
bors[bot] 2021-09-20 12:39:59 +00:00 committed by GitHub
commit 203aa727a7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 17 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
benchmarks/README.md
View File

@ -54,7 +54,7 @@ MILLI_BENCH_DATASETS_PATH=~/datasets cargo bench --bench songs # the code is com
## Comparison between benchmarks ## Comparison between benchmarks
The benchmark reports we push are generated with `critcmp`. Thus, we use `critcmp` to generate comparison results between 2 benchmarks. The benchmark reports we push are generated with `critcmp`. Thus, we use `critcmp` to show the result of a benchmark, or compare results between multiple benchmarks.
We provide a script to download and display the comparison report. We provide a script to download and display the comparison report.
@ -71,12 +71,18 @@ List the available file in the DO Space:
```bash ```bash
songs_main_09a4321.json songs_main_09a4321.json
songs_geosearch_24ec456.json songs_geosearch_24ec456.json
search_songs_main_cb45a10b.json
``` ```
Run the comparison script: Run the comparison script:
```bash ```bash
# we get the result of ONE benchmark, this give you an idea of how much time an operation took
./benchmarks/scripts/compare.sh son songs_geosearch_24ec456.json
# we compare two benchmarks
./benchmarks/scripts/compare.sh songs_main_09a4321.json songs_geosearch_24ec456.json ./benchmarks/scripts/compare.sh songs_main_09a4321.json songs_geosearch_24ec456.json
# we compare three benchmarks
./benchmarks/scripts/compare.sh songs_main_09a4321.json songs_geosearch_24ec456.json search_songs_main_cb45a10b.json
``` ```
## Datasets ## Datasets

26
benchmarks/scripts/compare.sh
View File

@ -17,21 +17,9 @@ if [[ "$?" -ne 0 ]]; then
exit 1 exit 1
fi fi
if [[ $# -ne 2 ]]
then
echo 'Need 2 arguments.'
echo 'Usage: '
echo ' $ ./compare.sh old new'
echo 'Ex:'
echo ' $ ./compare.sh songs_main_09a4321.json songs_geosearch_24ec456.json'
exit 1
fi
old_file="$1"
new_file="$2"
s3_url='https://milli-benchmarks.fra1.digitaloceanspaces.com/critcmp_results' s3_url='https://milli-benchmarks.fra1.digitaloceanspaces.com/critcmp_results'
for file in $old_file $new_file for file in $@
do do
file_s3_url="$s3_url/$file" file_s3_url="$s3_url/$file"
file_local_path="/tmp/$file" file_local_path="/tmp/$file"
@ -45,6 +33,12 @@ do
fi fi
done done
# Print the diff changes between the old and new benchmarks path_list=$(echo " $@" | sed 's/ / \/tmp\//g')
# by only displaying the lines that have a diff of more than 5%.
critcmp --threshold 5 "/tmp/$old_file" "/tmp/$new_file" if [[ ${#@} -gt 1 ]]; then
# Print the diff changes between the old and new benchmarks
# by only displaying the lines that have a diff of more than 5%.
critcmp --threshold 5 $path_list
else
critcmp $path_list
fi