10704 Commits

Author SHA1 Message Date
Kerollmops
024e06f7e3
Do not panic when the facet string is not found 2025-01-21 12:01:26 +01:00
Kerollmops
145fa3a8ff
Add a test to check the facet casing is good 2025-01-21 11:42:25 +01:00
Louis Dureuil
d3a7e10348
Unify facet strings by their normalized value 2025-01-21 00:11:50 +01:00
meili-bors[bot]
1c78447226
Merge #5246
Some checks failed
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests on ubuntu-20.04 (push) Failing after 1s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Run tests in debug (push) Failing after 17s
Test suite / Run Rustfmt (push) Failing after 16s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 40s
Test suite / Run Clippy (push) Failing after 1m24s
5246: Fix dump import r=Kerollmops a=dureuill

- Fix: handle the change of format of the update files
  - Correctly handle update files as JSON stream rather than obkv when exporting a dump with enqueued tasks
  - Correctly recreate update files as JSON stream rather than obkv when importing a dump
  - As the dump format itself didn't change, all dumps are still compatible
- Temporary workaround for https://github.com/meilisearch/meilisearch/issues/5247: set the batch uid of tasks to `null` at dump export time.
- Changes to meilitool
  - Export dump with update files in new format if DB >= v1.12
  - offline upgrade now supports upgrading from [1.9.0-1.12.5] to [1.10.0-1.12.5].
  - offline upgrade supports no-op upgrades and has better error messages 

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
Co-authored-by: ManyTheFish <many@meilisearch.com>
v1.12.5
2025-01-20 13:03:49 +00:00
Louis Dureuil
c55891f73b
Replace guards by OR patterns
Co-authored-by: Tamo <tamo@meilisearch.com>
2025-01-20 11:46:03 +01:00
Louis Dureuil
40f8c0d840
Remove batch ids on export 2025-01-20 11:16:18 +01:00
Louis Dureuil
34d8c1a903
Make offline upgrade more flexible 2025-01-20 10:43:47 +01:00
Louis Dureuil
3c9483b6e0
meilitool dumps old-style dump for older DBs, otherwise new-style 2025-01-20 10:43:47 +01:00
meili-bors[bot]
8c789b3c7a
Merge #5252
5252: Update version for the next release (v1.12.5) in Cargo.toml r=dureuill a=meili-bot

⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.

Co-authored-by: dureuill <dureuill@users.noreply.github.com>
2025-01-20 09:03:35 +00:00
dureuill
3403eae9ee Update version for the next release (v1.12.5) in Cargo.toml 2025-01-20 08:53:20 +00:00
Louis Dureuil
11458eefd9
Handle empty payloads 2025-01-20 09:51:07 +01:00
Louis Dureuil
289eb92bef
Fix warnings 2025-01-20 09:51:07 +01:00
ManyTheFish
cea0c89212
Change format of update file when importing dump 2025-01-20 09:51:07 +01:00
Louis Dureuil
1cadab9ad8
Also fix dump import from meilitool 2025-01-20 09:51:07 +01:00
Louis Dureuil
6383f8f19e
Do not explode on missing content file if the task has no docs 2025-01-20 09:51:06 +01:00
Louis Dureuil
8a9f952bda
Create update files in new format 2025-01-20 09:51:06 +01:00
meili-bors[bot]
a5c44b4d79
Merge #5242
Some checks failed
Test suite / Tests on ubuntu-20.04 (push) Failing after 13s
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 14s
Test suite / Run tests in debug (push) Failing after 11s
Test suite / Run Clippy (push) Successful in 7m16s
Test suite / Run Rustfmt (push) Successful in 2m41s
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Has been cancelled
5242: Fix infinite loop r=Kerollmops a=dureuill

- Fix possible infinite loop by releasing `writer_receiver` as soon as writing to DB panics
- Demote panic to error log

Co-authored-by: Louis Dureuil <louis@meilisearch.com>
v1.12.4
2025-01-16 14:13:07 +00:00
Louis Dureuil
8c35744848
Improve error log 2025-01-16 14:33:53 +01:00
meili-bors[bot]
c0d414fc3c
Merge #5243
5243: Update version for the next release (v1.12.4) in Cargo.toml r=dureuill a=meili-bot

⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.

Co-authored-by: dureuill <dureuill@users.noreply.github.com>
2025-01-16 10:57:55 +00:00
dureuill
b56358f606 Update version for the next release (v1.12.4) in Cargo.toml 2025-01-16 10:47:17 +00:00
Louis Dureuil
b84c0a5390
Demote panic to error log 2025-01-16 11:38:21 +01:00
Louis Dureuil
ce621e447e
Release writer_receiver as soon as writing to db panics 2025-01-16 11:37:37 +01:00
Clément Renault
aee74f47aa
Merge pull request #5229 from meilisearch/improve-unknown-entry-deletion-report
Improve the panic message when deleting an unknown entry
v1.12.3
2025-01-13 14:20:24 +01:00
Clément Renault
be2717edbd
Merge pull request #5224 from meilisearch/fix-facet-distribution
Fix facet distribution
2025-01-13 14:20:09 +01:00
Louis Dureuil
c66841626e
Update after review 2025-01-13 10:43:26 +01:00
Kerollmops
d0bc8c755a
Improve the panic message when deleting an unknown entry 2025-01-13 10:30:53 +01:00
meili-bors[bot]
031abfd281
Merge #5227
5227: Update version for the next release (v1.12.3) in Cargo.toml r=dureuill a=meili-bot

⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.

Co-authored-by: dureuill <dureuill@users.noreply.github.com>
2025-01-13 09:29:18 +00:00
dureuill
27169bc7b4 Update version for the next release (v1.12.3) in Cargo.toml 2025-01-13 08:47:07 +00:00
Louis Dureuil
181a01f8d8
Skip rebuilding field distribution if not coming from v1.12 2025-01-13 09:31:27 +01:00
Louis Dureuil
1d153c1867
write stats after rebuilding facet distribution 2025-01-09 18:13:36 +01:00
Louis Dureuil
5fde2a3ee1
Add support to upgrade to v1.12.3 in meilitool 2025-01-09 15:25:44 +01:00
Louis Dureuil
4465a1a3c9
Fix test 2025-01-09 13:26:17 +01:00
Louis Dureuil
e342ae1b46
Add currently failing test 2025-01-09 13:26:13 +01:00
meili-bors[bot]
dcb4c49cf2
Merge #5205
Some checks failed
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 12s
Test suite / Run tests in debug (push) Failing after 11s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 30s
Test suite / Run Rustfmt (push) Successful in 1m40s
Test suite / Run Clippy (push) Successful in 5m50s
5205: Incremental facets on v1.12 r=curquiza a=dureuill

# Pull Request

## Related issue

Fixes https://github.com/meilisearch/meilisearch/issues/5213

## What does this PR do?
- Add `new_incremental`module that computes incremental facet indexing for the new indexer
- Change heuristics for incremental vs bulk facet choice: under 1000 operations is now always incremental, over 100_000 operations is now always bulk)
- Add sanity checks in debug

## Future improvements

- Use multi ops from Roaring to decrease the number of allocations
- Consider removing or adding multiple levels at once instead of max once per update
- Consider using information about the tree structure + the operations that were done (e.g. only addition) to avoid recomputing the group from all children 
- Consider making the algorithm parallel and looking into the roaring values to know which actually changed



Co-authored-by: Louis Dureuil <louis@meilisearch.com>
v1.12.2
2025-01-08 21:52:02 +00:00
Louis Dureuil
e83c021755
When spilling on the next fid, no longer ignore children 2025-01-08 16:50:05 +01:00
Louis Dureuil
7ec7200378
Check valid_facet_value as part of a filter of the iterator 2025-01-08 16:25:44 +01:00
Louis Dureuil
6a577254fa
No longer ignore the first child without parent 2025-01-08 16:25:30 +01:00
Louis Dureuil
fd88c834c3
Modernize valid_lmdb_key 2025-01-08 15:22:11 +01:00
Louis Dureuil
b4005593f4
Switch to an iterative algorithm for find_changed_parents 2025-01-08 14:57:14 +01:00
Louis Dureuil
8ee3793259
Update after review 2025-01-08 13:58:14 +01:00
Louis Dureuil
3648abbfd5
Remove unused FacetFieldIdOperation 2025-01-07 15:26:09 +01:00
Louis Dureuil
4d2433de12
center groups prototype-incremental-facets-0 2025-01-06 18:23:35 +01:00
Louis Dureuil
28cc6df7a3
Fix uselessly deep stack trace 2025-01-06 18:07:49 +01:00
meili-bors[bot]
7b14cb10a1
Merge #5207
Some checks failed
Test suite / Tests on ${{ matrix.os }} (macos-13) (push) Waiting to run
Test suite / Tests almost all features (push) Has been skipped
Test suite / Test disabled tokenization (push) Has been skipped
Test suite / Tests on ubuntu-20.04 (push) Failing after 13s
Test suite / Tests on ${{ matrix.os }} (windows-2022) (push) Failing after 13s
Test suite / Run tests in debug (push) Failing after 12s
Test suite / Run Rustfmt (push) Successful in 1m40s
Test suite / Run Clippy (push) Successful in 5m49s
5207: Update version for the next release (v1.12.2) in Cargo.toml r=dureuill a=meili-bot

⚠️ This PR is automatically generated. Check the new version is the expected one and Cargo.lock has been updated before merging.

Co-authored-by: dureuill <dureuill@users.noreply.github.com>
2025-01-06 16:04:03 +00:00
Louis Dureuil
34f4602ae8
Update snapshot 2025-01-06 16:55:12 +01:00
dureuill
12e21a177b Update version for the next release (v1.12.2) in Cargo.toml 2025-01-06 14:11:58 +00:00
Louis Dureuil
7a9290aaae
Use new incremental facet indexing and enable sanity checks in debug 2025-01-06 15:08:48 +01:00
Louis Dureuil
5d219587b8
Add new incremental facet indexing 2025-01-06 15:08:36 +01:00
Louis Dureuil
6e9aa49893
add valid_facet_value utility function 2025-01-06 15:08:07 +01:00
Louis Dureuil
6b3a2c7281
Add sanity checks for facet values 2025-01-06 15:07:55 +01:00