Keyword repeat

[mustafa0x]

How to emulate keyword repeat in quickwit/tantivy?

If placed before a stemmer, the result will be that you will get the unstemmed token preserved on the same position as the stemmed one. Queries matching the original exact term will get a better score while still maintaining the recall benefit of stemming. Another advantage of keeping the original token is that wildcard truncation will work as expected.

https://solr.apache.org/guide/solr/latest/indexing-guide/language-analysis.html#keywordrepeatfilterfactory

[fulmicoton]

Same thing as for lucene. Tokenizer can emit token with the a notion of position and token length.
https://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html

But you would have to modify the stemmer tokenizer to keep the original token to have that work.

[mustafa0x]

Thanks! So if I understand correctly it requires a custom tokenizer? The docs mention record: position, but what is also needed is not incrementing the position for repeated tokens.