fix deprecated tokenizer methods, add test cases#139

Merged

johnml1135 merged 3 commits intomainfrom #137_lang_code

Nov 8, 2024

+150-36

Commits on Nov 8, 2024

fix deprecated tokenizer methods, add test cases

mshannon-sil
authored and
johnml1135
committed
make add_lang_code_to_tokenizer a public function again

mshannon-sil
authored and
johnml1135
committed
fix test import

mshannon-sil
authored and
johnml1135
committed