Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace deprecated tokenizer methods #137

Closed
mshannon-sil opened this issue Nov 5, 2024 · 0 comments · Fixed by #139
Closed

Replace deprecated tokenizer methods #137

mshannon-sil opened this issue Nov 5, 2024 · 0 comments · Fixed by #139
Assignees
@mshannon-sil
Labels
bug Something isn't working

Comments

@mshannon-sil
Copy link
Collaborator

With the upgrade to the latest version of transformers, the lang_code_to_id attribute was removed from the NllbTokenizer, which is now causing our NMT jobs to crash. This needs to be fixed, and test cases that cover the relevant lines need to be added.
@mshannon-sil mshannon-sil added the bug Something isn't working label Nov 5, 2024
@mshannon-sil mshannon-sil self-assigned this Nov 5, 2024
@github-project-automation github-project-automation bot moved this to 🆕 New in Serval Nov 5, 2024
@mshannon-sil mshannon-sil moved this from 🆕 New to 🏗 In progress in Serval Nov 5, 2024
@mshannon-sil mshannon-sil linked a pull request Nov 6, 2024 that will close this issue
fix deprecated tokenizer methods, add test cases #139
Merged
@github-project-automation github-project-automation bot moved this from 🏗 In progress to ✅ Done in Serval Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mshannon-sil mshannon-sil

Labels
bug Something isn't working
Projects
Serval
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix deprecated tokenizer methods, add test cases sillsdev/machine.py
1 participant
@mshannon-sil