Fix deprecated import for Transformers v5 compatibility
#19
by
hmellor HF Staff - opened
bytes_to_unicodehas been available insrc/transformers/convert_slow_tokenizer.pyfor a long time.bytes_to_unicodewill be removed fromsrc/transformers/models/gpt2/tokenization_gpt2.pyin Transformers v5.
This PR updates the import, I recommend you do the same for any other custom models you have which import from the old location.
yzhangcs changed pull request status to
merged