better handle noninstalled libs error during model loading

[eaidova]

What does this PR do?

I found an issue with loading models from extra-packages libraries (e.g. diffusers). TaskManager select and trying to load model even library is not installed in user environment and error message can be confusing for that case. This PR provides small improvement for this case.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@echarlaix, @IlyasMoutawwakil

[IlyasMoutawwakil]

can you give an example of the confusing error message you get with the current method ?

[eaidova]

File "/home/ea/miniforge3/lib/python3.12/site-packages/optimum/exporters/tasks.py", line 2066, in get_model_from_task
config = DiffusionPipeline.load_config(model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^
NameError: name 'DiffusionPipeline' is not defined

[IlyasMoutawwakil]

Yes I agree for the diffusers path the check is_diffusers_available was necessary, but for the other two, isn't it less verbose and more ambiguous like this ? loaded_library = importlib.import_module(library) will try to import the module and fail with the import error, whereas this suggestion will say that the module does not exist, for example when importing transformers, the import might fail due to incompatible tokenizers library or missing third party library, and not because transformers is missing.

[eaidova]

Yes I agree for the diffusers path the check is_diffusers_available was necessary, but for the other two, isn't it less verbose and more ambiguous like this ? loaded_library = importlib.import_module(library) will try to import the module and fail with the import error, whereas this suggestion will say that the module does not exist, for example when importing transformers, the import might fail due to incompatible tokenizers library or missing third party library, and not because transformers is missing.

I catch only ModuleNotFound exception for this case, so failure due to any other reason (e.g. issues with tokenizers) still will fail with original error, I do not see any problems and misleading here

But if you insist, I can stay only for diffusers case

[IlyasMoutawwakil]

I installed tokenizers==0.19 and ran the following:

ython 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.import_module("transformers")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/.local/lib/python3.10/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/home/user/.local/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 57, in <module>
    require_version_core(deps[pkg])
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/versions.py", line 117, in require_version_core
    return require_version(requirement, hint)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/versions.py", line 111, in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/versions.py", line 44, in _compare_versions
    raise ImportError(
ImportError: tokenizers>=0.20,<0.21 is required for a normal functioning of this module, but found tokenizers==0.19.0.
Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main

It gives better way to handle the issue than simply pip install transformers.
edit: ooh I see, so you only catch ModuleNotFound

[eaidova]

I installed tokenizers==0.19 and ran the following:

ython 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.import_module("transformers")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/.local/lib/python3.10/site-packages/transformers/__init__.py", line 26, in <module>
    from . import dependency_versions_check
  File "/home/user/.local/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 57, in <module>
    require_version_core(deps[pkg])
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/versions.py", line 117, in require_version_core
    return require_version(requirement, hint)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/versions.py", line 111, in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)
  File "/home/user/.local/lib/python3.10/site-packages/transformers/utils/versions.py", line 44, in _compare_versions
    raise ImportError(
ImportError: tokenizers>=0.20,<0.21 is required for a normal functioning of this module, but found tokenizers==0.19.0.
Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main

It gives better way to handle the issue than simply pip install transformers.

@IlyasMoutawwakil ImportError != ModuleNotFoundError, so you still get your desired error message, ther is no any problem for your provided case

[IlyasMoutawwakil]

LGTM, thnaks !

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[IlyasMoutawwakil]

@eaidova can you run styling please

[IlyasMoutawwakil]

I tried to fix styling from here but didn't work.. 😅

[echarlaix]

Could make sense to have function that works for all supported libraries (diffusers / timm / sentence-transformers), taking instead the library_name and doing a check with is_xxxxx_available, so that we can call this function and do this check for all libraries anytime needed