Load a python script that contains a main function

[dvmorris]

I want to load this file and call the main function: https://raw.githubusercontent.com/dmlc/xgboost/d3f06467797acd61e80e87be38fb9812a016a64e/demo/guide-python/categorical.py

How do I do this? I see your example that loads a file from a gist that contains a single function, but I can't get this technique to work with the file above.

[operatorequals]

Hello @dvmorris
Can you please respond with what you've tried so far?

[dvmorris]

This example works:

import httpimport

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"

with httpimport.remote_repo(url):
  import hello

hello.hello()
# Hello world

This works:

$ pip install xgboost_ray

from xgboost_ray.examples import simple
simple.main(1, 4)

This does not work:

import httpimport

url = "https://raw.githubusercontent.com/ray-project/xgboost_ray/master/xgboost_ray/examples/simple.py"

with httpimport.remote_repo(url):
  import simple

simple.main()

ModuleNotFoundError: No module named 'simple'

[operatorequals]

I believe that removing the filename from the url of the non-working example will fix it as well.
The import statement looks for the file itself.

[dvmorris]

import httpimport

url = "https://raw.githubusercontent.com/ray-project/xgboost_ray/master/xgboost_ray/examples/"

with httpimport.remote_repo(url):
  import simple

simple.main(1, 4)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-3-c2511d889ed5>](https://localhost:8080/#) in <cell line: 17>()
     15   import simple
     16 
---> 17 simple.main(1, 4)

AttributeError: module 'simple' has no attribute 'main'

[rkbennett]

Does this test assume that xboost_ray is installed on the system, and you're just trying to import it via a remote script?

[dvmorris]

Yes, the issue with this particular use case is that this pip package doesn't include the examples in it.

…

On Tue, Nov 7, 2023, 2:08 PM rkbennett ***@***.***> wrote: Does this test assume that xboost_ray is installed on the system, and you're just trying to import it via a remote script? — Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABNPDLLEYCOM3YWLOWPOFDYDKIK3AVCNFSM6AAAAAA6ZNB36GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJZHA2DAOBQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rkbennett]

So I was looking into this a bit, and I think xboost_ray uses some odd python files. Saw a bunch of pyx in my testing. Haven't had time to look into it more though.

[boludoz]

Traceback (most recent call last):
  File "c:\Users\franc\OneDrive\Escritorio\ti.py", line 6, in <module>
    import hello
ModuleNotFoundError: No module named 'hello'

[rkbennett]

Traceback (most recent call last):
  File "c:\Users\franc\OneDrive\Escritorio\ti.py", line 6, in <module>
    import hello
ModuleNotFoundError: No module named 'hello'

Can you paste your code in here?

[rkbennett]

Yes, the issue with this particular use case is that this pip package doesn't include the examples in it.
…
On Tue, Nov 7, 2023, 2:08 PM rkbennett @.> wrote: Does this test assume that xboost_ray is installed on the system, and you're just trying to import it via a remote script? — Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABNPDLLEYCOM3YWLOWPOFDYDKIK3AVCNFSM6AAAAAA6ZNB36GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJZHA2DAOBQHA . You are receiving this because you were mentioned.Message ID: @.>

Are you running this on linux? I suppose I should have started with that question

[rkbennett]

@dvmorris I did some testing with another tool and I don't think there is actually an issue with being unable to import packages with main functions. I think it's likely a different issue which is that when the import happens it's missing a dependency (likely ray[train] or sklearn) which is causing a failure in the import, this then cascades back to an actual issue in this project, which is that when an error happens during import with httpimport it silently continues and you end up with broken packages showing imported, but they are never fully imported. If you really want to test this theory you could try my code from my PR. Otherwise, you can try it with my od_import project. Testing there it definitely has a main function

import od_import
config = {'type':'pastebin', 'visibility': 'unlisted', 'paste_key': 'wgkj6xLj', "module": "simple"}
with od_import.remote_source('https://pastebin.com', config=config):
  import simple

Having said that, in my testing, while I do get simple to import and it has a main function, when you attempt to run simple.main(1,4) like you do in your example I get an error that xgboost-ray training isn't supported on Windows. So if you're trying it on Windows it's not going to work.

For reference:

[boludoz]

Traceback (most recent call last):
  File "c:\Users\franc\OneDrive\Escritorio\ti.py", line 6, in <module>
    import hello
ModuleNotFoundError: No module named 'hello'

Can you paste your code in here?

The problem seems to be python 3.12, it doesn't happen with od_import.

[rkbennett]

Traceback (most recent call last):
  File "c:\Users\franc\OneDrive\Escritorio\ti.py", line 6, in <module>
    import hello
ModuleNotFoundError: No module named 'hello'

Can you paste your code in here?

The problem seems to be python 3.12, it doesn't happen with od_import.

Yeah, httpimport doesn't currently support 3.12, there's an open issue for adding that support, but in 3.12 they depricated the load_module function and completely removed the find_module function, which has broken compatibility with httpimport.

[operatorequals]

As shown here: #58 (comment)
Python 3.12 support finally came!

If you are still on it, can you please test again?