Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] PyO3 walking skeleton for Python #3804

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

[WIP] PyO3 walking skeleton for Python #3804

wants to merge 21 commits into from

Conversation

dbrattli
Copy link
Collaborator

@dbrattli dbrattli commented Apr 10, 2024

Initial scaffolding of rewriting fable-library for Python in Rust using PyO3 and Maturin. Note that this will be a gradual migration, not a big-bang release. The idea is to over time move more of the Python code from fable-library over to Python extension modules written in Rust. This will give us much better performance and solve some edge cases where it's really hard to mimic .NET e.g for sized ints and arrays. It will also give us the opportunity to share most of the codebase with Fable Rust that will benefit both projects.

Fixes #3766

  • Use sized and signed integers from Rust instead of Python int
  • Use Rust array instead of Python list/array/bytearray
  • Reuse and share code with fable-library-rust as much as possible
    • Build fable library .fs files using Rust instead of Python so we can use them in extension modules
  • CI/CD now handles and publishes Python wheels for all architectures. It's probably better to publish the wheel to PyPI instead of distributing binaries ourselves in the nuget file, or?

@dbrattli dbrattli changed the title [Python] PyO3 walking skeleton [WIP] PyO3 walking skeleton for Python Apr 10, 2024
@ncave
Copy link
Collaborator

ncave commented Apr 10, 2024

@dbrattli You can convert this to draft PR so it won't be merged until ready.

@ncave
Copy link
Collaborator

ncave commented Apr 10, 2024

@dbrattli Just to clarify, while this is a great idea, do you have any concerns about losing some of the (potential) benefits of easy interoperability with other Python code by not using "native" Python types (or at least numpy types), or is that not a big deal if it can be provided through some interop layer?

@dbrattli dbrattli marked this pull request as draft April 10, 2024 20:05
@dbrattli
Copy link
Collaborator Author

@ncave It was so hard to find the convert to draft link 🙈 but fixed now 😊 The custom Rust sized and signed integers will be implemented so they appear as regular ints to other Python code. Python will also coerce them to normal Python int when needed, e.g a function that expects a regular Python int, or if you try to do any arithmetic using them. E.g:

>>> from array import array # This is a Python array that expects Python ints
>>> from core import uint16 # This is a `pub struct uint16(u16);` from Rust.
>>> x = uint16(42)
>>> array("H", [x]) # works where Python expects a Python int
array('H', [42])
>>> x+5
47
>>> 5+x
47

Most of Python (and NumPy) is already implemented the same way using extension modules. NumPy is implemented in C++ and Python (cpython) is implemented in C. So interop will not be a problem since other Python code will not know the difference. Everything will appear as regular Python modules and can be used by any Python code. A lot of new Python libraries are in fact implemented in Rust these days, e.g Pydantic v2. We just need to provide .pyi files so that static type checkers knows what is going on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Python] Boxing bytes looses type information
2 participants