WIP: Use poetry for dependency management

[weiji14]

Migrate dependency specification from setup.cfg to pyproject.toml. Using poetry to manage the dependencies.

References:

• https://realpython.com/dependency-management-python-poetry/

TODO:

• Copy minimum dependency pins from setup.cfg to pyproject.toml
• Modify CI workflow in .github/workflows/test.yaml to use poetry install instead of pip install
• Remove old pins in setup.cfg file
• Document new procedure somewhere

[adamjstewart]

Does poetry support pypa/pip#8085? This is the main feature I feel like pip is missing, and would allow us to test both the latest version and historical versions so that the minimum pins stay up to date.

[weiji14]

Does poetry support pypa/pip#8085? This is the main feature I feel like pip is missing, and would allow us to test both the latest version and historical versions so that the minimum pins stay up to date.

See python-poetry/poetry#3527. In the past, I have used a hacky solution which is to find-and-replace the >= with == in the pyproject.toml file and then run poetry lock to lock the dependencies to the minimum version. But not sure if I'd recommend it to be honest.

[adamjstewart]

If this PR is ever going to be merged, I'm going to need to see a clear and concise argument for why poetry is better than setuptools, flit, etc. If I was going to replace setuptools, I was actually thinking flit since it's simpler and designed for pure-Python packages like TorchGeo. But at the moment, setuptools seems to be the most stable and widely used build backend, and is used by the rest of the PyTorch community. All PyPA packaging guides still recommend using setuptools and pip for projects, and it's much easier to find support for these since they've been around longer.

I do really like poetry's ^ dependency specifier, which has better semver compliance than setuptools' ~= operator, but we can approximate that with >=/< if we need to. Other than that, I don't see a reason to move to poetry other than "it's new and exciting".

[adamjstewart]

One complaint I have with setuptools is that dependabot requires us to use a single requirements.txt, meaning that it's not possible to install only the documentation or style dependencies, you have to install everything. Does poetry solve this? For example, if dependabot requires the use of a single poetry.lock file, does this file allow for optional dependencies? Can you install only these optional dependencies, or do you also need to install the required dependencies too?

[weiji14]

Let's revisit this another time perhaps, I'm having trouble getting poetry to generate a lockfile for the 30+ dependencies in torchgeo. For some reason, the poetry solver doesn't finish even after 10+ minutes, I think due to the very tight pinning and/or crazy interdependencies. Also too many conflicts with the changes made recently in #551 and #557 so maybe better to start from scratch. There will be a poetry 1.2 release soon (see https://github.com/python-poetry/poetry/milestone/5) that might handle some of the complex requirements needed (e.g. optional dependencies and group dependencies), but still, it doesn't sound there's much appetite for beta products currently.

[adamjstewart]

I'm not opposed to beta products (many of our dependencies are still on 0.X versions), but there has to be a clear reason to switch. Setuptools/pip is more popular and seems to offer all of the same features as poetry, so it's unclear what advantage there is to switching. Anyway, I'll continue to keep my eyes on poetry/flit and see if that add any interesting new features.

[calebrob6]

@weiji14 full disclaimer, @adamjstewart is heavily biased towards using spack https://github.com/spack/spack/graphs/contributors :)

[adamjstewart]

Spack still calls pip to actually install packages, and pip calls the build backend (setuptools, poetry, flit) to build wheels, so Spack is certainly not mutually exclusive with poetry. But yes, many of the nice CLI features that poetry offers are lost on me because Spack already supports all of them.