-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
testing python is too hard / cannot use 'python3' / python transition will cause problems #26818
Labels
Comments
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
Here is one thought, in py3-pendelum.yaml. Similar to the multi-version
package:
name: py3-pendulum
version: 3.0.0
epoch: 1
description: Python datetimes made easy
copyright:
- license: MIT
dependencies:
runtime:
- py${{vars.pyver}}-dateutil
- py${{vars.pyver}}-tzdata
- py3-typing
provides:
- py${{vars.pyver}}-pendulum
vars:
pyver: "3.12"
environment:
contents:
packages:
- build-base
- busybox
- ca-certificates-bundle
- py${{vars.pyver}}-maturin-bin
- py${{vars.pyver}}-pip
- py${{vars.pyver}}-wheel
- python-${{vars.pyver}}
- python-${{vars.pyver}}-dev
- rust
- wolfi-base
pipeline:
- uses: git-checkout
with:
repository: https://github.com/sdispater/pendulum
tag: ${{package.version}}
expected-commit: 0fcd10217af0469b3edda072f2b152d5273f3d58
- uses: py/pip-build-install
test:
pipeline:
- uses: python/import
with:
imports: |
import pendulum
- runs: |
# test verifies that timezone data is present.
python${{vars.pyver}} -c '
from pendulum import datetime
tor = datetime(2012, 1, 1, tz="America/Toronto")
van = datetime(2012, 1, 1, tz="America/Vancouver")
assert 3 == van.diff(tor).in_hours()
print("PASS: time zones check out")
' |
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * py3-cppy has a runtime dep on setuptools * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 21, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * py3-cppy has a runtime dep on setuptools * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 22, 2024
…d tests There are lots of changes here, centered around removing all occurences of SETUPTOOLS_SCM_PRETEND_VERSION. That seems to be not necessary now, at least with the current pip-build-install workflow. * do not require 'python' setting in pip-build-install. Default to using the target of the 'python3' link. If not present, user has to supply it. Also use contextdir rather than subpkgdir so it can be used from "main" packages also. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * py3-cppy has a runtime dep on setuptools * Add tests to touched packages * Drop 'python3' runtime dep from touched packages. * Fix uninstallable py3-gcloud-aio-storage which conflicted with its dependency py3-gcloud-aio-auth as they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That also fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * Fix pendulum's dependency that had py3-pytzdata, it's deps list 'tzdata' * Drop the now-unused py3-pytzdata Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 22, 2024
I set out to get rid of SETUPTOOLS_SCM_PRETEND_VERSION, and found some issues along the way. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages - melange sca does the right thing. * py3-cppy - add setuptools dep, found when adding a test. * py3-gcloud-aio-storage - Fix uninstallable paackage which conflicted with its dependency py3-gcloud-aio-auth. they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * py3-maturin - Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * py3-pendulum - fix dependency it should be tzdata not pytzdata * py3-pytzdata - Drop this, it is not used anywhere. Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Aug 22, 2024
I set out to get rid of SETUPTOOLS_SCM_PRETEND_VERSION, and found some issues along the way. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages - melange sca does the right thing. * py3-cppy - add setuptools dep, found when adding a test. * py3-gcloud-aio-storage - Fix uninstallable paackage which conflicted with its dependency py3-gcloud-aio-auth. they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * py3-maturin - Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * py3-pendulum - fix dependency it should be tzdata not pytzdata * py3-pytzdata - Drop this, it is not used anywhere. Note the 'python3' shell snippet in py3-rich and others is painful. See wolfi-dev#26818 for more context.
pnasrat
pushed a commit
that referenced
this issue
Aug 22, 2024
I set out to get rid of SETUPTOOLS_SCM_PRETEND_VERSION, and found some issues along the way. * replace any packages that had SETUPTOOLS_SCM_PRETEND_VERSION with py/pip-build-install. * Add tests to touched packages * Drop 'python3' runtime dep from touched packages - melange sca does the right thing. * py3-cppy - add setuptools dep, found when adding a test. * py3-gcloud-aio-storage - Fix uninstallable paackage which conflicted with its dependency py3-gcloud-aio-auth. they both provided usr/lib/python3.12/site-packages/gcloud/py.typed * py3-maturin - Fix the provides in py3-maturin-bin packages to provide py3-maturin-bin rather than just py3-maturin. That fixes py3-pendulum which was failing to build because it actually needs 'maturin' executable but that was not being provided by the 'py3-maturin' dep. * py3-pendulum - fix dependency it should be tzdata not pytzdata * py3-pytzdata - Drop this, it is not used anywhere. Note the 'python3' shell snippet in py3-rich and others is painful. See #26818 for more context.
smoser
added a commit
to smoser/melange
that referenced
this issue
Oct 2, 2024
Installing py3-pandas does not guarantee you a /usr/bin/python3. This is fallout of our choice to make pythons co-installable. More info on this wolfi-dev/os#26818 There isn't any serious value gained in this test here versus the pipeline import tests that are still there, so just drop it.
smoser
added a commit
to smoser/melange
that referenced
this issue
Oct 2, 2024
Installing py3-pandas does not guarantee you a /usr/bin/python3. This is fallout of our choice to make pythons co-installable. More info on this wolfi-dev/os#26818 Adding 'python-3' will get you _some_ python3, but currently not strictly guaranteed to be the same python3 that the pandas install was using. Another option would be to just drop the test here and in e2e-tests/py3-pandas-nopkg-test.yaml. diff --git a/e2e-tests/py3-pandas-test.yaml b/e2e-tests/py3-pandas-test.yaml index 2fedd23..8235448 100644 --- a/e2e-tests/py3-pandas-test.yaml +++ b/e2e-tests/py3-pandas-test.yaml @@ -24,5 +24,3 @@ test: with: import: ma from: numpy - - runs: | - python3 ./py3-pandas-test.py Signed-off-by: Scott Moser <[email protected]>
smoser
added a commit
to smoser/melange
that referenced
this issue
Oct 2, 2024
Installing py3-pandas does not guarantee you a /usr/bin/python3. This is fallout of our choice to make pythons co-installable. More info on this wolfi-dev/os#26818 Adding 'python-3' will get you _some_ python3, but currently not strictly guaranteed to be the same python3 that the pandas install was using. Another option would be to just drop the test here and in e2e-tests/py3-pandas-nopkg-test.yaml. Signed-off-by: Scott Moser <[email protected]>
smoser
added a commit
to smoser/melange
that referenced
this issue
Oct 2, 2024
Installing py3-pandas does not guarantee you a /usr/bin/python3. This is fallout of our choice to make pythons co-installable. More info on this wolfi-dev/os#26818 There isn't any serious value gained in this test here versus the pipeline import tests that are still there, so just drop it.
smoser
added a commit
to smoser/wolfi-os
that referenced
this issue
Oct 9, 2024
Convert 9 packages to multi-version python. This 'one-python' pipeline will create a 'python3' in the PATH pointing to the one /usr/bin/python3.XX that is present. That way the caller can be blissfully ignorant that their package didn't actually need 'python3' installed at all. This makes an improvement here on wolfi-dev#26818
smoser
added a commit
that referenced
this issue
Oct 9, 2024
Convert 9 packages to multi-version python. This 'one-python' pipeline will create a 'python3' in the PATH pointing to the one /usr/bin/python3.XX that is present. That way the caller can be blissfully ignorant that their package didn't actually need 'python3' installed at all. This makes an improvement here on #26818
lyoung-confluent
pushed a commit
to lyoung-confluent/melange
that referenced
this issue
Oct 10, 2024
Installing py3-pandas does not guarantee you a /usr/bin/python3. This is fallout of our choice to make pythons co-installable. More info on this wolfi-dev/os#26818 There isn't any serious value gained in this test here versus the pipeline import tests that are still there, so just drop it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Because we have multiple pythons and we want them to be co-installable, we have packages like:
The -base package provides basically everything and the python-3.11 package provides a symlink making 'python3' default to python 3.11 (
/usr/bin/python3 -> python3.11
)When we build packages, melange correctly identifies that a package that installs python files into
/usr/lib/python3.12/site-packages/
depends onpython-3.12-base
. It does not depend onpython-3.12
(it never uses that symlink. even the installed scripts have /usr/bin/python3.12).The pipeline 'python/imports' pipeline takes a 'python' input, but if it is not provided it will use a shell snippet to find the right one (if there is only one installed).
That works well for the imports pipeline.
The problem is that anything that wants to run 'python3' in a test has to have a similar snippet of code.
Consider py3-async-generator.yaml.
It looks like this:
Two things to call out there
python-3
dep. In the future, when python-3.13 exists, python-3.13 will also provide python-3 and will be preferred to python-3.12. So installation of this package would install 2 python versions. Yikes.python-3
in the test environment in order to get apython
program to execute. But similarly to 1, that will install the wrong python in the future, and the test will begin to fail (it works today, but as soon as there is a python-3 package it will fail).So, how should that test look? Do we have to copy a shell snippet like this:
The text was updated successfully, but these errors were encountered: