Add a docker/dagger-based testing workflow, adopt PEP 517/621

[Zeitsperre]

Changes

• Stages a docker-based testing workflow (using Dagger) for running integration testing workflows both locally and on CI systems.
  • Include a linting stage using pre-commit
• Adds a Dockerfile to stage the testing image
• Adds a yaml linting step and a documentation code style checking step to the pre-commit configuration.
• Updates the name of sphinx-rtd-theme (previously sphinx_rtd_theme).
• Updates the setup build system to use PEP 517/PEP 621 using flit.

[tlvu]

Great stuff. I am not too familiar with Dagger and Github actions in general but I just don't see where the same pytest as our Jenkins is being run.

Also same questions as all other pipelines you've been adding lately: can we trigger this manually without having to push new commits?

[tlvu]

latest is whenever someone push something to master, which is never tested ! The current version in production is 230601, this one should be the default. Hum, how to keep this one up-to-date without additional manual commit on each Jupyter env release ...

[Zeitsperre]

@tlvu @tlogan2000

While it takes several hours to run, it does indeed run! I haven't checked why, but a handful of notebooks are currently failing (many of them are passing). I imagine that there might be some missing configurations/environment variables.

Given the egregious amount of time needed to run these, I think we might need to either simplify some tests or only run this action via workflow dispatch or on a weekly/biweekly schedule. Thoughts?

[Zeitsperre]

@tlvu

Issues affecting the notebooks:

• Docker image is missing pyogrio (eccc-geoapi-climate-stations.ipynb, eccc-geoapi-xclim)
• Docker image is missing pyesgf (esgf-dap.ipynb)
• Issues with ESMPy installation (ImportError: The ESMFMKFILE environment variable is not available.; regridding.ipynb)
• Mismatch in expected return (?) (mismatch 'application/javascript'; forecasts.ipynb::Cell 2)

[huard]

xESMF error may be related to pangeo-data/xESMF#246

[tlvu]

@tlvu

Issues affecting the notebooks:

* Docker image is missing `pyogrio` (eccc-geoapi-climate-stations.ipynb, eccc-geoapi-xclim)

* Docker image is missing `pyesgf` (esgf-dap.ipynb)

* Issues with ESMPy installation (`ImportError: The ESMFMKFILE environment variable is not available.`; regridding.ipynb)

@Zeitsperre

Nothing is missing in the Jupyter docker image. As I said in comment #300 (comment) earlier, the default Jupyter image should not be latest because that one is rebuilt randomly and never tested. Please use the current tested production version 230601.

* Mismatch in expected return (?) (`mismatch 'application/javascript'`; forecasts.ipynb::Cell 2)

This one is not expected. Intermittent or 100% reproducible? Might be due to latest image again. Retry with production 230601.

[tlvu]

@tlvu @tlogan2000

While it takes several hours to run, it does indeed run! I haven't checked why, but a handful of notebooks are currently failing (many of them are passing). I imagine that there might be some missing configurations/environment variables.

https://github.com/Ouranosinc/pavics-sdi/actions/runs/5600724938/job/15171619608

Wow =========== 53 failed, 89 passed, 15 warnings in 17512.32s (4:51:52) ===========

@Zeitsperre
All the html and embeded javascript diff error because I forgot to tell you how to ignore them in output change, sorry !: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/d2c89f6ef72e189fbba90e7370d6c1aacdd7bbdf/conftest.py#L1-L6

Once we clear those noise and use the proper Jupyter image version, we can better see what's left to fix.

Given the egregious amount of time needed to run these, I think we might need to either simplify some tests or only run this action via workflow dispatch or on a weekly/biweekly schedule. Thoughts?

Can we run daily to catch errors faster? Agreed that almost 5h runtime duration is too much to run on every PR merge. However, if not run on each PR merge, how to automatically associate any errors found to the proper PR?

[Zeitsperre]

@tlvu

Wow =========== 53 failed, 89 passed, 15 warnings in 17512.32s (4:51:52) ===========

@Zeitsperre All the html and embeded javascript diff error because I forgot to tell you how to ignore them in output change, sorry !: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/d2c89f6ef72e189fbba90e7370d6c1aacdd7bbdf/conftest.py#L1-L6

Added the override in and directed pytest to use it, but it doesn't seem to do anything? Am I missing something here?

Can we run daily to catch errors faster? Agreed that almost 5h runtime duration is too much to run on every PR merge. However, if not run on each PR merge, how to automatically associate any errors found to the proper PR?

Done. I've set it to run at 4:00 AM every day (so that we see the errors bright and early a few hours later).

[tlvu]

Wow =========== 53 failed, 89 passed, 15 warnings in 17512.32s (4:51:52) ===========
@Zeitsperre All the html and embeded javascript diff error because I forgot to tell you how to ignore them in output change, sorry !: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/d2c89f6ef72e189fbba90e7370d6c1aacdd7bbdf/conftest.py#L1-L6

Added the override in and directed pytest to use it, but it doesn't seem to do anything? Am I missing something here?

@Zeitsperre
Nothing special to add.

However I noticed your conftest.py file is located at /code/tests while all your notebooks are under /code/docs/. Not sure, but maybe they should be under the same tree? Or maybe your starting working dir should be were the conftest.py file is located?

[tlvu]

@Zeitsperre See comment #300 (comment)

Nothing is missing in the Jupyter docker image. As I said in comment #300 (comment) earlier, the default Jupyter image should not be latest because that one is rebuilt randomly and never tested. Please use the current tested production version 230601.