release: fix release pipeline

[robaerd]

Fixes issues of the release pipeline mentioned in #3528, adds the ubuntu focal docker image and improves log output of the release script.

Basics

These points need to be fulfilled for every PR:

• Short descriptions of your changes are in the release notes
  (added as entry in doc/news/_preparation_next_release.md which
  contains _(my name)_)
  Please always add something to the release notes.
• Details of what you changed are in commit messages
  (first line should have module: short statement syntax)
• References to issues, e.g. close #X, are in the commit messages.
• The buildservers are happy.
• The PR is rebased with current master.

If you have any troubles fulfilling these criteria, please write
about the trouble as comment in the PR. We will help you.
But we cannot accept PRs that do not fulfill the basics.

Checklist

Check relevant points but please do not remove entries.
For docu fixes, spell checking, and similar none of these points below
need to be checked.

• I added unit tests for my code
• I fully described what my PR does in the documentation
  (not in the PR description)
• I fixed all affected documentation
• I added code comments, logging, and assertions as appropriate (see Coding Guidelines)
• I updated all meta data (e.g. README.md of plugins and METADATA.ini)
• I mentioned every code not directly written by me in THIRD-PARTY-LICENSES

Review

Reviewers will usually check the following:

• Documentation is introductory, concise, good to read and describes everything what the PR does
• Examples are well chosen and understandable
• Code is conforming to our Coding Guidelines
• APIs are conforming to our Design Guidelines
• Code is consistent to our Design Decisions

Labels

If you are already Elektra developer:

• Add the "work in progress" label if you do not want the PR to be reviewed yet.
• Add the "ready to merge" label if the basics are fulfilled and you also
  say that everything is ready to be merged.

[markus2330]

Thank you for creating the PR!

scripts/dev/configure-debian needs to be adapted to use the cmake flags (KDB_DB_SYSTEM, etc.).

Please copy the script then to scripts/release or simply do the cmake call directly in your release script. The scripts in scripts/dev are meant to be for development, which sometimes has other goals than release management.

[mpranj]

Please copy the script then to scripts/release or simply do the cmake call directly in your release script. The scripts in scripts/dev are meant to be for development, which sometimes has other goals than release management.

Yes, do whatever is simple for now (probably call CMake directly). Which parameters are used for release is a separate issue now #3531.

[robaerd]

I'm having issues building the deb package in ubuntu focal because of some missing python files.
gbp buildpackage -sa returns following error:

....
make[1]: Entering directory '/home/jenkins/DEV-elektra/libelektra'
dh_install --list-missing
dh_install: warning: Please use dh_missing --list-missing/--fail-missing instead
dh_install: warning: This feature will be removed in compat 12.
dh_install: warning: Cannot find (any matches for) "usr/lib/python3/*-packages/kdb/*.so" (tried in ., debian/tmp)

dh_install: warning: python3-elektra missing files: usr/lib/python3/*-packages/kdb/*.so
dh_install: warning: Cannot find (any matches for) "usr/lib/python3/*-packages/kdb/*.py" (tried in ., debian/tmp)

dh_install: warning: python3-elektra missing files: usr/lib/python3/*-packages/kdb/*.py
dh_install: warning: Cannot find (any matches for) "usr/share/libelektra-test/test-data/python/*" (tried in ., debian/tmp)

dh_install: warning: elektra-tests missing files: usr/share/libelektra-test/test-data/python/*
dh_install: error: missing files, aborting
make[1]: *** [debian/rules:72: override_dh_install] Error 25
make[1]: Leaving directory '/home/jenkins/DEV-elektra/libelektra'
make: *** [debian/rules:57: binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui -i -I -sa failed
gbp:error: 'debuild -i -I -sa' failed: it exited with 29

$ dh_missing --fail-missing
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v1/again_two_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v1/meta_data.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v1/one_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v1/three_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v1/two_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/again_two_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/demo.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/fullnames.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/meta_data.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/one_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/three_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/dump/v2/two_value.dump exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/kconfig/meta_example.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/kconfig/meta_examplerc exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/kconfig/simple_example.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/kconfig/simple_examplerc exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/kconfig/test_valid.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/share/libelektra-test/test-data/kconfig/test_validrc exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/lib/x86_64-linux-gnu/pkgconfig/elektra-merge.pc exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/lib/x86_64-linux-gnu/elektra/tool_exec/cmerge-config-files exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/lib/x86_64-linux-gnu/elektra/tool_exec/install-config-file exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/lib/x86_64-linux-gnu/elektra/tool_exec/gen-gpg-testkey exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/include/elektra/kdbglobbing.h exists in debian/tmp but is not installed to anywhere
dh_missing: warning: usr/include/elektra/kdbmerge.h exists in debian/tmp but is not installed to anywhere
	The following debhelper tools have reported what they installed (with files per package)
	 * dh_install: elektra-bin (81), elektra-bin-extra (12), elektra-dbg (0), elektra-doc (637), elektra-qt-gui (6), elektra-tests (372), libelektra-dev (78), libelektra4 (85), libelektra4-all (0), libelektra4-augeas (1), libelektra4-crypto (1), libelektra4-curl (1), libelektra4-dbus (2), libelektra4-experimental (14), libelektra4-extra (39), libelektra4-full (7), libelektra4-gen (0), libelektra4-journald (1), libelektra4-lua (1), libelektra4-python (1), libelektra4-xerces (1), libelektra4-xmltool (1), libelektra4-yajl (1), libelektra4-zeromq (2), lua-elektra (1), python-elektra (0), python3-elektra (0)
	If the missing files are installed by another tool, please file a bug against it.
	When filing the report, if the tool is not part of debhelper itself, please reference the
	"Logging helpers and dh_missing" section from the "PROGRAMMING" guide for debhelper (10.6.3+).
	  (in the debhelper package: /usr/share/doc/debhelper/PROGRAMMING.gz)
	Be sure to test with dpkg-buildpackage -A/-B as the results may vary when only a subset is built
	For a short-term work-around: Add the files to debian/not-installed
dh_missing: error: missing files, aborting

@markus2330 @mpranj Do you maybe know why this is failing?

[mpranj]

Do you maybe know why this is failing?

Most likely the python files are missing because the bindings were not built (see #3522). So #3522 has to be solved first.

[robaerd]

So should I build the package without the bindings or wait until #3522 is solved or try building it using SWIG 4.0.2 like suggested in the issue?

[markus2330]

try building it using SWIG 4.0.2 like suggested in the issue?

Yes, please try as suggested 💖 I assigned you the issue.

First try in the console: if cmake does not say Including Binding python you do not need to try any further steps, describe what happens in #3522.

[mpranj]

Some of these things, like:

libelektra/scripts/release/release.sh

Line 55 in 724ec34

git commit -m "Debian Package $DVERSION (UNRELEASED)"

will have to be pushed. Otherwise we will need to reconstruct it manually. What do you think @markus2330?

The alternative would be to archive all git repos completely to the artifacts. Then whoever does the release downloads the artifacts and does the push.

[markus2330]

will have to be pushed. Otherwise we will need to reconstruct it manually. What do you think @markus2330?

I am not sure if I understand the question. I think we should keep pushing the Debian branch, at least for this release.

The alternative would be to archive all git repos completely to the artifacts. Then whoever does the release downloads the artifacts and does the push.

I thought we will do this for this release? Or simply leave it in the workspace if it gets too big for the artifacts.

[mpranj]

I am not sure if I understand the question.

The release script merges master to the debian branch. It never pushes these changes. Then it builds packages based on the unpushed debian branch. After the release is basically done, the debian branch stays outdated. We then need to manually re-do the steps again to the debian branch s.t. everything is synced.

[markus2330]

After the release is basically done, the debian branch stays outdated. We then need to manually re-do the steps again to the debian branch s.t. everything is synced.

Long term we could consider that the release scripts also push everything. For now, yes: get the whole git repo, check it, and push it. Or did I misunderstand something?

[mpranj]

Long term we could consider that the release scripts also push everything. For now, yes: get the whole git repo, check it, and push it. Or did I misunderstand something?

Yes, but the repo is not part of the artifacts, so it's up to your luck if the workspace has been cleared and to find on which agent it was done. Or am I misunderstanding something?

[markus2330]

In manually triggered pipelines that do not clean the workspace after running there should be no problem, at least as long we have a single stage. (With parallel stages there might be multiple workspaces on different computers.)

[robaerd]

Yes, but the repo is not part of the artifacts

The libelektra repo and the doc repo is now bundled and part of the artifacts.

... at least as long we have a single stage. (With parallel stages there might be multiple workspaces on different computers.)

We already have parallel stages. The Ubuntu Focal and the Debian Buster are built in parallel stages.

[mpranj]

@robaerd this is already looking really good. Do you think we can use it for a release today or what's the state? I see that you solved most obstacles quickly.

Will you do the version increments on master etc. or can I do anything that is not part of your thesis/project?

[robaerd]

@robaerd this is already looking really good. Do you think we can use it for a release today or what's the state? I see that you solved most obstacles quickly.

Thank you! I hope so, but I'm still waiting for jenkins to finish the current test run.

Will you do the version increments on master etc. or can I do anything that is not part of your thesis/project?

I already updated the versions locally and will soon make a PR.
Could you maybe update the doc/COMPILE.md to reflect the actually tested setups?

Should the tasks in prep for next release be done now, or after the final jenkins release run?

[mpranj]

I already updated the versions locally and will soon make a PR.

I will update the release notes on top of your PR, if this is OK for you!

Could you maybe update the doc/COMPILE.md to reflect the actually tested setups?

Yep I will take a look too, but it is not high priority.

Should the tasks in prep for next release be done now, or after the final jenkins release run?

After everything is done and released.

[robaerd]

I'm getting following memory leak error in the release pipeline:

99% tests passed, 1 tests failed out of 135

Label Time Summary:
kdbtests    = 243.95 sec*proc (9 tests)

Total Test time (real) = 308.12 sec

The following tests FAILED:
	243 - testkdb_allplugins (Failed)
-- Processing memory checking output:
135/135 MemCheck: #243: testkdb_allplugins ...............   Defects: 2
MemCheck log files can be found here: ( * corresponds to test number)
/home/jenkins/workspace/libelektra-release@2/libelektra/build/Testing/Temporary/MemoryChecker.*.log
Memory checking results:
Memory Leak - 2
        Errors while running CTest
   Site: 2ce001983854
   Build name: Linux-c++
Memory check project /home/jenkins/workspace/libelektra-release@2/libelektra/build
    Start 243: testkdb_allplugins
135/135 MemCheck #243: testkdb_allplugins ...............***Failed    5.39 sec

The documentation build stage is still scheduled and it will probably take a few hours till its is run, because of the PR jobs that are currently being run. But you can view the log of the release stage here and the artifacts here.
@markus2330 @mpranj Do you have an idea why this is happening?

[mpranj]

I have no idea judging by the logs.. Can you archive the memory checker logs to the artifacts as well? Maybe they know something

build/Testing/Temporary/MemoryChecker.*.log

I think the build dir is cleaned so I could not find it on the build agents either.

[markus2330]

The load was very high, it peaked at 30 (v2) and 7 (a7), now it is low again.

During release it is totally fine if you pause or cancel (if you restart later) the jobs. In this specific case, however, it will not help as #3540 is relevant for the release.

[mpranj]

I am actively aborting any irrelevant jobs from my side to keep the infrastructure available for release testing. Only really needed other tests are running.

[robaerd]

Can you archive the memory checker logs to the artifacts as well? Maybe they know something

I archived the logs. Until the job is done the memory checker logs can already be accessed here

[mpranj]

When setting the -DENABLE_DEBUG=ON CMake option to see the symbols the problem is gone. This might be another instance of #2320, but I do not know.

[mpranj]

@robaerd please ping me when you think you are done and it makes sense for me to take a look and do the release.

As Markus already mentioned don't spend too much time on perfecting every detail. When doing a release by hand mistakes can also happen. Your work will even enable us to release fixes more quickly.

A note for later: currently the memchecks are failing (irrelevant as Markus said), but no red flag is raised. I think in the long run we should either abort or somehow make it really clear when a step of the release failed. Otherwise it would be easy to overlook when reading 50k lines of logs.

[robaerd]

I think in the long run we should either abort or somehow make it really clear when a step of the release failed.

I fully agree. Initially the script exited on errors, but I had to remove this configuration because of the memcheck error.

@robaerd please ping me when you think you are done and it makes sense for me to take a look and do the release.

It looks like everything works like expected, so the PR can probably be already merged. The "build artifacts" stage on the normal jenkins pipeline will fail since the master branch is not yet merged into the debian branch.
After the merge of this PR we will need to re-run the release pipeline since I always replayed it with my fork of the libelektra repository.

[mpranj]

Thank you, this is really great work! 🏅

I also think it's ready for master. Can you please point jenkins to the main repo after that?

[mpranj]

Thank you, this looks great! We should also add it to the normal pipeline.

[robaerd]

I will do this as soon as possible.