New scheduler breaks new installation with circular dependencies

[kevemueller]

The new scheduler implemented via #2330 changes previously accepted package installations.
Previously two packages depending on each other could be installed. E.g.

• libarchive depends on runtime
• runtime depends on libarchive

On an empty database pkg install runtime installed both packages.

The new scheduler determines the needed packages to be installed, but then bails with

DBG(3)[90946]> (scheduler) checking job scheduling graph for cycles...
DBG(4)[90946]> (scheduler) job: install runtime
DBG(4)[90946]> (scheduler)   edge to install libarchive, new depends on new
DBG(4)[90946]> (scheduler) job: install libarchive
DBG(4)[90946]> (scheduler)   edge to install runtime, new depends on new
DBG(3)[90946]> (scheduler) job scheduling graph cycle found
pkg: found job scheduling cycle without upgrade job
pkg: Package database is busy while closing!

@ifreund
I wonder if this use case was considered and the change breaking previously acceptable lenient functionality made deliberately.

[ifreund]

Hmm, from my perspective I don't see a convincing use-case for circular dependencies. In the example given where libarchive depends on runtime and runtime depends on libarchive it seems to me that this is a strong indication that they should be merged into a single package.

The FreeBSD porter's handbook agrees that circular dependencies are to be avoided: https://docs.freebsd.org/en/books/porters-handbook/makefiles/#makefile-circular-dependencies

Do not introduce any circular dependencies into the ports tree!

However, since pkg has allowed installing circular dependencies in the past, there are almost certainly existing pkg databases in the wild with circular dependencies installed. In its current state, the new scheduler will not only refuse to install packages part of a circular dependency loop but refuse to upgrade or delete such packages as well. This of course poses a problem.

It's impossible for any scheduler to uphold the desired ordering guarantees in the presence of circular dependencies. We could however add some kind of optional bypass that breaks the strict ordering guarantees made by the scheduler and allows fixing an old pkg database with circular dependencies installed by uninstalling/upgrading packages to break the dependency loop.

It's not clear to me how widespread existing databases with circular dependencies installed are in practice. Neither @bapt or I had any issues with this while testing. Could you share more about your use case/where you are obtaining these packages with circular dependencies from?

[bapt]

pkg should not accept circular dependencies and we should kill them if any

[kevemueller]

Hi Isaac,

thank you for the detailed answer. I ran into this while looking at https://bugs.freebsd.org/bugzilla//show_bug.cgi?id=265061
I have created the packages myself resolving at package creation time the shared library dependencies and making them hard dependencies between the packages.
This leads to the unfortunate circularity in package dependencies, but was not an issue with pkg install until now.

With externalizing the problem into shlibs_required/shlibs_provided you are allowing and in fact relying on "soft" circularity between packages.

I am always in favour of more strict/well defined, so for my use case I will need to look for a different angle to solve it.
Feel free to close this as "works as intended".

Cheers,

[ifreund]

@kevemueller I've actually been looking into PR 265061 as well, I just opened #2333 and outlined the next steps I plan to take as a comment on #2331. Let's keep discussion specific to that issue/potential solutions out of this thread though.

In any case, if we hit circular dependencies in pkgbase packages I think the correct solution is to restructure the packages such that there are no circular dependencies.

I personally would be fine closing this issue for now and kicking the can down the road to when a user complains about pkg refusing to delete/upgrade circular dependencies in their existing pkg database. It seems too optimistic to assume that will never happen, though it will be good to identify where the circular dependencies came from when it eventually does I guess.

[kevemueller]

Thx & keep up the good work!