Make bulk/testport -b URL not only support official repositories, but any repository

[michael-o]

Prerequisites

• Have you checked for an existing issue describing your idea?

What is your proposal?

Remove the terms "official only" from this option and if any of the symbolic names, latest, quarterly, release_X keep repo name FreeBSD and enforce SRV RR, signatures, but when a custom URL is provided, set FreeBSD to no, add a new repo, e.g., custom where priority is higher (10?) and hardcode mirror_type, pubkey, signature_type to NONE. A more sophisticated way is to accept a file path where the admin can supply custom .conf which contains arbitrary repos copied to LOCALBASE into the jail. In this case it needs to be clarified how to provide public keys, if required. (This will provide nice layering of repos)

What is the existing behavior, if any?

When providing the custom, selfhosted repo with -b <URL> the fetch fails because my packages aren't signed which I don't need nor is this an official repo.

What is the motivation / use case for the change?

I build a base set of ports with custom options because the default ones do not work for our environment. Those options aren't only for direct, but also for automatic ones. On top there are several other use cases (sets) which use those base ports partially as dependency which I don't want to build over and over again. The option for bulk/testport -b <URL> expects only official repos which quite limits this nice feature. I want to use it with any selfhosted repo which will satisfy the fetch requirements.

Did you consider any alternatives?

Yes, patching common.sh, but this is an ugly hack. More specificly, in

poudriere/src/share/poudriere/common.sh

Lines 3805 to 3809 in 5f69efb

	cat >> "${MASTERMNT}/etc/pkg/poudriere.conf" <<-EOF
	FreeBSD: {
	url: ${packagesite};
	}
	EOF

I had to switch of signature check and DNS SRV RRs to use my custom repo.
Hack:

@@ -3804,7 +3797,10 @@
    fi
    cat >> "${MASTERMNT}/etc/pkg/poudriere.conf" <<-EOF
    FreeBSD: {
-           url: ${packagesite};
+           url: ${packagesite},
+            mirror_type: "NONE",
+            pubkey: "NONE",
+            signature_type: "NONE";
    }
    EOF

Is this really a ports feature request?

Yes, it is a poudriere issue.

Additional context

This was found in #883.

Real world example

port set ldadw_base:

poudriere bulk -j 123-release-amd64 -z ldadw_base -O ldadw-custom -f /usr/local/etc/poudriere.d/123-release-amd64-default-ldadw_base-ports

with the following in /usr/local/etc/poudriere.d/:

123-release-amd64-default-ldadw_base-make.conf
123-release-amd64-default-ldadw_base-options/
123-release-amd64-default-ldadw_base-ports

123-release-amd64-default-ldadw_base-ports:

devel/git
devel/subversion-lts
devel/tig
editors/vim
editors/vim-colorschemes-legacy
ftp/curl
lang/python3
lang/python37
devel/py-pip
misc/mc
net/openldap26-client
security/krb5
shells/bash
shells/bash-completion
sysutils/htop
sysutils/msktutil
sysutils/tree
textproc/colordiff
textproc/rubygem-asciidoctor
net-mgmt/zabbix5-agent
security/siemens-cacerts
security/nss-siemens-cacerts-java
dns/register-hostnames
sysutils/system-update-periodic

Available internally at https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base

Now for a specific purpose bastille jail I have the following:

poudriere bulk -j 123-release-amd64 -z ldadocgen -O ldadw-custom -f /usr/local/etc/poudriere.d/123-release-amd64-default-ldadocgen-ports -b 'https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base'

with the following in /usr/local/etc/poudriere.d/:

123-release-amd64-default-ldadocgen-make.conf
123-release-amd64-default-ldadocgen-options/
123-release-amd64-default-ldadocgen-ports

123-release-amd64-default-ldadocgen-ports:

devel/apache-ant
java/openjdk8
math/gnuplot-lite
misc/getopt
net/py-ldap
www/apache24
www/mod_auth_gssapi
www/mod_wsgi4
www/py-flask
www/py-requests
net/py-urllib3
www/tomcat85
www/tomcat-native
x11-toolkits/libXmu
x11-fonts/fontconfig
print/ghostscript9-agpl-base

123-release-amd64-default-ldadocgen-options/ contains merged options from ldadw_base and ldadocgen to satisfy -b's requirements. Many of the ports in ldadocgen will use a lot from ldadw_base, e.g., apache24, tomcat85, tomcat-native will use apr which in base has been pulled by subversion already along with many other build dependencies , e.g., ninja, cmake, GNU autotools, etc.

[michael-o]

@bdrewery As promised, here is the issue to improve -b.

[michael-o]

@bdrewery gave it some thoughts today since I don't want to live with a patch forever. Here is my idea how we could move this forward: As far as I understand only a single repo is supported since packagesite is used throughout, would be much more work to rewrite to multiple -b options or supplying an external file. Compromise: If a URL is passed, don't treat is as official, disable FreeBSD repo and add a repo custom using that supplied URL.

WDYT? I'd be willing to work on a PR if this is acceptable for you...

Lengthly edit: I wanted to figure out whether poudriere would support a multi-repo setup. Here is what it looks like now and what I would like to have ideally in the future:

graph
    subgraph Wish
    direction BT
    FreeBSD_w[FreeBSD] --> ldadw-base_w[ldadw-base] & ldadocgen_w[ldadocgen] & general_w[general]
    ldadw-base_w --> ldadocgen_w & general_w
    end
    subgraph Possible
    direction BT
    subgraph Currently Used
    direction BT
    FreeBSD_cu[FreeBSD] --> ldadw-base_cu[ldadw-base]
    ldadw-base_cu[ldadw-base] --> ldadocgen_cu[ldadocgen]
    end
    subgraph ldadw-base
    direction BT
    FreeBSD_p_lb[FreeBSD] --> ldadw-base_p_lb[ldadw-base]
    end
    subgraph ldadocgen
    direction BT
    FreeBSD_p_ldg[FreeBSD] --> ldadocgen_p_ldg[ldadocgen]
    end
    subgraph general
    direction BT
    FreeBSD_p_g[FreeBSD] --> general_p_g[general]
    end
    end

Loading

I hacked common.sh and added additionally to the default official repo for -b a custom repo in a a single conf file. Immediately

poudriere/src/share/poudriere/common.sh

Line 3831 in abc58e8

remote_pkg_ver=$(injail ${pkg_bin} rquery -U %v ${P_PKG_PKGBASE:?})

failed with freebsd/pkg#2201.
My "Wish" is not possible at all without making poudriere supporting such a multi-repo/multi-inheritance setup. My current setup is flawed as well because is is very limited. As far as I can see, I can only go with the middle setup for now, wasting cycles by rebuilding packages with custom options :-(

I think that issue could be closed, but I would like to keep it open for reference for others at least until #798 has been merged and tested by people.

[yannk]

I found this issue on my way to open my own.

I came up with the same type of fix:

        FreeBSD: {
                url: ${packagesite};
                mirror_type: NONE;
        }

I use -b https://myownbinaryrepo

Seems like an easy fix would be to detect that -b is set to an URL and disable SRV in that case

[michael-o]

I found this issue on my way to open my own.

I came up with the same type of fix:

        FreeBSD: {
                url: ${packagesite};
                mirror_type: NONE;
        }

I use -b https://myownbinaryrepo

Seems like an easy fix would be to detect that -b is set to an URL and disable SRV in that case

That was my observation as well, but does not solve my wish situation where still only one repo is supported.