The standard set of conformance tests is currently those defined by the
[Conformance] tag in the
kubernetes e2e
suite.
The standard tool for running these tests is Sonobuoy. Sonobuoy is regularly built and kept up to date to execute against all currently supported versions of kubernetes.
Download a binary release of the CLI, or build it yourself by running:
$ go get -u -v github.com/vmware-tanzu/sonobuoy
Deploy a Sonobuoy pod to your cluster with:
$ sonobuoy run --mode=certified-conformance
NOTE: The --mode=certified-conformance flag is required for certification runs since Kubernetes v1.16 (and Sonobuoy v0.16). Without this flag, tests which may be disruptive to your other workloads may be skipped. A valid certification run may not skip any conformance tests. If you're setting the test focus/skip values manually, certification runs require E2E_FOCUS=\[Conformance\] and no value for E2E_SKIP.
NOTE: You can run the command synchronously by adding the flag --wait but be aware that running the conformance tests can take an hour or more.
View actively running pods:
$ sonobuoy status
To inspect the logs:
$ sonobuoy logs
Once sonobuoy status shows the run as completed, copy the output directory from the main Sonobuoy pod to a local directory:
$ outfile=$(sonobuoy retrieve)
This copies a single .tar.gz snapshot from the Sonobuoy pod into your local
. directory. Extract the contents into ./results with:
mkdir ./results; tar xzf $outfile -C ./results
NOTE: The two files required for submission are located in the tarball under plugins/e2e/results/global/{e2e.log,junit_01.xml}.
To clean up Kubernetes objects created by Sonobuoy, run:
sonobuoy delete
Prepare a PR to
https://github.com/cncf/k8s-conformance.
Here are directions to
prepare a pull request from a fork.
In the descriptions below, X.Y refers to the kubernetes major and minor
version, and $dir is a short subdirectory name to hold the results for your
product. Examples would be gke or openshift.
Description: Conformance results for vX.Y/$dir
For simplicity you can submit the tarball or extract the relevant information from the tarball to compose your submission.
If submitting test results for multiple versions, submit a PR for each product, ie. one PR for vX.Y results and a second PR for vX.Z
vX.Y/$dir/README.md: A script or human-readable description of how to reproduce
your results.
vX.Y/$dir/sonobuoy.tar.gz: Raw output from sonobuoy. (optional)
vX.Y/$dir/e2e.log: Test log output (from Sonobuoy).
vX.Y/$dir/junit_01.xml: Machine-readable test log (from Sonobuoy).
vX.Y/$dir/PRODUCT.yaml: See below.
This file describes your product. It is YAML formatted with the following root-level fields. Please fill in as appropriate.
| Field | Description |
|---|---|
vendor |
Name of the legal entity that is certifying. This entity must have a signed participation form on file with the CNCF |
name |
Name of the product being certified. |
version |
The version of the product being certified (not the version of Kubernetes it runs). |
website_url |
URL to the product information website |
repo_url |
If your product is open source, this field is necessary to point to the primary GitHub repo containing the source. It's OK if this is a mirror. OPTIONAL |
documentation_url |
URL to the product documentation |
product_logo_url |
URL to the product's logo, (must be in SVG, AI or EPS format -- not a PNG -- and include the product name). OPTIONAL. If not supplied, we'll use your company logo. Please see logo guidelines |
type |
Is your product a distribution, hosted platform, or installer (see definitions) |
description |
One sentence description of your offering |
Examples below are for a fictional Kubernetes implementation called Turbo Encabulator produced by a company named Yoyodyne.
vendor: Yoyodyne
name: Turbo Encabulator
version: v1.7.4
website_url: https://yoyo.dyne/turbo-encabulator
repo_url: https://github.com/yoyo.dyne/turbo-encabulator
documentation_url: https://yoyo.dyne/turbo-encabulator/docs
product_logo_url: https://yoyo.dyne/assets/turbo-encabulator.svg
type: distribution
description: 'The Yoyodyne Turbo Encabulator is a superb Kubernetes distribution for all of your Encabulating needs.'If you need a private review for an unreleased product, please email a zip file containing what you would otherwise submit as a pull request to [email protected]. We'll review and confirm that you are ready to be Certified Kubernetes as soon as you open the pull request. We can then often arrange to accept your pull request soon after you make it, at which point you become Certified Kubernetes.
A reviewer will shortly comment on and/or accept your pull request, following this process. If you don't see a response within 3 business days, please contact [email protected].
Combining the steps provided here, the process looks like this:
$ k8s_version=vX.Y
$ prod_name=example
$ go get -u -v github.com/vmware-tanzu/sonobuoy
$ sonobuoy run --mode=certified-conformance --wait
$ outfile=$(sonobuoy retrieve)
$ mkdir ./results; tar xzf $outfile -C ./results
$ mkdir -p ./${k8s_version}/${prod_name}
$ cp ./results/plugins/e2e/results/global/* ./${k8s_version}/${prod_name}/
$ cat << EOF > ./${k8s_version}/${prod_name}/PRODUCT.yaml
vendor: Yoyodyne
name: Turbo Encabulator
version: v1.7.4
website_url: https://yoyo.dyne/turbo-encabulator
repo_url: https://github.com/yoyo.dyne/turbo-encabulator
documentation_url: https://yoyo.dyne/turbo-encabulator/docs
product_logo_url: https://yoyo.dyne/assets/turbo-encabulator.svg
type: distribution
description: 'The Yoyodyne Turbo Encabulator is a superb Kubernetes distribution for all of your Encabulating needs.'
EOF
If you have problems certifying that you feel are an issue with the conformance program itself (and not just your own implementation), you can file an issue in the repository. Questions and comments can also be sent to the working group's mailing list and slack channel. SIG Architecture is the change controller of the conformance definition. We track a list of issue resolutions with SIG Architecture.