ProwJob Loader

[hh]

Using APISnoopDB as general model, load https://github.com/kubernetes/test-infra/tree/master/config/jobs into a Postgres table as jsonb.

We need a json based table of prow job yaml (from the test-infra repo) with a secondary table with data pulled in from the running prow cluster.

Combined we should be able the
job-name, job-github-url, job definition data (from within the url, we'll pick as we go), job result data (from the ruling prow instance)

Ideally we can determine job:

• name
• github-url
• definition (this is mostly what we cherry pick from GitHub yaml to jsonb)
• owning sig (from owners-file for github-url)
• time to run (from prow.k8s.io search with job)
• recent success log text : #prow-k8s-io
• recent failure log text : #prow-k8s-io
• If the job creates clusters
  • (based on if logs show cluster creation, and possible other parts)
  • and IF it creates clusters, have we configured it to run Kubecost YET (Likely something we add to the labels of the job)

Docs for jobs here:

• https://docs.prow.k8s.io/docs/jobs/
  SRC Yaml files here:
• https://github.com/kubernetes/test-infra/tree/master/config/jobs
  Living Results of jobs here:
• https://prow.k8s.io

(shell) Steps for init.db

JOBS

• git clone github.com/kubernetes/test-infra
• cd config/jobs
• find . -name *.yaml | xargs -n 1 -IX cat X | yq > blob.json
• Load json into job table, keeping track of the src code folder the yaml came from

OWNERS

• for above repo
• find config/jobs \OWNERS* | > json
• Load json into owners table, keeping track of the src code folder the yaml from
• find way to connect job to owners

DECK

Get the playing cards

• load deck json (20k)

Limiting to

• including log text
• including job time
• 1 success and 1 failure