Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits #4278

Open
smerle33 opened this issue Sep 4, 2024 · 7 comments
Open
Assignees

Comments

@smerle33
Copy link
Contributor

smerle33 commented Sep 4, 2024

  • We need to find another implementation (most probably a cron job running every 72 and uploading data to the file share using azcopy) => an infra.ci job ?

Originally posted by @dduportal in #4240 (comment)

We need a specific Docker Image with both geoipupdater and azcopy : #4278 (comment)
We will then be able to schedule a cronjob from kubernetes to update the geoip database directly in the file share.

@smerle33
Copy link
Contributor Author

smerle33 commented Sep 4, 2024

WIP Custom Docker Image:
need to perform with 2 FROM, one with official azcopy support and the official geoipupdate.

  • -> use the TTD to ensure this contract
  • -> provide a multiplatform image (arm64/intel)
  • -> local testing of the custom image as cron
    • k3d cluster create --config=.tmp/k3d.yaml
    • k3d image import customgeoipupdate -c jenkins-infra
    • kubectl apply -f .tmp/smerle-geoipupdate.yaml
    • kubectl create job test-job --from=cronjob/geoipupdate
    • kubectl logs test-job-gdl8h
  • -> create the process: update geoip in temporary folder --> azcopy in the file share
  • -> create a dedicated repository

@smerle33 smerle33 self-assigned this Sep 4, 2024
@smerle33 smerle33 added this to the infra-team-sync-2024-09-10 milestone Sep 4, 2024
@dduportal dduportal changed the title Cron a geoip updater task for mirrorbits [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits Sep 16, 2024
dduportal pushed a commit to jenkins-infra/azure that referenced this issue Sep 23, 2024
@dduportal
Copy link
Contributor

Update: with jenkins-infra/azure#844, we now have an Azure Client (App + SP + password) allowed to get temp. token to write on the file share.

Can be tested locally (inside the docker image) and will be added in SOPS (and used as kubernetes secret passed as pod env var) by the chart

@dduportal
Copy link
Contributor

dduportal commented Oct 24, 2024

Update:

@dduportal
Copy link
Contributor

* [x]  DockerHub "repository" to be created (and set up with proper permissions) - https://hub.docker.com/repository/docker/jenkinsciinfra/geoipupdate/general

@smerle33 please check that the DockerHub repository name is properly specified for the image name in the Jenkinsfile_k8s

@dduportal
Copy link
Contributor

  • Set up release-drafter (setting file + GHA files) to ensure a next draft release is generated (otherwise the infra.ci main branch build will fail)

Additional task (worth a PR @smerle33 )

@smerle33
Copy link
Contributor Author

specific repository done: https://github.com/jenkins-infra/docker-geoipupdate
release-drafter setup: https://github.com/jenkins-infra/docker-geoipupdate/blob/main/.github/workflows/release-drafter.yml
and gha updated (by damien)
first release deployed on a dedicated docker hub repository: https://hub.docker.com/r/jenkinsciinfra/geoipupdate/tags

next step: helm/helm chart to integrate to our kubernetes.

@smerle33
Copy link
Contributor Author

updating the docker image to be able to specify the fileshare (hardcoded until then) jenkins-infra/docker-geoipupdate#11
and remove the mount on the fileshare as we use azcopy for now (we may roll back if we switch to use rsync)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants