Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LGA-2574 use new irsa service account #971

Merged
merged 22 commits into from
Aug 17, 2023
Merged

LGA-2574 use new irsa service account #971

merged 22 commits into from
Aug 17, 2023

Conversation

said-moj
Copy link
Contributor

@said-moj said-moj commented Aug 2, 2023 •
edited
Loading

What does this pull request do?

  • Use new irsa service account instead of long lived acess key and secret key
  • Remove long-lived aws environment variable settings
  • Upgrade Celery to 4.0.2 to use Kombu version that uses botocore/boto3 for SQS communication
  • Install celery[sqs] instead of just celery so that dependencies required to communicate with sqs are also installed

Any other changes that would benefit highlighting?

Celery versions 4.0 and above require pycurl package and pycurl has some build dependencies that need to be installed. I added installingcurl-dev to the built docker image and libcurl4-gnutls-dev to the pipeline.

Celery 4.0 does not automatically register class based tasks which means we have to manually register these tasks. More information https://docs.celeryq.dev/en/v4.0.0/whatsnew-4.0.html#the-task-base-class-no-longer-automatically-register-tasks and celery/celery#5992

Checklist

  • Provided JIRA ticket number in the title, e.g. "LGA-152: Sample title"

@said-moj said-moj requested a review from a team as a code owner August 2, 2023 15:34
@said-moj said-moj force-pushed the feature/LGA-2574-use-new-irsa-service-account branch from b906756 to 924f4fd Compare August 3, 2023 08:21
github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 4, 2023
View reviewed changes
cla_backend/apps/reports/views.py Dismissed Show resolved Hide resolved
@said-moj said-moj force-pushed the feature/LGA-2574-use-new-irsa-service-account branch 11 times, most recently from 71807b4 to f2cb7e1 Compare August 9, 2023 15:53
@said-moj said-moj changed the title WIP - LGA-2574 use new irsa service account LGA-2574 use new irsa service account Aug 10, 2023
@said-moj said-moj changed the title LGA-2574 use new irsa service account WIP - LGA-2574 use new irsa service account Aug 10, 2023
@said-moj said-moj changed the title WIP - LGA-2574 use new irsa service account LGA-2574 use new irsa service account Aug 10, 2023
@said-moj said-moj force-pushed the feature/LGA-2574-use-new-irsa-service-account branch from ca634d0 to 21fc038 Compare August 10, 2023 11:56
@said-moj
Starting from version 4.0, Celery uses message protocol version 2 as …
93887bd 
…default value. Explicitly using old protocol
@said-moj
Remove unused sqs keys as we are using short lived tokens
7590550
@said-moj
Adding service account to worker deployment
b672347
@said-moj
Kombu versions above 4.1 require celery 4.2 otherwise you will get a …
a74b713 
…AttributeError: async when trying to run worker pods

See celery/kombu#873 for more info
@said-moj
Install celery[sqs] so that dependencies like pycurl get installed
60b6638
@said-moj
Adding steps to install ibcurl4-gnutls-dev because it is required for…
f6261c7 
… pycurl to the pipeline
@said-moj
Manually register reason for contacting export task
0bd0938
@said-moj
Set name of default export task
7e00724
@said-moj
Kombu versions above 4.1 require celery 4.2 otherwise you will get a …
6930fd9 
…AttributeError: async when trying to run worker pods

See celery/kombu#873 for more info
@said-moj
Clean s3 file name and remove trailing slash
29883e6
@said-moj said-moj force-pushed the feature/LGA-2574-use-new-irsa-service-account branch from 21fc038 to 662d385 Compare August 16, 2023 10:03
@said-moj said-moj force-pushed the feature/LGA-2574-use-new-irsa-service-account branch from 0f311be to 1e2cfa9 Compare August 16, 2023 10:49
@sonarcloud
Copy link

sonarcloud bot commented Aug 16, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 1 Security Hotspot
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

jogilder
jogilder approved these changes Aug 16, 2023
View reviewed changes
Copy link
Contributor

@jogilder jogilder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

psweeting1
psweeting1 approved these changes Aug 16, 2023
View reviewed changes
Copy link
Contributor

@psweeting1 psweeting1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@said-moj said-moj merged commit 60190c2 into master Aug 17, 2023
3 of 4 checks passed
@said-moj said-moj deleted the feature/LGA-2574-use-new-irsa-service-account branch August 17, 2023 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@psweeting1 psweeting1 psweeting1 approved these changes

@jogilder jogilder jogilder approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@said-moj @jogilder @psweeting1