Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Only use a GPU value for Kubernetes that is non-null and non-zero #2006

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Only use a GPU value for Kubernetes that is non-null and non-zero #2006

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8a192a0
Only use a GPU value for Kubernetes that is non-null and non-zero
dmcguire81 Aug 30, 2024
e3421a8
Fix logic to work in the general case of unordered decorators
dmcguire81 Aug 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions metaflow/plugins/kubernetes/kubernetes_decorator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,9 +271,10 @@ def step_init(self, flow, graph, step, decos, environment, flow_datastore, logge

for deco in decos:
if isinstance(deco, ResourcesDecorator):
self.attributes["gpu"] = None
for k, v in deco.attributes.items():
# If GPU count is specified, explicitly set it in self.attributes.
if k == "gpu" and v != None:
# If GPU count is specified and non-zero, explicitly set it in self.attributes.
if k == "gpu" and v:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking into alternatives for this at the moment. Note that the default from resources deco is actually "0" and not int, so it will pass this check.

Copy link
Author

@dmcguire81 dmcguire81 Oct 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I wonder if it's cast before it ends up here, because the type in the comments in int. Also, this seems to fix the problem, which is my biggest priority for removing the footgun; setting the default in the decorator to None would also expedite the fix, so that works, as well.

I can add some logging to capture the default values, here, to demonstrate.

self.attributes["gpu"] = v

if k in self.attributes:
Expand Down