Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modify how losses are computed in a multi-replicas hyperopt #2145

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Modify how losses are computed in a multi-replicas hyperopt #2145

wants to merge 6 commits into from

Conversation

Radonirinaunimi
Copy link
Member

@Radonirinaunimi Radonirinaunimi commented Aug 16, 2024 •
edited
Loading

Addresses the following:

  • include the Positivity filter in the hyperopt card
  • select a fraction of the replicas to compute the hyperopt $\chi^2$ losses
  • fix how $\varphi^2$ of the hold out folds are computed during hyperopt (this line)
  • select a fraction of the replica models to evaluate $\varphi^2$

@Radonirinaunimi
Copy link
Member Author

Radonirinaunimi commented Aug 16, 2024 •
edited
Loading

@Cmurilochem, are you actually including penalties in losses?

nnpdf/n3fit/runcards/hyperopt_studies/restricted_search_space_renew_hyperopt.yml

Line 131 in e37aa4e

penalties_in_loss: True

At some point in then draft you mentioned that this is not the case.

@goord
Copy link
Collaborator

goord commented Aug 18, 2024

Hi @Radonirinaunimi I'm afraid most of our GPU budget is burnt and I'm not sure the folks at SURF are willing to give us more again...

About the penalties in the hyperloss: I'm pretty sure they are excluded. Also, I received an invite for the slack channel but it won't allow me in!

@Radonirinaunimi
Copy link
Member Author

Radonirinaunimi commented Aug 18, 2024 •
edited
Loading

Hi @Radonirinaunimi I'm afraid most of our GPU budget is burnt and I'm not sure the folks at SURF are willing to give us more again...

Hi @goord, I hope it was not our runs that burnt the GPUs 😬

For the paper, I think it would be realistic to only perform 250 trials. This is because from the PDF point of view we are doing a proof of concept, and we use restricted parameter space anyway. Do you think this would not even be possible? At some point, @Cmurilochem was planning to run 3 more sets of 250.

About the penalties in the hyperloss: I'm pretty sure they are excluded.

Ok, good! As it should be. I will modify the card here.

Also, I received an invite for the slack channel but it won't allow me in!

Hmhm, what is the message that you received? Maybe @juanrojochacon knows how to solve this?

@Radonirinaunimi Radonirinaunimi changed the title Filter positivity datapoints Filter positivity datapoints in hyperopt Aug 18, 2024
@goord
Copy link
Collaborator

goord commented Aug 18, 2024

Well we can at least run a 5-day job, there is enough budget for that. In the meantime we can explore our options for more compute (Leonardo or new pilot project on Snellius?). @Cmurilochem maybe you can find the time to start a job?

Regarding the slack: tried again and now it works

@juanrojochacon
Copy link

yes @goord @Cmurilochem Slack has been a mess in the last few weeks but it is sorted out now, back to our Pro plan so all communication can proceed via there as usual now. Thanks!

@Cmurilochem
Copy link
Collaborator

Hi @Radonirinaunimi I'm afraid most of our GPU budget is burnt and I'm not sure the folks at SURF are willing to give us more again...

Hi @goord, I hope it was not our runs that burnt the GPUs 😬

For the paper, I think it would be realistic to only perform 250 trials. This is because from the PDF point of view we are doing a proof of concept, and we use restricted parameter space anyway. Do you think this would not even be possible? At some point, @Cmurilochem was planning to run 3 more sets of 250.

About the penalties in the hyperloss: I'm pretty sure they are excluded.

Ok, good! As it should be. I will modify the card here.

Also, I received an invite for the slack channel but it won't allow me in!

Hmhm, what is the message that you received? Maybe @juanrojochacon knows how to solve this?

Hi @Radonirinaunimi and @goord. Yes. I excluded penalties in all runs. So, if this is the problem, nothing to be worried about.

Also, @goord is right. We have a limited budget and I suspect that we left a 3-4 days job. Tomorrow I am back home and will submit it again. But we currently have more than 250 trials for sure.

@Radonirinaunimi
Copy link
Member Author

Also, @goord is right. We have a limited budget and I suspect that we left a 3-4 days job. Tomorrow I am back home and will submit it again.

Perfect, thanks! Please use this branch for the runs.

But we currently have more than 250 trials for sure.

I fear that we cannot use those, unfortunately. But we should always make backups of them, just in case.

@Cmurilochem
Copy link
Collaborator

Hi @Radonirinaunimi and @goord. Just submitted the new hyperopt from this branch; we currently have just ~3.5 days of budget. I am currently on holidays with family, but will find some time to give you feedback on the progress of the calculation.

@Radonirinaunimi
Copy link
Member Author

Unrelated: it looks like the polarized theories have been updated (cc, @giacomomagni, @scarlehoff)? Now the C-factors are no longer present. This is the reason why the tests are failing.

@giacomomagni
Copy link
Contributor

giacomomagni commented Sep 13, 2024 •
edited
Loading

Unrelated: it looks like the polarized theories have been updated (cc, @giacomomagni, @scarlehoff)? Now the C-factors are no longer present. This is the reason why the tests are failing.

maybe I've forgot them during my last update... I'll check it

EDIT: Something went wrong when removing the eko.tar, now it should be okay.

giacomomagni
giacomomagni reviewed Sep 24, 2024
View reviewed changes
n3fit/src/n3fit/hyper_optimization/rewards.py
@@ -44,7 +44,7 @@
log = logging.getLogger(__name__)


def _average_best(fold_losses: np.ndarray, proportion: float = 0.9, axis: int = 0) -> float:
def _average_best(fold_losses: np.ndarray, proportion: float = 0.05, axis: int = 0) -> float:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure we do want to have a default value for proportion ?

n3fit/src/n3fit/hyper_optimization/rewards.py
# If a proportion allow as a keyword argument, use 80% and 10%
# as a proxy of
# "80% of the replicas should be good, but only a small % has to cover the folds"
# The values of 80% and 10% are completely empirical and should be investigated further
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on a similar line maybe we can pass the values from the runcard?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh, it would be a good idea. It was added quickly there for the sake of the meeting, but it would be good to have it as an input parameter.

@Radonirinaunimi Radonirinaunimi changed the title Filter positivity datapoints in hyperopt Modify how losses are computed in a multi-replicas hyperopt Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@giacomomagni giacomomagni giacomomagni left review comments

@scarlehoff scarlehoff scarlehoff left review comments

@goord goord Awaiting requested review from goord

@RoyStegeman RoyStegeman Awaiting requested review from RoyStegeman

@Cmurilochem Cmurilochem Awaiting requested review from Cmurilochem

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
escience hyperoptimization
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Radonirinaunimi @goord @juanrojochacon @Cmurilochem @giacomomagni @scarlehoff