Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a master notebook for evaluating performances #78

Open
Laurits7 opened this issue Oct 11, 2024 · 0 comments
Open

Create a master notebook for evaluating performances #78

Laurits7 opened this issue Oct 11, 2024 · 0 comments
Assignees

Comments

@Laurits7
Copy link
Member

Currently various plotting-evaluation scripts and notebooks are scattered around the repository. Create a master notebook that calculates all the results from the trainings of the three training scenarios - dm_multiclass, jet_regression and binary_classification.
There is an evaluator written for all of these training scenarios under enreg/tools/ - decay_mode_evaluator.py, regression_evaluator.py and tagger_evaluator.py.
An example of their use can be found in notebooks/paper_performance_plotting.ipynb

The master notebook should be executable by papermill. An example of its usage can be found using a simple google search.

Ideally dm_evaluator would need also the capability to combine different evaluators in order to make direct comparisons. The direct comparison would be Fig.6 from our most recent paper. Fig.6 is produced currently with the notebooks/DM_CM.ipynb notebook.

As a next step, it would be preferred if also the metrics tracked during training could be plotted within this master script for each training scenario - currently the comparisons are done with notebooks/foundation_model_eval.ipynb and notebooks/losses.ipynb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants