Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: surrogate models of the objectives #121

Open
sdaulton opened this issue Nov 2, 2022 · 2 comments
Open

Feature Request: surrogate models of the objectives #121

sdaulton opened this issue Nov 2, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@sdaulton
Copy link

sdaulton commented Nov 2, 2022

Hey @sgbaird!

This repo is super cool! It is great to see Ax is useful for these optimization problems.

In the interest of lightweight R&D, it would be awesome if this repo had multi-fidelity surrogate models of the objective functions. This would make it easier to develop better Bayesian optimization methods (and run multiple replications of optimization loops), without needing the custom hardware.

Would it be possible to add some multi-fidelity surrogate models of the objective functions (e.g. Random Forests) to the repo that could be downloaded and used?

Thanks!

cc @eytan @Balandat

@sgbaird
Copy link
Member

sgbaird commented Nov 2, 2022

Hi @sdaulton.

This repo is super cool! It is great to see Ax is useful for these optimization problems.
Thank you!

In the interest of lightweight R&D, it would be awesome if this repo had multi-fidelity surrogate models of the objective functions. This would make it easier to develop better Bayesian optimization methods (and run multiple replications of optimization loops), without needing the custom hardware.

I'm planning to submit a precomputed dataset as part of Olympus aspuru-guzik-group/olympus#17. Agreed about the benefit of using it without the hardware. While I intend to maintain the publicly accessible hardware for a long time, I recognize that something may come up in the future such that it's no longer accessible. The data also changes based on whether the lights in the room are on or not 💡😄.

Would it be possible to add some multi-fidelity surrogate models of the objective functions (e.g. Random Forests) to the repo that could be downloaded and used?

Great suggestion. Lmk what you think about the following in terms of recording a multi-fidelity dataset and adding surrogate models to the repo.

  • Probably just host the data as a CSV file in this repo, and make the data easy to retrieve via the Python API
  • create separate models for each of the eight discrete wavelength objectives and make it easy to compute the scalarized objectives (MAE, RMSE, and Fréchet distance) based on a random color that should be matched
  • For simplicity and compatibility, maybe just sklearn's RandomForestClassifier for the model. Probably not too critical to use something heavier-duty as long as there are enough datapoints

Open to feedback here, and I think the notebook that creates the above dataset and surrogate models will also be helpful for people to look at.

@sdaulton
Copy link
Author

sdaulton commented Nov 3, 2022

That sounds great! Random Forests make sense (perhaps on the average value from multiple measurements if noise is a concern). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants