Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sampling hyperparameters are not universal among models #42

Open
domenic opened this issue Sep 18, 2024 · 1 comment
Open

Sampling hyperparameters are not universal among models #42

domenic opened this issue Sep 18, 2024 · 1 comment
Labels
interop Potential concerns about interoperability among multiple implementations of the API

Comments

@domenic
Copy link
Collaborator

domenic commented Sep 18, 2024

The explainer currently assumes that a model is best controlled by setting its temperature and top-K sampling hyperparameters.

However, these aren't universal among all models. And, various other models expose more. Others to consider are top-P, max tokens (#36), repetition penalty, presence penalty, frequency penalty, and more.

This poses a challenge for creating an interoperable API for which each browser can bring their own model.

One path here is to pick a set and require that every implementation allow control over such hyperparameters. (Possibly including no real control, e.g. a frequency penalty with max = min = 1.)

It would be especially helpful if others interested in implementing the prompt API were able to chime in with their implementation constraints.

@tomayac
Copy link
Contributor

tomayac commented Sep 18, 2024

+1. It's worthwhile to not that also not all models allow for top-k to be specified. For example, Open AI's Chat API only lets you modify top-p (they call it top_p), but not top-k.

Similar to #41, should new parameters get added to the interface, we'd want to make sure that the (if applicable) min-* and max-* values are queryable, since there are model differences.

@domenic domenic added the interop Potential concerns about interoperability among multiple implementations of the API label Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interop Potential concerns about interoperability among multiple implementations of the API
Projects
None yet
Development

No branches or pull requests

2 participants