Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix:Moirai inference in foundation-time-series-arena #389

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

gorold
Copy link

@gorold gorold commented Jun 10, 2024

Thanks for evaluating Moirai. There are some issues with the way inference is being done for Moirai:

  1. Using the mean of sample forecasts instead of median. In GluonTS, metrics involving an absolute error typically use the median of sample forecasts rather than the mean.
  2. Incorrect default parameters were being used. Our paper uses context_length 1000 and patch size 32 as default hyperparameters for the Monash experiments.
  3. Extremely small batch size is being used, slowing down the model significantly. Differing batch sizes are being used across the various models - with the new default parameters, a larger batch size should be possible.

This PR fixes these issues, and below contain the updated results. The results for the fix was run on an A100-40G GPU, but the appropriate batch size can be updated when running on your hardware.

Accuracy Inference Time
Monthly Weekly Daily Hourly Monthly Weekly Daily Hourly
LagLLama 1.176 0.835 0.804 0.783 9.477 0.954 2.806 6.596
Chronos 0.960 0.709 0.652 0.735 38.581 5.081 7.908 11.662
TimesFM 0.918 0.718 0.658 0.672 0.380 0.353 0.471 0.929
TimeGPT 0.904 0.686 0.669 0.682 0.082 0.172 0.244 0.844
TimeGPT-Long 0.907 0.715 0.673 0.643 0.084 0.169 0.397 1.184
Moirai 1.221 0.884 0.792 0.798 3.115 3.103 1.790 1.601
Moirai (Fix) 0.973 0.750 0.676 0.723 0.606 0.643 0.499 0.866

We note a significant improvement in accuracy across all metrics, as well as inference time.

@CLAassistant
Copy link

CLAassistant commented Jun 10, 2024

CLA assistant check
All committers have signed the CLA.

@mergenthaler
Copy link
Member

Hi @gorold, thank you for taking the time to run the benchmarks and improve Morai. It's great to see the increase in speed and performance. You probably saw that the Chronos also opened a PR suggesting some changes.
We will take a close look at both PR's and answer soon. Meanwhile I included a note on the readme to point readers to this PR.

@AzulGarza
Copy link
Member

hey @gorold! thank you. could you please sign the CLA?

@liu-jc
Copy link

liu-jc commented Aug 28, 2024

Hi @AzulGarza and @mergenthaler ,

Thanks for taking the time to look into this PR. May we ask if there are any updates? Have you verified the performance as we provided? Are we good to merge?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants