Predicting with Time-LLM using GPT2 has awful performance

[odinhorizontal]

I follow the script in https://towardsdatascience.com/time-llm-reprogram-an-llm-for-time-series-forecasting-e2558087b8ac to Predicting with Time-LLM using GPT2.
My code is exactly the same as the script. But got the terrible result.

My result:

script:
https://towardsdatascience.com/time-llm-reprogram-an-llm-for-time-series-forecasting-e2558087b8ac

code:

import time
import numpy as np
import pandas as pd
import pytorch_lightning as pl
import matplotlib.pyplot as plt

from neuralforecast import NeuralForecast
from neuralforecast.models import TimeLLM
from neuralforecast.losses.pytorch import MAE
from neuralforecast.tsdataset import TimeSeriesDataset
from neuralforecast.utils import AirPassengers, AirPassengersPanel, AirPassengersStatic, augment_calendar_df

from transformers import GPT2Config, GPT2Model, GPT2Tokenizer

AirPassengersPanel, calendar_cols = augment_calendar_df(df=AirPassengersPanel, freq='M')

Y_train_df = AirPassengersPanel[AirPassengersPanel.ds<AirPassengersPanel['ds'].values[-12]]
Y_test_df = AirPassengersPanel[AirPassengersPanel.ds>=AirPassengersPanel['ds'].values[-12]].reset_index(drop=True)

gpt2_config = GPT2Config.from_pretrained('openai-community/gpt2')
gpt2 = GPT2Model.from_pretrained('openai-community/gpt2',config=gpt2_config)
gpt2_tokenizer = GPT2Tokenizer.from_pretrained('openai-community/gpt2')

prompt_prefix = "The dataset contains data on monthly air passengers. There is a yearly seasonality"

timellm = TimeLLM(h=12,
                 input_size=36,
                 llm=gpt2,
                 llm_config=gpt2_config,
                 llm_tokenizer=gpt2_tokenizer,
                 prompt_prefix=prompt_prefix,
                 max_steps=100,
                 batch_size=24,
                 windows_batch_size=24)

nf = NeuralForecast(
    models=[timellm],
    freq='M'
)

nf.fit(df=Y_train_df, val_size=12)
forecasts = nf.predict(futr_df=Y_test_df)

[odinhorizontal]

Can somebody tell me why my code has such awful outcome? qaq

[odinhorizontal]

Really thank you for your reply! I also feel that this project is indeed difficult to replicate the results reported in the paper. I'm wondering if it's because the authors haven't disclosed some of their tricks or techniques. But my issue #161 is an easy work，I 've got such poor result. It doesn't feel like it should be this way. Have you tried the performance of timellm on simple tasks? How did it perform?

…

---- Replied Message ---- | From | ***@***.***> | | Date | 11/27/2024 11:08 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [KimMeen/Time-LLM] Predicting with Time-LLM using GPT2 has awful performance (Issue #161) | #109 related issue. It is observed that their results are worse than reported in the paper. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>