Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series and Spatio-Temporal Data

A professionally curated list of Large (Language) Models and Foundation Models (LLM, LM, FM) for Temporal Data (Time Series, Spatio-temporal, and Event Data) with awesome resources (paper, code, data, etc.), which aims to comprehensively and systematically summarize the recent advances to the best of our knowledge.

We will continue to update this list with the newest resources. If you find any missed resources (paper/code) or errors, please feel free to open an issue or make a pull request.

For general AI for Time Series (AI4TS) Papers, Tutorials, and Surveys at the Top AI Conferences and Journals, please check This Repo.

Survey paper

Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook

Authors: Ming Jin, Qingsong Wen*, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, Shirui Pan*, Vincent S. Tseng (IEEE Fellow), Yu Zheng (IEEE Fellow), Lei Chen (IEEE Fellow), Hui Xiong (IEEE Fellow)

🌟 If you find this resource helpful, please consider to star this repository and cite our survey paper:

@article{jin2023lm4ts,
  title={Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook}, 
  author={Ming Jin and Qingsong Wen and Yuxuan Liang and Chaoli Zhang and Siqiao Xue and Xue Wang and James Zhang and Yi Wang and Haifeng Chen and Xiaoli Li and Shirui Pan and Vincent S. Tseng and Yu Zheng and Lei Chen and Hui Xiong},
  journal={arXiv preprint arXiv:2310.10196},
  year={2023}
}

LLMs for Time Series

General Time Series Analysis

• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, in arXiv 2023, [paper]
• TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting, in arXiv 2023, [paper]
• TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series, in arXiv 2023, [paper]
• LLM4TS: Two-Stage Fine-Tuning for Time-Series Forecasting with Pre-Trained LLMs, in arXiv 2023, [paper]
• The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models, in arXiv 2023, [paper]
• PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting, in arXiv 2022. [paper]
• One Fits All: Power General Time Series Analysis by Pretrained LM, in NeurIPS 2023, [paper] [official code]
• Large Language Models Are Zero-Shot Time Series Forecasters, in NeurIPS 2023, [paper] [official code]

Transportation Application

• Leveraging Language Foundation Models for Human Mobility Forecasting, in SIGSPATIAL 2022, [paper]

Finance Application

• Temporal Data Meets LLM -- Explainable Financial Time Series Forecasting, in arXiv 2023, [paper]
• BloombergGPT: A Large Language Model for Finance, in arXiv 2023, [paper]
• WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine, in arXiv 2023, [paper][official-code]
• Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models, in arXiv 2023, [paper]
• Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models, in arXiv 2023, [paper]
• The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges, in arXiv 2023, [paper]

Healthcare Application

• Large Language Models are Few-Shot Health Learners, in arXiv 2023. [paper]
• Health system-scale language models are all-purpose prediction engines, in Nature 2023, [paper]
• A large language model for electronic health records, in NPJ Digit. Med. 2022, [paper]

Event Analysis

• Drafting Event Schemas using Language Models, in arXiv 2023, [paper]
• Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning, in NeurIPS 2023, [paper], [official-code]

PFMs for Time Series

General Time Series Analysis

• SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling, in NeurIPS 2023, [paper]
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, in ICLR 2023, [paper] [official code]
• Contrastive Learning for Unsupervised Domain Adaptation of Time Series, in ICLR 2023, [paper]
• TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting, in KDD 2023, [paper]
• Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency, in NeurIPS 2022, [paper] [official code]
• Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting, in KDD 2022, [paper]
• TS2Vec: Towards Universal Representation of Time Series, in AAAI 2022, [paper] [official code]
• Voice2Series: Reprogramming Acoustic Models for Time Series Classification, in ICML 2021. [paper] [official code]

Event Analysis

• Prompt-augmented Temporal Point Process for Streaming Event Sequence, in NeurIPS 2023, [paper] [official code]

LLMs for Spatio-Temporal Graphs

• Language knowledge-Assisted Representation Learning for Skeleton-based Action Recognition, in arXiv 2023. [paper]
• Chatgpt-Informed Graph Neural Network for Stock Movement Prediction, in arXiv 2023. [paper]

PFMs for Spatio-Temporal Graphs

General Purposes

• When do Contrastive Learning Signals help Spatio-Temporal Graph Forecasting? in SIGSPATIAL 2023. [paper]
• Mining Spatio-Temporal Relations via Self-Paced Graph Contrastive Learning, in KDD 2022. [paper]

Climate

• Accurate Medium-Range Global Weather Forecasting with 3D Neural Networks, in Nature 2023. [paper]
• ClimaX: A Foundation Model for Weather and Climate, in ICML 2023. [paper] [official code]
• GraphCast: Learning Skillful Medium-Range Global Weather Forecasting, in arXiv 2022. [paper]
• FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operator, in arXiv 2022. [paper]
• Accurate Medium-Range Global Weather Forecasting with 3d Neural Networks, in Nature 2023. [paper]
• W-MAE: Pre-Trained Weather Model with Masked Autoencoder for Multi-Variable Weather Forecasting, in arXiv 2023. [paper]
• FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead, in arXiv 2023. [paper]

Transportation

• Pre-trained Bidirectional Temporal Representation for Crowd Flows Prediction in Regular Region, in IEEE Access 2019. [paper]
• Trafficbert: Pretrained Model with Large-Scale Data for Long-Range Traffic Flow Forecasting, in Expert Systems with Applications 2021. [paper]
• Building Transportation Foundation Model via Generative Graph Transformer, in arXiv 2023. [paper]

LLMs for Video Data

• Zero-shot Video Question Answering via Frozen Bidirectional Language Models, in NeurIPS 2022. [paper]
• Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners, in NeurIPS 2022. [paper]
• VideoLLM: Modeling Video Sequence with Large Language Models, in arXiv 2023. [paper]
• VALLEY: Viode Assitant with Large Language Model Enhanced Ability, in arXiv 2023. [paper]
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, in CVPR 2023. [paper]
• Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models, in NeurIPS 2022. [paper]
• VideoChat: Chat-Centric Video Understanding, in arXiv 2023. [paper]
• MovieChat: From Dense Token to Sparse Memory for Long Video Understanding, in arXiv 2023. [paper]
• Language Models are Causal Knowledge Extractors for Zero-shot Video Question Answering, in CVPR 2023. [paper]
• Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding, in arXiv 2023. [paper]
• Learning Video Representations from Large Language Models, in CVPR 2023. [paper]
• Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models, in arXiv 2023. [paper]
• Traffic-Domain Video Question Answering with Automatic Captioning, in arXiv 2023. [paper]
• LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling, in CVPR 2023. [paper]

PFMs for Video Data

• OmniVL: One Foundation Model for Image-Language and Video-Language Tasks, in NeurIPS 2022. [paper]
• Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks, in arXiv 2023. [paper]
• PAXION: Patching Action Knowledge in Video-Language Foundation Models, in arXiv 2023. [paper]
• mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video, in ICML 2023. [paper]

Datasets

Traffic Application

• METR-LA [link]
• PEMS-BAY [link]
• PEMS04 [link]
• SUTD-TrafficQA [link]
• TaxiBJ \link]
• BikeNYC [link]
• TaxiNYC [link]
• Mobility [link]
• LargeST [link]

Healthcare Application

• PTB-XL [link]
• NYUTron [link]
• UF Health clinical corpus [link]
• i2b2-2012 [link]
• MIMIC-III [link]
• CirCor DigiScope [link]

Weather Application

• SEVIR [link]
• Shifts-Weather Prediction [link]
• AvePRE [link]
• SurTEMP [link]
• SurUPS [link]
• ERA5 reanalysis data [link]
• CMIP6 [link]

Finance Application

• Finance (Employment) [link]
• StockNet [link]
• EDT [link]
• NASDAQ-100 [link]

Video Application

• TGIF-QA [link]
• MSR-VTT [link]
• WebVid [link]
• MSVD [link]
• DiDeMo [link]
• COCO [link]

Event Analysis Application

• Amazon [link]
• Taobao [link]
• Retweet [link]
• StackOverflow [link]
• Taxi [link]

Other General Application

• ETT [link]
• M4 [link]
• Electricity [link]
• Alibaba Cluster Trace [link]
• TSSB [link]
• UCR TS Classification Archive [link]

Related LLM/LM/FM Resources

Survey

• A Survey of Large Language Models, in arXiv 2023. [paper] [link]
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond, in arXiv 2023. [paper] [link]
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models, in arXiv 2023. [paper] [link]
• Beyond One-Model-Fits-All: A Survey of Domain Specialization for Large Language Models, in arXiv 2023. [paper]
• Large AI Models in Health Informatics: Applications, Challenges, and the Future, in arXiv 2023. [paper] [link]
• FinGPT: Open-Source Financial Large Language Models, in arXiv 2023. [paper] [link]
• On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence, in arXiv 2023. [paper]

Github

• Awesome LLM. [link]
• Open LLMs. [link]
• Awesome LLMOps. [link]
• Awesome Foundation Models. [link]
• Awesome Graph LLM. [link]
• Awesome Multimodal Large Language Models. [link]

Related Resources

Surveys of Time Series

• Transformers in Time Series: A Survey, in IJCAI 2023. [paper] [GitHub Repo]
• Time series data augmentation for deep learning: a survey, in IJCAI 2021. [paper]
• Time-series forecasting with deep learning: a survey, in Philosophical Transactions of the Royal Society A 2021. [paper]
• A review on outlier/anomaly detection in time series data, in CSUR 2021. [paper]
• Deep learning for time series classification: a review, in Data Mining and Knowledge Discovery 2019. [paper]
• A Survey on Time-Series Pre-Trained Models, in arXiv 2023. [paper] [link]
• Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects, in arXiv 2023. [paper] [Website]
• A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection, in arXiv 2023. [paper] [Website]

Surveys of AIOps

• AIOps: real-world challenges and research innovations, in ICSE 2019. [paper]
• A Survey of AIOps Methods for Failure Management, in TIST 2021. [paper]
• AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges, in arXiv 2023. [paper]

LLM/LM/FM Papers for AIOps

• Empowering Practical Root Cause Analysis by Large Language Models for Cloud Incidents, in arXiv 2023. [paper]
• Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models, in arXiv 2023. [paper]
• OpsEval: A Comprehensive Task-Oriented AIOps Benchmark for Large Language Models, in arXiv 2023. [paper]

Citation

If you find this repository helpful for your work, please kindly cite:

@article{jin2023lm4ts,
  title={Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook}, 
  author={Ming Jin and Qingsong Wen and Yuxuan Liang and Chaoli Zhang and Siqiao Xue and Xue Wang and James Zhang and Yi Wang and Haifeng Chen and Xiaoli Li and Shirui Pan and Vincent S. Tseng and Yu Zheng and Lei Chen and Hui Xiong},
  journal={arXiv preprint arXiv:2310.10196},
  year={2023}
}