Abstract

Scaling laws for large language models (LLMs) offer valuable insights into how increasing model size and training data leads to predictable performance improvements. Time series forecasting, which shares a sequential structure similar to language, is well-suited to large-scale transformer architectures. In this talk, I will demonstrate that foundational decoder-only time series transformer models exhibit scaling behaviour analogous to LLMs. I will begin with a general introduction to scaling laws and how they can inform efficient, optimised model training. I will then focus on their specific application to time series data, highlighting the emergence of power law behaviour. Finally, I will discuss the broader implications of these findings, and potential scientific applications.

Related papers: https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2405.13867