T3Time: Tri-Modal Time Series Forecasting via Adaptive Multi-Head Alignment and Residual Fusion

Abstract

Multivariate time series forecasting (MTSF) seeks to model temporal dynamics among variables to predict future trends. Transformer-based models and large language models (LLMs) have shown promise due to their ability to capture long-range dependencies and patterns. However, current methods often rely on rigid inductive biases, ignore inter-variable interactions, or apply static fusion strategies that limit adaptability across forecast horizons. These limitations create bottlenecks in capturing nuanced, horizon-specific relationships in time-series data. To solve this problem, we propose T3Time, a novel trimodal framework consisting of time, spectral, and prompt branches, where the dedicated frequency encoding branch captures the periodic structures along with a gating mechanism that learns prioritization between temporal and spectral features based on the prediction horizon. We also proposed a mechanism which adaptively aggregates multiple cross-modal alignment heads by dynamically weighting the importance of each head based on the features. Extensive experiments on benchmark datasets demonstrate that our model consistently outperforms state-of-the-art baselines, achieving an average reduction of 3.28% in MSE and 2.29% in MAE. Furthermore, it shows strong generalization in few-shot learning settings: with 5% training data, we see a reduction in MSE and MAE by 4.13% and 1.91%, respectively; and with 10% data, by 3.62% and 1.98% on average.

Three ways to read one series

Motivation

Forecasting models usually commit to one view of the data. Time-domain Transformers such as PatchTST and iTransformer model local dynamics. Frequency models such as FEDformer capture periodicity. LLM-based models such as Time-LLM and TimeCMA describe the series in a text prompt and borrow a language model's priors. Each view sees something the others miss.

• Isolated modalities. Prompt-based models pair time with language but ignore the spectral view that summarizes global periodicity.
• Narrow alignment. A single cross-attention head limits how modalities interact.
• Horizon rigidity. The same fusion is used for every forecast length, although short horizons lean on local detail and long horizons on periodic structure.

Method

Encode three views, gate by horizon, align, keep a residual path

1. Tri-modal encoding

Frequency. The input is transformed with a real FFT and each magnitude bin becomes a token. After a one-layer Transformer, an attention-weighted pooling summarizes the spectrum per variable.

Time. Each variable's window is linearly embedded and encoded by a Transformer.

Language. Each variable gets a prompt such as "From [t1] to [t2], the values were [v1, …, vn] every [hour]. The total trend value was [T]." A frozen GPT-2 encodes it, and the last-token embedding is kept.

2. Horizon-aware gating

A small MLP reads the pooled time encoding together with the normalized prediction length and outputs per-channel gates. Each forecast horizon gets its own mix of spectral and temporal evidence.

3. Adaptive multi-head cross-modal alignment

Instead of one cross-attention head, \(H\) independent heads align the gated series features (queries) with the prompt embeddings (keys and values). A gating network then scores the heads separately for every variable, so different variables can rely on different alignments.

4. Channel-wise residual fusion

Learned per-channel coefficients decide how much each latent channel takes from the aligned features versus the raw temporal–spectral path, before a Transformer decoder and a linear forecasting head.

Results

Long-term forecasting · averaged over four horizons · lower is better

Following TimeCMA, inputs are 96 steps and horizons are {96, 192, 336, 720} (36 steps and {24, 36, 48, 60} for ILI), averaged over three seeds.

MSE

MAE

Few-shot forecasting

With only a tenth or a twentieth of the training data, T3Time beats the strongest baseline on all five datasets.

MSE, lower is better. Average reductions are 3.62% (10%) and 4.13% (5%).

What matters

Ablations · MSE

• The residual path matters most. Without it, MSE rises on every dataset, from 1.705 to 2.176 on ILI.
• The spectral view pays off on periodic data, such as Exchange (0.353 → 0.374) and ILI (1.705 → 1.786).
• Multi-head alignment and horizon gating each give consistent smaller gains.

What the three views learn

Limitations

Citation

@inproceedings{chowdhury2026t3time,
  title     = {{T3Time}: Tri-Modal Time Series Forecasting via Adaptive
               Multi-Head Alignment and Residual Fusion},
  author    = {Chowdhury, Abdul Monaf and Akter, Rabeya and Arib, Safaeid Hossain},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {40},
  number    = {25},
  pages     = {20597--20605},
  year      = {2026},
  doi       = {10.1609/aaai.v40i25.39196}
}