SignFormer-GCN: Continuous Sign Language Translation Using Spatio-Temporal Graph Convolutional Networks

Abstract

Sign language is a complex visual language system that uses hand gestures, facial expressions, and body movements to convey meaning. It is the primary means of communication for millions of deaf and hard-of-hearing individuals worldwide. Tracking physical actions, such as hand movements and arm orientation, alongside expressive actions, including facial expressions, mouth movements, eye movements, eyebrow gestures, head movements, and body postures, using only RGB features can be limiting due to discrepancies in backgrounds and signers across different datasets. Despite this limitation, most Sign Language Translation (SLT) research relies solely on RGB features. We used keypoint features, and RGB features to capture better the pose and configuration of body parts involved in sign language actions and complement the RGB features. Similarly, most works on SLT research have used transformers, which are good at capturing broader, high-level context and focusing on the most relevant video frames. Still, the inherent graph structure associated with sign language is neglected and fails to capture low-level details. To solve this, we used a joint encoding technique using a transformer and STGCN architecture to capture the context of sign language expressions and spatial and temporal dependencies on skeleton graphs. Our method, SignFormer-GCN, achieves competitive performance in RWTH-PHOENIX-2014T, How2Sign, and BornilDB v1.0 datasets experimentally, showcasing its effectiveness in enhancing translation accuracy through different sign languages.

Context and skeleton

Motivation

Sign language translation maps a video of signing directly to a spoken-language sentence. It is hard for two reasons. Signers move at different speeds, so videos vary in length. And frames do not line up one-to-one with words, because sign and spoken languages order meaning differently.

Most translators use only RGB video and a transformer. Transformers are good at broad, high-level context, but RGB features are sensitive to backgrounds and signer appearance, and they ignore the topology of the human body. Signing is, at its core, the motion of joints. A spatio-temporal skeleton graph captures exactly that, so SignFormer-GCN reads both.

Method

Two streams, one decoder

1. Appearance streamAn I3D network embeds 16-frame clips. The embeddings get positional encodings and pass through a transformer encoder that models context across the whole video.
2. Skeleton streamMediaPipe keypoints form a graph over joints and time. Stacked STGCN blocks model joint relationships and an LSTM models how they evolve.
3. Fusion and decodingThe two encodings are fused by summation, and a transformer decoder generates the sentence directly, with no gloss supervision.

Given a video \(V = \{f_1, \dots, f_t\}\), the model learns \(P(S \mid V)\) for a sentence \(S = \{w_1, \dots, w_n\}\). The appearance stream embeds each clip and adds temporal position.

The whole model has 9.43M parameters and trains in about 2.5 minutes per epoch on a single RTX 3090.

Datasets

Results

Gloss-free translation · BLEU · higher is better

RWTH-PHOENIX-2014T (German)

SignFormer-GCN is competitive with the much larger GFSLT-VLP at about one-twelfth of the parameters, and ahead of every other gloss-free method.

How2Sign (American)

BornilDB v1.0 (Bangla)

BornilDB v1.0 is a new, low-resource Bangla benchmark with only three signers, which makes it far harder than the other two datasets.

Example translations

Bangla examples are shown with the paper's English translations.

What matters

Ablations on RWTH-PHOENIX-2014T · test BLEU-4

Citation

@article{arib2025signformer,
  title   = {{SignFormer-GCN}: Continuous Sign Language Translation Using
             Spatio-Temporal Graph Convolutional Networks},
  author  = {Arib, Safaeid Hossain and Akter, Rabeya and Rahman, Sejuti and Rahman, Shafin},
  journal = {PLOS ONE},
  volume  = {20},
  number  = {2},
  pages   = {e0316298},
  year    = {2025},
  doi     = {10.1371/journal.pone.0316298}
}