01 · Perceive
Multimodal Learning
I combine vision, language, pose and time series so models see what a single view misses, and test whether that understanding holds in low-resource languages like Bangla.
Rabeya Akter
I am a Research Assistant at North South University, working with Md Salman Shamil and Dr. Md Adnan Arefeen on egocentric video understanding (EK-Bangla).
My latest work, OPASD, teaches reasoning language models where to look. This work is supervised by Dr. Md Mofijul Islam (Amazon GenAI, USA) and Dr. Tasnim Mohiuddin (Qatar Computing Research Institute, Qatar). I am also working with Dr. Ser-Nam Lim (University of Central Florida, USA) on world models.
I earned my B.Sc. in Robotics and Mechatronics Engineering from the University of Dhaka, where I worked on amodal counting with Dr. Md Mehedi Hasan and on sign language translation with Dr. Shafin Rahman and Dr. Sejuti Rahman.
Outside of research, I love good books and well-written blogs, and I am always happy to chat about either, so please feel free to reach out.
I am actively seeking PhD opportunities for Fall 2027.
Research interests
- Language Model Post-Training
- Reinforcement Learning
- Multimodal Learning
- Vision-Language Models
- Embodied AI
- Computer Vision
Research
Most agents learn from outcomes, which tell them whether they succeeded but not why. My work gives an agent richer signals across the three abilities it needs. To perceive, it fuses modalities so that one view fills in what another misses. To reason, it learns where to look, not just which answer is right. To act, it learns why it failed, as a vision–language model turns its mistakes into rewards.
News
-
New Our new preprint, Teach Yourself Where to Look, is now on arXiv.
-
New Introducing EK-Bangla, my first-author benchmark for fine-grained egocentric hand-object interaction in Bangla. The project page is live.
-
Our paper LAGEA: Language Guided Embodied Agents for Robotic Manipulation has been accepted at ICML 2026.
-
Joined the Dept. of Electrical and Computer Engineering, North South University, as a Research Assistant, working on egocentric video understanding.
-
Our work T3Time has been accepted at the AAAI 2026 Main Technical Track.
-
SignFormer-GCN has been accepted at the WiML Workshop at NeurIPS 2025.
-
Started as a Research Assistant at the Dept. of Robotics and Mechatronics Engineering, University of Dhaka, working on amodal counting.
-
SignFormer-GCN is officially published in PLOS ONE.
-
Joined Therap (BD) Ltd. as an Associate Software Engineer on the Machine Learning team.
-
Graduated from the Department of Robotics and Mechatronics Engineering, University of Dhaka.
-
Successfully defended my undergraduate thesis — a huge milestone in my academic journey.
-
Presented our poster Classical machine learning approach for human activity recognition using location data at UbiComp/ISWC 2021.
Publications & Preprints
Google Scholar →
Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
On-policy self-distillation tells a student what to predict, but not where to look. OPASD adds solution-conditioned attention distillation, projecting a privileged teacher's attention onto student-visible context. It beats token-only OPSD by 4.98–8.40 points across three Qwen3 sizes while cutting rollout tokens by 73.9%.
EK-Bangla: When Hand-Object Interaction Meets Low-Resource Generation
EK-Bangla is the first benchmark for fine-grained egocentric hand-object interaction in Bangla, built on EPIC-KITCHENS-100: 9,600 native-speaker-verified captions and 2,000 four-way MCQs. Across nine MLLMs, rankings flip between captioning and MCQ, and every model scores lower in Bangla than in English on identical clips.
Project page
Paper & code coming soon
LAGEA: Language Guided Embodied Agents for Robotic Manipulation
LAGEA turns structured vision-language-model reflections on a robot's own failures into temporally grounded, progress-based shaping rewards. It localizes the decisive moments of an episode, aligns feedback with visual states, and fades the guidance as the policy improves, outperforming FuRL on Meta-World MT10 and Fetch.
T3Time: Tri-Modal Time Series Forecasting via Adaptive Multi-Head Alignment and Residual Fusion
T3Time reads a time series three ways (temporal, spectral, and through an LLM prompt), gates temporal versus spectral features by forecast horizon, and fuses modalities with adaptively weighted cross-modal attention heads. It sets the best MSE on 7 of 8 long-term benchmarks and stays strong with only 5–10% of the training data.
Counting Through Occlusion: Framework for Open World Amodal Counting
State-of-the-art counters only count what they can see. CountOCC reconstructs the features of hidden objects from visible context and text/exemplar priors, and enforces attention consistency between occluded and unoccluded views. It sets a new state of the art on our occlusion benchmarks FSC-147-OCC and CARPK-OCC, and on CAPTURe-Real.
SignFormer-GCN: Continuous Sign Language Translation Using Spatio-Temporal Graph Convolutional Networks
SignFormer-GCN jointly encodes RGB video with a transformer and skeletal keypoints with an STGCN-LSTM, capturing both broad context and fine-grained body motion. It delivers competitive gloss-free translation on German, American, and Bangla sign language benchmarks with only 9.43M parameters.
Bornil: An Open-Source Sign Language Data Crowdsourcing Platform for AI Enabled Dialect-Agnostic Communication
Bornil is an open-source platform for recording, annotating, and validating multilingual sign language data. It was used to build BornilDB v1.0, the largest Bangladeshi Sign Language dataset (73 hours, 21,000+ samples), with recognition benchmarks for this low-resource setting.
Classical Machine Learning Approach for Human Activity Recognition Using Location Data
Our entry to the Sussex-Huawei Locomotion Challenge 2021 recognizes eight locomotion and transportation activities from smartphone location data using statistical features and a random forest, reaching 78.14% validation accuracy and a 78.28% weighted F1 score.
Experience
Research
-
Mar 2026 – Present
Research Assistant
Dept. of Electrical and Computer Engineering, North South University · with Md Salman Shamil and Dr. Md Adnan Arefeen
- Working on egocentric video understanding, modeling hand-object interactions and daily task actions with LLM-guided reasoning.
- Led EK-Bangla, the first Bangla benchmark for fine-grained egocentric hand-object interaction, with 9,600 verified captions and 2,000 MCQs.
-
Jun 2025 – Feb 2026
Research Assistant
Dept. of Robotics and Mechatronics Engineering, University of Dhaka · with Dr. Md Mehedi Hasan and Md. Jubair Ahmed Sourov
- Designed and implemented CountOCC, the first open-world amodal counting framework, and assembled occlusion-augmented datasets to evaluate it.
-
Jan 2023 – Dec 2023
Undergraduate Research Assistant
Dept. of Robotics and Mechatronics Engineering, University of Dhaka · with Dr. Shafin Rahman and Dr. Sejuti Rahman
- Designed and implemented a novel architecture for continuous sign language translation that achieved competitive performance across Bangla, American, and German benchmarks.
- Co-authored the resulting work, SignFormer-GCN, published in PLOS ONE.
- Secured a research grant from the Bangladesh Ministry of Science and Technology's Special Innovation Fund (Project ID: SRG-232431, FY 2023–2024).
Industry
-
Apr 2024 – May 2025
Associate Software Engineer, QA
Therap (BD) Ltd. · Machine Learning team · Dhaka, Bangladesh
- Researched Amazon Alexa devices and sound detection systems, reviewed action recognition datasets, and contributed to the design of data collection for action recognition.
- Led quality assurance for the video redaction system, building test plans and cases to validate the machine learning algorithms used in video processing.
-
Jan 2024 – Mar 2024
AIM Intern
Pathao Limited · Data Science, Analytics & Insights · Dhaka, Bangladesh
- Developed five KPIs and metrics for measuring the brand health of Pathao Courier, aligned with company-wide strategic goals.
- Designed data-driven questionnaires to assess brand health, improving response quality and depth of insights.
Selected Projects
Course project · Aug – Nov 2022
Automated Stock Trading
A reinforcement learning agent that learns when to buy, sell, or hold three Bangladeshi stocks, using Q-learning with linear function approximation, tested with and without the COVID-19 period.
View project →
Course project · Aug – Nov 2023
Emotional Support: NAO Robot vs. Humans
Can a robot comfort people as well as a human? We programmed NAO to respond to happiness, sadness, and fear, then compared it with human support on emotional impact, comfort, clarity, and satisfaction.
View project →