Course project Reinforcement LearningFinance

Automated Stock Trading Using Approximate Q-Learning

Rabeya Akter, Safaeid Hossain Arib

RME 3211 · Intelligent Systems and Robotics Lab, University of Dhaka · Aug – Nov 2022

Supervised by Dr. Sejuti Rahman

In short

Instead of hand-written trading rules, a reinforcement learning agent learns when to buy, sell or hold three Bangladeshi stocks from price history alone.

  • 13.29%return on investment, test period including COVID-19
  • 3stocks: Grameenphone, Square Pharma, ACI Formulations
  • 3actions: buy, sell or hold
Illustration for Automated Stock Trading Using Approximate Q-Learning
Learned trading policy on Square Pharmaceuticals. Price over the test period with the agent's buy (red) and sell (blue) decisions.

Approach

Trading is framed as a sequential decision problem. At each step the agent observes features of recent prices and chooses to buy, sell or hold. Because the state space is continuous, a tabular Q-table is replaced by approximate Q-learning, where Q-values are a linear function of the state features and the weights are updated from the temporal-difference error.

\[ Q(s,a) = \mathbf{w}^\top \mathbf{f}(s,a), \qquad \mathbf{w} \leftarrow \mathbf{w} + \alpha \big[ r + \gamma \max_{a'} Q(s',a') - Q(s,a) \big]\, \mathbf{f}(s,a) \]

Stocks

Results

Return on investment on the test period

Test periodROI
Including the COVID-19 timeline13.29%
Excluding the COVID-19 timeline−37.46%
Grameenphone price with buy and sell points
Grameenphone Ltd. Buy (red) and sell (blue) decisions.
ACI Formulations price with buy and sell points
ACI Formulations Ltd. Buy (red) and sell (blue) decisions.

The code builds on this open-source implementation.

Tools: Python, Jupyter