Dynamic Portfolio Management with Deep Reinforcement Learning

DRL is used to provide dynamic asset allocation for a portfolio

Investment Industry
Reinforcement Learning
A2C
PPO
OpenAI Gym
finrl
stable baselines3
pyfolio
Author

Kobus Esterhuysen

Published

November 11, 2021

1. Problem: Dynamic Asset Allocation

Asset allocation refers to the partitioning of available funds in a portfolio among various investment products. Investment products may be categorized in multiple ways. One common (and high level) classification contains three classes:

  • Stocks
  • Bonds (fixed income)
  • Cash and cash equivalents

Other classification systems may sub-classify each of these categories into subclasses, for example stocks into US and International stocks, or into small-cap, medium-cap, and large-cap stocks, etc.

Another aspect of asset allocation concerns the underlying strategy. Some strategies are:

  • Tactical asset allocation
  • Strategic asset allocation
  • Constant weight asset allocation
  • Integrated asset allocation
  • Insured asset allocation
  • Dynamic asset allocation

We will focus on the last of these strategies: Dynamic asset allocation. With this approach the combination of assets are adjusted at regular intervals to capitalize on the strengthening and weakening of the economy and rise and fall of markets. This strategy depends on the decisions of a portfolio manager. In this project, however, we will attempt to replace the portfolio manager with an AI agent. The agent will be trained on past market behavior by means of Deep Reinforcement Learning.

For this Proof-Of-Concept (POC) project we will make things as simple as possible. The portfolio will consist of the 30 investment products in the Dow-Jones (DJIA). The fixed allocation in the DJIA also provides a handy reference against which the performance of our agent’s dynamic behavior can be juxtaposed. Our agent will have the opportunity to adjust the mix of stocks on a daily basis. In practice, a 401K agent, for example, might be setup to make monthly adjustments to reduce transation costs or to avoid constraints related to trading. For simplicity, trading costs will not be taken into account for now.

Our agent will have $ 1,000,000 when the project starts.

To implement this POC we will make use of the FinRL framework as well as some Yahoo technology to acquire financial data.

2. Solution Proposal

Investment decisions are sequential by nature. Furthermore, an optimal decision in the present may turn out not to be optimal over the longer term. Then there are the complexities of the investment landscape like varying market conditions, political disruptive events, and other economic uncertainties. The ideal tool for this kind of problem is Reinforcement Learning.

The solution requires the setup of a digital twin for the investor’s portfolio. In RL terms this model of the portfolio is called an environment. The environment contains states which are modified by applying actions to it.

We will choose the following state vector (measured daily in our case) for the environment:

\[ \Large \begin{aligned} s_1 &= \text{Value of AAPL holdings} \\ s_2 &= \text{Value of AMGN holdings} \\ s_3 &= \text{Value of AXP holdings} \\ \text{...} \\ s_{29} &= \text{Value of WBA holdings} \\ s_{30} &= \text{Value of WMT holdings} \end{aligned} \]

The following action vector (applied daily in our case) will be setup to influence the environment/portfolio:

\[ \Large \begin{aligned} a_1 &= \text{Fraction to be invested in AAPL this cycle} \\ a_2 &= \text{Fraction to be invested in AMGN this cycle} \\ a_3 &= \text{Fraction to be invested in AXP this cycle} \\ \text{...} \\ a_{29} &= \text{Fraction to be invested in WBA this cycle} \\ a_{30} &= \text{Fraction to be invested in WMT this cycle} \end{aligned} \]

All action values are in the [0, 1] interval.

The reward r is given by

\[ \Large \begin{aligned} r(s,a,s') &= log(v'/v) \end{aligned} \]

where v and v’ are the portfolio value at states \(s'\) and \(s\) respectively.

The model of the portfolio/environment will have the following parameters:

\[ \Large \begin{aligned} \theta_1 &= \text{Initial Amount} \\ \theta_2 &= \text{Transaction Cost Pct} \\ \theta_3 &= \text{Size of State and Action Spaces} \\ \theta_4 &= \text{Reward Scaling} \\ \theta_5 &= \text{Technical Indicator List} \end{aligned} \]

3. Implementation of the Solution

To implement the environment we use the OpenAI Gym tools. We will use the FinRL python library to implement the agent. For a function approximator for the agent, two algorithms will be investigated:

  • Advantage Actor-Critic (A2C)
  • Proximal Policy Optimization (PPO)

This implementation allows the agent to allocate the investor’s funds between the 30 DJIA instruments. The goal is to maximize net worth at the end of the investment horizon.

The plotly library will be used for visualization. The code will run on the Google Colab platform. To start with, we install the python packages needed.

# hide
# install plotly and finrl library
!pip install plotly==4.4.1
!wget https://github.com/plotly/orca/releases/download/v1.2.1/orca-1.2.1-x86_64.AppImage -O /usr/local/bin/orca
!chmod +x /usr/local/bin/orca
!apt-get install xvfb libgtk2.0-0 libgconf-2-4
!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git
!pip install PyPortfolioOpt
Requirement already satisfied: plotly==4.4.1 in /usr/local/lib/python3.7/dist-packages (4.4.1)
Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.7/dist-packages (from plotly==4.4.1) (1.3.3)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from plotly==4.4.1) (1.15.0)
--2021-11-12 21:57:22--  https://github.com/plotly/orca/releases/download/v1.2.1/orca-1.2.1-x86_64.AppImage
Resolving github.com (github.com)... 52.192.72.89
Connecting to github.com (github.com)|52.192.72.89|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/99037241/9dc3a580-286a-11e9-8a21-4312b7c8a512?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211112T215722Z&X-Amz-Expires=300&X-Amz-Signature=a52ba98db790aac775a4219741ecb736accaf01243a4de8b88e500183bd5767c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=99037241&response-content-disposition=attachment%3B%20filename%3Dorca-1.2.1-x86_64.AppImage&response-content-type=application%2Foctet-stream [following]
--2021-11-12 21:57:22--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/99037241/9dc3a580-286a-11e9-8a21-4312b7c8a512?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211112T215722Z&X-Amz-Expires=300&X-Amz-Signature=a52ba98db790aac775a4219741ecb736accaf01243a4de8b88e500183bd5767c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=99037241&response-content-disposition=attachment%3B%20filename%3Dorca-1.2.1-x86_64.AppImage&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 51607939 (49M) [application/octet-stream]
Saving to: ‘/usr/local/bin/orca’

/usr/local/bin/orca 100%[===================>]  49.22M   134MB/s    in 0.4s    

2021-11-12 21:57:39 (134 MB/s) - ‘/usr/local/bin/orca’ saved [51607939/51607939]

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gconf-service gconf-service-backend gconf2-common libdbus-glib-1-2
  libgail-common libgail18 libgtk2.0-bin libgtk2.0-common
Suggested packages:
  gvfs
The following NEW packages will be installed:
  gconf-service gconf-service-backend gconf2-common libdbus-glib-1-2
  libgail-common libgail18 libgconf-2-4 libgtk2.0-0 libgtk2.0-bin
  libgtk2.0-common xvfb
0 upgraded, 11 newly installed, 0 to remove and 37 not upgraded.
Need to get 3,715 kB of archives.
After this operation, 17.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libdbus-glib-1-2 amd64 0.110-2 [58.3 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 gconf2-common all 3.2.6-4ubuntu1 [700 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libgconf-2-4 amd64 3.2.6-4ubuntu1 [84.8 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 gconf-service-backend amd64 3.2.6-4ubuntu1 [58.1 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 gconf-service amd64 3.2.6-4ubuntu1 [2,036 B]
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgtk2.0-common all 2.24.32-1ubuntu1 [125 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgtk2.0-0 amd64 2.24.32-1ubuntu1 [1,769 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgail18 amd64 2.24.32-1ubuntu1 [14.2 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgail-common amd64 2.24.32-1ubuntu1 [112 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libgtk2.0-bin amd64 2.24.32-1ubuntu1 [7,536 B]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.9 [784 kB]
Fetched 3,715 kB in 3s (1,432 kB/s)
Selecting previously unselected package libdbus-glib-1-2:amd64.
(Reading database ... 155219 files and directories currently installed.)
Preparing to unpack .../00-libdbus-glib-1-2_0.110-2_amd64.deb ...
Unpacking libdbus-glib-1-2:amd64 (0.110-2) ...
Selecting previously unselected package gconf2-common.
Preparing to unpack .../01-gconf2-common_3.2.6-4ubuntu1_all.deb ...
Unpacking gconf2-common (3.2.6-4ubuntu1) ...
Selecting previously unselected package libgconf-2-4:amd64.
Preparing to unpack .../02-libgconf-2-4_3.2.6-4ubuntu1_amd64.deb ...
Unpacking libgconf-2-4:amd64 (3.2.6-4ubuntu1) ...
Selecting previously unselected package gconf-service-backend.
Preparing to unpack .../03-gconf-service-backend_3.2.6-4ubuntu1_amd64.deb ...
Unpacking gconf-service-backend (3.2.6-4ubuntu1) ...
Selecting previously unselected package gconf-service.
Preparing to unpack .../04-gconf-service_3.2.6-4ubuntu1_amd64.deb ...
Unpacking gconf-service (3.2.6-4ubuntu1) ...
Selecting previously unselected package libgtk2.0-common.
Preparing to unpack .../05-libgtk2.0-common_2.24.32-1ubuntu1_all.deb ...
Unpacking libgtk2.0-common (2.24.32-1ubuntu1) ...
Selecting previously unselected package libgtk2.0-0:amd64.
Preparing to unpack .../06-libgtk2.0-0_2.24.32-1ubuntu1_amd64.deb ...
Unpacking libgtk2.0-0:amd64 (2.24.32-1ubuntu1) ...
Selecting previously unselected package libgail18:amd64.
Preparing to unpack .../07-libgail18_2.24.32-1ubuntu1_amd64.deb ...
Unpacking libgail18:amd64 (2.24.32-1ubuntu1) ...
Selecting previously unselected package libgail-common:amd64.
Preparing to unpack .../08-libgail-common_2.24.32-1ubuntu1_amd64.deb ...
Unpacking libgail-common:amd64 (2.24.32-1ubuntu1) ...
Selecting previously unselected package libgtk2.0-bin.
Preparing to unpack .../09-libgtk2.0-bin_2.24.32-1ubuntu1_amd64.deb ...
Unpacking libgtk2.0-bin (2.24.32-1ubuntu1) ...
Selecting previously unselected package xvfb.
Preparing to unpack .../10-xvfb_2%3a1.19.6-1ubuntu4.9_amd64.deb ...
Unpacking xvfb (2:1.19.6-1ubuntu4.9) ...
Setting up gconf2-common (3.2.6-4ubuntu1) ...

Creating config file /etc/gconf/2/path with new version
Setting up libgtk2.0-common (2.24.32-1ubuntu1) ...
Setting up libdbus-glib-1-2:amd64 (0.110-2) ...
Setting up xvfb (2:1.19.6-1ubuntu4.9) ...
Setting up libgconf-2-4:amd64 (3.2.6-4ubuntu1) ...
Setting up libgtk2.0-0:amd64 (2.24.32-1ubuntu1) ...
Setting up libgail18:amd64 (2.24.32-1ubuntu1) ...
Setting up libgail-common:amd64 (2.24.32-1ubuntu1) ...
Setting up libgtk2.0-bin (2.24.32-1ubuntu1) ...
Setting up gconf-service-backend (3.2.6-4ubuntu1) ...
Setting up gconf-service (3.2.6-4ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.3) ...
/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link

Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Collecting git+https://github.com/AI4Finance-LLC/FinRL-Library.git
  Cloning https://github.com/AI4Finance-LLC/FinRL-Library.git to /tmp/pip-req-build-nlro72g1
  Running command git clone -q https://github.com/AI4Finance-LLC/FinRL-Library.git /tmp/pip-req-build-nlro72g1
Collecting pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2
  Cloning https://github.com/quantopian/pyfolio.git to /tmp/pip-install-dn7avd_q/pyfolio_54a8fde5869f4ed987d7243fe4fabcc6
  Running command git clone -q https://github.com/quantopian/pyfolio.git /tmp/pip-install-dn7avd_q/pyfolio_54a8fde5869f4ed987d7243fe4fabcc6
Collecting elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl
  Cloning https://github.com/AI4Finance-Foundation/ElegantRL.git to /tmp/pip-install-dn7avd_q/elegantrl_774a7934ab324172bbba319a79ce7984
  Running command git clone -q https://github.com/AI4Finance-Foundation/ElegantRL.git /tmp/pip-install-dn7avd_q/elegantrl_774a7934ab324172bbba319a79ce7984
Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (1.19.5)
Requirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (1.1.5)
Collecting stockstats
  Downloading stockstats-0.3.2-py2.py3-none-any.whl (13 kB)
Collecting yfinance
  Downloading yfinance-0.1.64.tar.gz (26 kB)
Collecting elegantrl
  Downloading elegantrl-0.3.2-py3-none-any.whl (73 kB)
     |████████████████████████████████| 73 kB 1.6 MB/s 
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (3.2.2)
Requirement already satisfied: scikit-learn>=0.21.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (0.22.2.post1)
Requirement already satisfied: gym>=0.17 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (0.17.3)
Collecting stable-baselines3[extra]
  Downloading stable_baselines3-1.3.0-py3-none-any.whl (174 kB)
     |████████████████████████████████| 174 kB 27.8 MB/s 
Collecting ray[default]
  Downloading ray-1.8.0-cp37-cp37m-manylinux2014_x86_64.whl (54.7 MB)
     |████████████████████████████████| 54.7 MB 31 kB/s 
Collecting lz4
  Downloading lz4-3.1.3-cp37-cp37m-manylinux2010_x86_64.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 38.6 MB/s 
Collecting tensorboardX
  Downloading tensorboardX-2.4-py2.py3-none-any.whl (124 kB)
     |████████████████████████████████| 124 kB 51.3 MB/s 
Collecting gputil
  Downloading GPUtil-1.4.0.tar.gz (5.5 kB)
Collecting trading_calendars
  Downloading trading_calendars-2.1.1.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 50.3 MB/s 
Collecting alpaca_trade_api
  Downloading alpaca_trade_api-1.4.1-py3-none-any.whl (36 kB)
Collecting ccxt
  Downloading ccxt-1.61.9-py2.py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 48.5 MB/s 
Collecting jqdatasdk
  Downloading jqdatasdk-1.8.10-py3-none-any.whl (153 kB)
     |████████████████████████████████| 153 kB 43.4 MB/s 
Collecting wrds
  Downloading wrds-3.1.1-py3-none-any.whl (12 kB)
Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (3.6.4)
Requirement already satisfied: setuptools>=41.4.0 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (57.4.0)
Requirement already satisfied: wheel>=0.33.6 in /usr/local/lib/python3.7/dist-packages (from finrl==0.3.3) (0.37.0)
Collecting pre-commit
  Downloading pre_commit-2.15.0-py2.py3-none-any.whl (191 kB)
     |████████████████████████████████| 191 kB 47.0 MB/s 
Collecting pybullet
  Downloading pybullet-3.2.0-cp37-cp37m-manylinux1_x86_64.whl (89.3 MB)
     |████████████████████████████████| 89.3 MB 24 kB/s 
Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.3) (1.10.0+cu111)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.3) (4.1.2.30)
Collecting box2d-py
  Downloading box2d_py-2.3.8-cp37-cp37m-manylinux1_x86_64.whl (448 kB)
     |████████████████████████████████| 448 kB 48.5 MB/s 
Requirement already satisfied: ipython>=3.2.3 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (5.5.0)
Requirement already satisfied: pytz>=2014.10 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (2018.9)
Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (1.4.1)
Requirement already satisfied: seaborn>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (0.11.2)
Collecting empyrical>=0.5.0
  Downloading empyrical-0.5.5.tar.gz (52 kB)
     |████████████████████████████████| 52 kB 1.4 MB/s 
Requirement already satisfied: pandas-datareader>=0.2 in /usr/local/lib/python3.7/dist-packages (from empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (0.9.0)
Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.17->finrl==0.3.3) (1.5.0)
Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.17->finrl==0.3.3) (1.3.0)
Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (0.8.1)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (5.1.1)
Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (4.4.2)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (1.0.18)
Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (4.8.0)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (0.7.5)
Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (2.6.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.3) (1.3.2)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.3) (2.8.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.3) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->finrl==0.3.3) (0.11.0)
Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (4.2.6)
Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (2.23.0)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (1.15.0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (0.2.5)
Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym>=0.17->finrl==0.3.3) (0.16.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (2021.10.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->pandas-datareader>=0.2->empyrical>=0.5.0->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (2.10)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.0->finrl==0.3.3) (1.1.0)
Collecting PyYAML==5.4.1
  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
     |████████████████████████████████| 636 kB 44.9 MB/s 
Collecting aiohttp==3.7.4
  Downloading aiohttp-3.7.4-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 36.1 MB/s 
Collecting websocket-client<2,>=0.56.0
  Downloading websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 1.3 MB/s 
Collecting websockets<10,>=8.0
  Downloading websockets-9.1-cp37-cp37m-manylinux2010_x86_64.whl (103 kB)
     |████████████████████████████████| 103 kB 50.8 MB/s 
Requirement already satisfied: msgpack==1.0.2 in /usr/local/lib/python3.7/dist-packages (from alpaca_trade_api->finrl==0.3.3) (1.0.2)
Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp==3.7.4->alpaca_trade_api->finrl==0.3.3) (3.10.0.2)
Collecting yarl<2.0,>=1.0
  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)
     |████████████████████████████████| 271 kB 48.8 MB/s 
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp==3.7.4->alpaca_trade_api->finrl==0.3.3) (21.2.0)
Collecting async-timeout<4.0,>=3.0
  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting multidict<7.0,>=4.5
  Downloading multidict-5.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (160 kB)
     |████████████████████████████████| 160 kB 41.2 MB/s 
Collecting aiodns>=1.1.1
  Downloading aiodns-3.0.0-py3-none-any.whl (5.0 kB)
Collecting cryptography>=2.6.1
  Downloading cryptography-35.0.0-cp36-abi3-manylinux_2_24_x86_64.whl (3.5 MB)
     |████████████████████████████████| 3.5 MB 40.2 MB/s 
Collecting yarl<2.0,>=1.0
  Downloading yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294 kB)
     |████████████████████████████████| 294 kB 20.1 MB/s 
Collecting pycares>=4.0.0
  Downloading pycares-4.1.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (291 kB)
     |████████████████████████████████| 291 kB 43.6 MB/s 
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=2.6.1->ccxt->finrl==0.3.3) (1.15.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt->finrl==0.3.3) (2.21)
Collecting pymysql>=0.7.6
  Downloading PyMySQL-1.0.2-py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 2.0 MB/s 
Requirement already satisfied: SQLAlchemy>=1.2.8 in /usr/local/lib/python3.7/dist-packages (from jqdatasdk->finrl==0.3.3) (1.4.26)
Collecting thriftpy2>=0.3.9
  Downloading thriftpy2-0.4.14.tar.gz (361 kB)
     |████████████████████████████████| 361 kB 13.2 MB/s 
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.3) (4.8.2)
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.7/dist-packages (from SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.3) (1.1.2)
Collecting ply<4.0,>=3.4
  Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 5.3 MB/s 
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->SQLAlchemy>=1.2.8->jqdatasdk->finrl==0.3.3) (3.6.0)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect->ipython>=3.2.3->pyfolio@ git+https://github.com/quantopian/pyfolio.git#egg=pyfolio-0.9.2->finrl==0.3.3) (0.7.0)
Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pre-commit->finrl==0.3.3) (0.10.2)
Collecting identify>=1.0.0
  Downloading identify-2.3.5-py2.py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 6.4 MB/s 
Collecting cfgv>=2.0.0
  Downloading cfgv-3.3.1-py2.py3-none-any.whl (7.3 kB)
Collecting virtualenv>=20.0.8
  Downloading virtualenv-20.10.0-py2.py3-none-any.whl (5.6 MB)
     |████████████████████████████████| 5.6 MB 33.8 MB/s 
Collecting nodeenv>=0.11.1
  Downloading nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
Collecting platformdirs<3,>=2
  Downloading platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting backports.entry-points-selectable>=1.0.4
  Downloading backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl (6.2 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.3-py2.py3-none-any.whl (496 kB)
     |████████████████████████████████| 496 kB 46.0 MB/s 
Requirement already satisfied: filelock<4,>=3.2 in /usr/local/lib/python3.7/dist-packages (from virtualenv>=20.0.8->pre-commit->finrl==0.3.3) (3.3.2)
Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.3) (8.10.0)
Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.3) (1.11.0)
Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.3) (1.4.0)
Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.7/dist-packages (from pytest->finrl==0.3.3) (0.7.1)
Requirement already satisfied: protobuf>=3.15.3 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.3) (3.17.3)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.3) (7.1.2)
Collecting redis>=3.5.0
  Downloading redis-3.5.3-py2.py3-none-any.whl (72 kB)
     |████████████████████████████████| 72 kB 475 kB/s 
Requirement already satisfied: jsonschema in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.3) (2.6.0)
Requirement already satisfied: grpcio>=1.28.1 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.3) (1.41.1)
Collecting py-spy>=0.2.0
  Downloading py_spy-0.3.10-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 49.6 MB/s 
Collecting aiohttp-cors
  Downloading aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)
Collecting aioredis<2
  Downloading aioredis-1.3.1-py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 1.6 MB/s 
Requirement already satisfied: prometheus-client>=0.7.1 in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.3) (0.12.0)
Collecting colorful
  Downloading colorful-0.5.4-py2.py3-none-any.whl (201 kB)
     |████████████████████████████████| 201 kB 47.8 MB/s 
Collecting gpustat>=1.0.0b1
  Downloading gpustat-1.0.0b1.tar.gz (82 kB)
     |████████████████████████████████| 82 kB 242 kB/s 
Collecting opencensus
  Downloading opencensus-0.8.0-py2.py3-none-any.whl (128 kB)
     |████████████████████████████████| 128 kB 38.8 MB/s 
Collecting hiredis
  Downloading hiredis-2.0.0-cp37-cp37m-manylinux2010_x86_64.whl (85 kB)
     |████████████████████████████████| 85 kB 3.5 MB/s 
Requirement already satisfied: nvidia-ml-py3>=7.352.0 in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[default]->finrl==0.3.3) (7.352.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from gpustat>=1.0.0b1->ray[default]->finrl==0.3.3) (5.4.8)
Collecting blessed>=1.17.1
  Downloading blessed-1.19.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 5.0 MB/s 
Collecting opencensus-context==0.1.2
  Downloading opencensus_context-0.1.2-py2.py3-none-any.whl (4.4 kB)
Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from opencensus->ray[default]->finrl==0.3.3) (1.26.3)
Requirement already satisfied: google-auth<2.0dev,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (1.35.0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (1.53.0)
Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (21.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (4.2.4)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.21.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]->finrl==0.3.3) (0.4.8)
Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from ray[default]->finrl==0.3.3) (0.8.9)
Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.3) (7.1.2)
Requirement already satisfied: atari-py~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.3) (0.2.9)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]->finrl==0.3.3) (2.7.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (0.4.6)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (1.8.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (0.12.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (1.0.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (3.3.4)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (1.3.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]->finrl==0.3.3) (3.1.1)
Collecting int-date>=0.1.7
  Downloading int_date-0.1.8-py2.py3-none-any.whl (5.0 kB)
Requirement already satisfied: toolz in /usr/local/lib/python3.7/dist-packages (from trading_calendars->finrl==0.3.3) (0.11.2)
Collecting mock
  Downloading mock-4.0.3-py3-none-any.whl (28 kB)
Collecting psycopg2-binary
  Downloading psycopg2_binary-2.9.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 17.4 MB/s 
Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from yfinance->finrl==0.3.3) (0.0.9)
Collecting lxml
  Downloading lxml-4.6.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.3 MB)
     |████████████████████████████████| 6.3 MB 27.1 MB/s 
Building wheels for collected packages: finrl, elegantrl, pyfolio, empyrical, gputil, thriftpy2, gpustat, trading-calendars, yfinance
  Building wheel for finrl (setup.py) ... done
  Created wheel for finrl: filename=finrl-0.3.3-py3-none-any.whl size=3883610 sha256=0333e20cc0b430e76f2663b22ca8a5753f4ecb81ceeb6f0b2fd615c7706fc7e3
  Stored in directory: /tmp/pip-ephem-wheel-cache-wc_7pvc9/wheels/17/ff/bd/1bc602a0352762b0b24041b88536d803ae343ed0a711fcf55e
  Building wheel for elegantrl (setup.py) ... done
  Created wheel for elegantrl: filename=elegantrl-0.3.2-py3-none-any.whl size=77488 sha256=c5dd619660980e1a7c8b7a8dcf49052cc5491ecd9d8f391418ffd66feed5d949
  Stored in directory: /tmp/pip-ephem-wheel-cache-wc_7pvc9/wheels/99/85/5e/86cb3a9f47adfca5e248295e93113e1b298d60883126d62c84
  Building wheel for pyfolio (setup.py) ... done
  Created wheel for pyfolio: filename=pyfolio-0.9.2+75.g4b901f6-py3-none-any.whl size=75775 sha256=0464192356aec68b023b75a389adbaa52803f3e31db912e20145c58b6f5b2fcf
  Stored in directory: /tmp/pip-ephem-wheel-cache-wc_7pvc9/wheels/ef/09/e5/2c1bf37c050d22557c080deb1be986d06424627c04aeca19b9
  Building wheel for empyrical (setup.py) ... done
  Created wheel for empyrical: filename=empyrical-0.5.5-py3-none-any.whl size=39777 sha256=497050237fd528a326e5f2d955b8ee7e098e824c3f04055f3fb6ba9e33426c81
  Stored in directory: /root/.cache/pip/wheels/d9/91/4b/654fcff57477efcf149eaca236da2fce991526cbab431bf312
  Building wheel for gputil (setup.py) ... done
  Created wheel for gputil: filename=GPUtil-1.4.0-py3-none-any.whl size=7411 sha256=a40a5dc70978886728f3b5d9c60393e3921884f599ea91cc50cd859ae6470993
  Stored in directory: /root/.cache/pip/wheels/6e/f8/83/534c52482d6da64622ddbf72cd93c35d2ef2881b78fd08ff0c
  Building wheel for thriftpy2 (setup.py) ... done
  Created wheel for thriftpy2: filename=thriftpy2-0.4.14-cp37-cp37m-linux_x86_64.whl size=940442 sha256=91ef050f6a5b4de013322081734d600dd513beda23fab6e7c7f44c5c7d207841
  Stored in directory: /root/.cache/pip/wheels/2a/f5/49/9c0d851aa64b58db72883cf9393cc824d536bdf13f5c83cff4
  Building wheel for gpustat (setup.py) ... done
  Created wheel for gpustat: filename=gpustat-1.0.0b1-py3-none-any.whl size=15979 sha256=278bbde3c7fd20e96255dee7cfbd38f5e371a5a25a2cf3ff39af81a625761253
  Stored in directory: /root/.cache/pip/wheels/1a/16/e2/3e2437fba4c4b6a97a97bd96fce5d14e66cff5c4966fb1cc8c
  Building wheel for trading-calendars (setup.py) ... done
  Created wheel for trading-calendars: filename=trading_calendars-2.1.1-py3-none-any.whl size=140937 sha256=47f99595ffc22c8a43cd331064a3ff6c63b3fa338f9a58b2ba4615673d4ec3e2
  Stored in directory: /root/.cache/pip/wheels/62/9c/d1/46a21e1b99e064cba79b85e9f95e6a208ac5ba4c29ae5962ec
  Building wheel for yfinance (setup.py) ... done
  Created wheel for yfinance: filename=yfinance-0.1.64-py2.py3-none-any.whl size=24109 sha256=aeced28ff4b7c392e008831c5c62a79f90c29000f0b794ddfeeb3a58128391f2
  Stored in directory: /root/.cache/pip/wheels/86/fe/9b/a4d3d78796b699e37065e5b6c27b75cff448ddb8b24943c288
Successfully built finrl elegantrl pyfolio empyrical gputil thriftpy2 gpustat trading-calendars yfinance
Installing collected packages: multidict, yarl, lxml, async-timeout, redis, PyYAML, pycares, ply, platformdirs, opencensus-context, hiredis, distlib, blessed, backports.entry-points-selectable, aiohttp, websockets, websocket-client, virtualenv, thriftpy2, tensorboardX, stable-baselines3, ray, pymysql, pybullet, py-spy, psycopg2-binary, opencensus, nodeenv, mock, int-date, identify, gpustat, empyrical, cryptography, colorful, cfgv, box2d-py, aioredis, aiohttp-cors, aiodns, yfinance, wrds, trading-calendars, stockstats, pyfolio, pre-commit, lz4, jqdatasdk, gputil, elegantrl, ccxt, alpaca-trade-api, finrl
  Attempting uninstall: lxml
    Found existing installation: lxml 4.2.6
    Uninstalling lxml-4.2.6:
      Successfully uninstalled lxml-4.2.6
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 3.13
    Uninstalling PyYAML-3.13:
      Successfully uninstalled PyYAML-3.13
Successfully installed PyYAML-5.4.1 aiodns-3.0.0 aiohttp-3.7.4 aiohttp-cors-0.7.0 aioredis-1.3.1 alpaca-trade-api-1.4.1 async-timeout-3.0.1 backports.entry-points-selectable-1.1.1 blessed-1.19.0 box2d-py-2.3.8 ccxt-1.61.9 cfgv-3.3.1 colorful-0.5.4 cryptography-35.0.0 distlib-0.3.3 elegantrl-0.3.2 empyrical-0.5.5 finrl-0.3.3 gpustat-1.0.0b1 gputil-1.4.0 hiredis-2.0.0 identify-2.3.5 int-date-0.1.8 jqdatasdk-1.8.10 lxml-4.6.4 lz4-3.1.3 mock-4.0.3 multidict-5.2.0 nodeenv-1.6.0 opencensus-0.8.0 opencensus-context-0.1.2 platformdirs-2.4.0 ply-3.11 pre-commit-2.15.0 psycopg2-binary-2.9.2 py-spy-0.3.10 pybullet-3.2.0 pycares-4.1.2 pyfolio-0.9.2+75.g4b901f6 pymysql-1.0.2 ray-1.8.0 redis-3.5.3 stable-baselines3-1.3.0 stockstats-0.3.2 tensorboardX-2.4 thriftpy2-0.4.14 trading-calendars-2.1.1 virtualenv-20.10.0 websocket-client-1.2.1 websockets-9.1 wrds-3.1.1 yarl-1.6.3 yfinance-0.1.64
Collecting PyPortfolioOpt
  Downloading PyPortfolioOpt-1.5.1-py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 3.0 MB/s 
Requirement already satisfied: pandas>=0.19 in /usr/local/lib/python3.7/dist-packages (from PyPortfolioOpt) (1.1.5)
Requirement already satisfied: numpy<2.0,>=1.12 in /usr/local/lib/python3.7/dist-packages (from PyPortfolioOpt) (1.19.5)
Collecting cvxpy<2.0.0,>=1.1.10
  Downloading cvxpy-1.1.17-cp37-cp37m-manylinux_2_24_x86_64.whl (2.8 MB)
     |████████████████████████████████| 2.8 MB 9.0 MB/s 
Requirement already satisfied: scipy<2.0,>=1.3 in /usr/local/lib/python3.7/dist-packages (from PyPortfolioOpt) (1.4.1)
Requirement already satisfied: ecos>=2 in /usr/local/lib/python3.7/dist-packages (from cvxpy<2.0.0,>=1.1.10->PyPortfolioOpt) (2.0.7.post1)
Requirement already satisfied: scs>=1.1.6 in /usr/local/lib/python3.7/dist-packages (from cvxpy<2.0.0,>=1.1.10->PyPortfolioOpt) (2.1.4)
Requirement already satisfied: osqp>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from cvxpy<2.0.0,>=1.1.10->PyPortfolioOpt) (0.6.2.post0)
Requirement already satisfied: qdldl in /usr/local/lib/python3.7/dist-packages (from osqp>=0.4.1->cvxpy<2.0.0,>=1.1.10->PyPortfolioOpt) (0.1.5.post0)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.19->PyPortfolioOpt) (2018.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.19->PyPortfolioOpt) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=0.19->PyPortfolioOpt) (1.15.0)
Installing collected packages: cvxpy, PyPortfolioOpt
  Attempting uninstall: cvxpy
    Found existing installation: cvxpy 1.0.31
    Uninstalling cvxpy-1.0.31:
      Successfully uninstalled cvxpy-1.0.31
Successfully installed PyPortfolioOpt-1.5.1 cvxpy-1.1.17

Import the packages needed:

import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
matplotlib.use('Agg')
%matplotlib inline
import datetime
from finrl.apps import config
from finrl.neo_finrl.preprocessor.yahoodownloader import YahooDownloader
from finrl.neo_finrl.preprocessor.preprocessors import FeatureEngineer, data_split
from finrl.neo_finrl.env_portfolio_allocation.env_portfolio import StockPortfolioEnv
from finrl.drl_agents.stablebaselines3.models import DRLAgent
from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline,convert_daily_return_to_pyfolio_ts

import gym
from gym.utils import seeding
from gym import spaces
from stable_baselines3.common.vec_env import DummyVecEnv

from pyfolio import timeseries
import plotly
import plotly.graph_objs as go
/usr/local/lib/python3.7/dist-packages/pyfolio/pos.py:27: UserWarning: Module "zipline.assets" not found; multipliers will not be applied to position notionals.
  'Module "zipline.assets" not found; multipliers will not be applied'
import sys
sys.path.append("../FinRL-Library")
# hide
pd.set_option('display.max_rows', 100)

Setup some directories:

import os
if not os.path.exists("./" + config.DATA_SAVE_DIR):
    os.makedirs("./" + config.DATA_SAVE_DIR)
if not os.path.exists("./" + config.TRAINED_MODEL_DIR):
    os.makedirs("./" + config.TRAINED_MODEL_DIR)
if not os.path.exists("./" + config.TENSORBOARD_LOG_DIR):
    os.makedirs("./" + config.TENSORBOARD_LOG_DIR)
if not os.path.exists("./" + config.RESULTS_DIR):
    os.makedirs("./" + config.RESULTS_DIR)

3.0 PARAMETERS

DATA_START = '2008-01-01' 
TRAIN_START = '2009-01-01'
TRADE_START = '2020-07-01'
# TRADE_START = '2021-01-01'
# DATA_END = '2021-07-01'
# DATA_END = '2021-09-02'
DATA_END = '2021-09-01'
LOOKBACK = 252 #trading days in one year
INITIAL_AMOUNT = 1_000_000 #dollars
TRANSACTION_COST_PCT = 0
REWARD_SCALING = 1e-1 #scaling factor applied to the reward signal

3.1 Download Data

We use the data from Yahoo Finance.

df = YahooDownloader(start_date = DATA_START,
                     end_date = DATA_END,
                     ticker_list = config.DOW_30_TICKER).fetch_data()
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
[*********************100%***********************]  1 of 1 completed
Shape of DataFrame:  (100355, 8)
df
date open high low close volume tic day
0 2008-01-02 7.116786 7.152143 6.876786 5.966036 1079178800 AAPL 2
1 2008-01-02 46.599998 47.040001 46.259998 36.337456 7934400 AMGN 2
2 2008-01-02 52.090000 52.320000 50.790001 40.694561 8053700 AXP 2
3 2008-01-02 87.570000 87.839996 86.000000 63.481636 4303000 BA 2
4 2008-01-02 72.559998 72.669998 70.050003 47.633060 6337800 CAT 2
... ... ... ... ... ... ... ... ...
100350 2021-08-31 417.000000 420.739990 414.220001 414.813965 4249100 UNH 1
100351 2021-08-31 230.910004 231.839996 228.660004 228.697403 9405100 V 1
100352 2021-08-31 54.709999 55.139999 54.580002 54.354485 18405600 VZ 1
100353 2021-08-31 48.459999 50.869999 48.389999 50.750000 11814400 WBA 1
100354 2021-08-31 147.990005 148.440002 147.240005 148.100006 8359500 WMT 1

100355 rows × 8 columns

# hide
print(df['day'].unique())
# df.loc[100:150, 'day'] #assume day-of-week: 0 - 4
[2 3 4 0 1]
# 
# Verify 30 unique tickers
# https://www.investopedia.com/ask/answers/who-or-what-is-dow-jones/
lst = list(df['tic'].unique())
print(len(lst), lst)
30 ['AAPL', 'AMGN', 'AXP', 'BA', 'CAT', 'CRM', 'CSCO', 'CVX', 'DIS', 'GS', 'HD', 'HON', 'IBM', 'INTC', 'JNJ', 'JPM', 'KO', 'MCD', 'MMM', 'MRK', 'MSFT', 'NKE', 'PG', 'TRV', 'UNH', 'VZ', 'WBA', 'WMT', 'V', 'DOW']

3.2 Data Understanding and Preparation

We will keep showing snippets of the data set as it evolves to assist with understanding. There is a need to check for missing data and also to do some feature engineering. We rely on the FeatureEngineer class to take care of these needs. Some indicators used are:

  • Moving Average Convergence Divergence (MACD)

The MACD is primarily used to gauge the strength of stock price movement. It does this by measuring the divergence of two exponential moving averages (EMAs), commonly a 12-period EMA and a 26-period EMA.

  • Relative Strength Index (RSI)

The RSI aims to indicate whether a market is considered to be overbought or oversold in relation to recent price levels.

  • Commodity Channel Index (CCI)

The Commodity Channel Index​ (CCI) is a momentum-based oscillator used to help determine when an investment vehicle is reaching a condition of being overbought or oversold.

FinRL also uses the financial turbulence index that measures extreme asset price fluctuation.

3.2.1 Add technical indicators

%%time
fe = FeatureEngineer(
  use_technical_indicator=True,
  use_turbulence=False,
  user_defined_feature=False)
df = fe.preprocess_data(df)
Successfully added technical indicators
CPU times: user 1min 2s, sys: 6.75 s, total: 1min 8s
Wall time: 1min 2s
# df.head(100)
df
date open high low close volume tic day macd boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma
0 2008-01-02 7.116786 7.152143 6.876786 5.966036 1079178800 AAPL 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 5.966036 5.966036
3441 2008-01-02 46.599998 47.040001 46.259998 36.337456 7934400 AMGN 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 36.337456 36.337456
6882 2008-01-02 52.090000 52.320000 50.790001 40.694561 8053700 AXP 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 40.694561 40.694561
10323 2008-01-02 87.570000 87.839996 86.000000 63.481636 4303000 BA 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 63.481636 63.481636
13764 2008-01-02 72.559998 72.669998 70.050003 47.633060 6337800 CAT 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 47.633060 47.633060
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
82583 2021-08-31 158.850006 160.800003 158.710007 158.824631 1568700 TRV 1 2.301368 163.977861 149.229152 56.038937 73.949247 27.768804 153.781405 152.358006
86024 2021-08-31 417.000000 420.739990 414.220001 414.813965 4249100 UNH 1 1.795430 428.868267 402.859321 52.549565 11.116244 9.546535 415.015272 409.123889
89465 2021-08-31 54.709999 55.139999 54.580002 54.354485 18405600 VZ 1 -0.250367 55.499877 53.860358 44.653841 -120.717132 26.176223 54.859158 55.087388
92906 2021-08-31 48.459999 50.869999 48.389999 50.750000 11814400 WBA 1 0.388522 50.322940 46.070231 56.376587 155.229924 20.132924 47.624491 48.772953
96347 2021-08-31 147.990005 148.440002 147.240005 148.100006 8359500 WMT 1 1.379987 152.957874 143.371276 57.798913 35.525528 14.285037 146.036804 142.410712

96348 rows × 16 columns

We see that the FeatureEngineer has added some features:

  • macd
  • boll_ub (upper Bollinger Band)
  • boll_lb (lower Bollinger Band)
  • rsi_30 (with a lookback of 30)
  • cci_30 (with a lookback of 30)
  • dx_30 (with a lookback of 30)
  • close_30_sma (close price simple moving average with a lookback of 30)
  • close_60_sma (close price simple moving average with a lookback of 60)
# hide
# on stockstats library: (seems like FeatureEngineer makes use of it)
# https://medium.com/codex/this-python-library-will-help-you-get-stock-technical-indicators-in-one-line-of-code-c11ed2c8e45f

3.2.2 Add covariance matrix as a feature

Adding the portfolio’s covariance matrix as a feature has some advantages. It can be used to quantify the risk (standard deviation) associated with a portfolio.

df = df.sort_values(['date','tic'], ignore_index=True)
df
date open high low close volume tic day macd boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma
0 2008-01-02 7.116786 7.152143 6.876786 5.966036 1079178800 AAPL 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 5.966036 5.966036
1 2008-01-02 46.599998 47.040001 46.259998 36.337456 7934400 AMGN 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 36.337456 36.337456
2 2008-01-02 52.090000 52.320000 50.790001 40.694561 8053700 AXP 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 40.694561 40.694561
3 2008-01-02 87.570000 87.839996 86.000000 63.481636 4303000 BA 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 63.481636 63.481636
4 2008-01-02 72.559998 72.669998 70.050003 47.633060 6337800 CAT 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 47.633060 47.633060
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
96343 2021-08-31 158.850006 160.800003 158.710007 158.824631 1568700 TRV 1 2.301368 163.977861 149.229152 56.038937 73.949247 27.768804 153.781405 152.358006
96344 2021-08-31 417.000000 420.739990 414.220001 414.813965 4249100 UNH 1 1.795430 428.868267 402.859321 52.549565 11.116244 9.546535 415.015272 409.123889
96345 2021-08-31 54.709999 55.139999 54.580002 54.354485 18405600 VZ 1 -0.250367 55.499877 53.860358 44.653841 -120.717132 26.176223 54.859158 55.087388
96346 2021-08-31 48.459999 50.869999 48.389999 50.750000 11814400 WBA 1 0.388522 50.322940 46.070231 56.376587 155.229924 20.132924 47.624491 48.772953
96347 2021-08-31 147.990005 148.440002 147.240005 148.100006 8359500 WMT 1 1.379987 152.957874 143.371276 57.798913 35.525528 14.285037 146.036804 142.410712

96348 rows × 16 columns

df.index = df.date.factorize()[0] #. now each new date has a new index
df
date open high low close volume tic day macd boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma
0 2008-01-02 7.116786 7.152143 6.876786 5.966036 1079178800 AAPL 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 5.966036 5.966036
0 2008-01-02 46.599998 47.040001 46.259998 36.337456 7934400 AMGN 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 36.337456 36.337456
0 2008-01-02 52.090000 52.320000 50.790001 40.694561 8053700 AXP 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 40.694561 40.694561
0 2008-01-02 87.570000 87.839996 86.000000 63.481636 4303000 BA 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 63.481636 63.481636
0 2008-01-02 72.559998 72.669998 70.050003 47.633060 6337800 CAT 2 0.000000 5.971313 5.963515 100.000000 -66.666667 100.000000 47.633060 47.633060
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
3440 2021-08-31 158.850006 160.800003 158.710007 158.824631 1568700 TRV 1 2.301368 163.977861 149.229152 56.038937 73.949247 27.768804 153.781405 152.358006
3440 2021-08-31 417.000000 420.739990 414.220001 414.813965 4249100 UNH 1 1.795430 428.868267 402.859321 52.549565 11.116244 9.546535 415.015272 409.123889
3440 2021-08-31 54.709999 55.139999 54.580002 54.354485 18405600 VZ 1 -0.250367 55.499877 53.860358 44.653841 -120.717132 26.176223 54.859158 55.087388
3440 2021-08-31 48.459999 50.869999 48.389999 50.750000 11814400 WBA 1 0.388522 50.322940 46.070231 56.376587 155.229924 20.132924 47.624491 48.772953
3440 2021-08-31 147.990005 148.440002 147.240005 148.100006 8359500 WMT 1 1.379987 152.957874 143.371276 57.798913 35.525528 14.285037 146.036804 142.410712

96348 rows × 16 columns

# hide
# len(df.index.unique())
# range(lookback, len(df.index.unique()))
lst = [i for i in range(LOOKBACK, len(df.index.unique()))]
lst[:20], lst[-1]
([252,
  253,
  254,
  255,
  256,
  257,
  258,
  259,
  260,
  261,
  262,
  263,
  264,
  265,
  266,
  267,
  268,
  269,
  270,
  271],
 3440)
%%time
cov_list = []
return_list = []
for i in range(LOOKBACK, len(df.index.unique())):
  data_lookback = df.loc[i-LOOKBACK:i, :]
  price_lookback = data_lookback.pivot_table(index='date', columns='tic', values='close')
  return_lookback = price_lookback.pct_change().dropna()
  return_list.append(return_lookback)
  covs = return_lookback.cov().values 
  cov_list.append(covs)
CPU times: user 1min 24s, sys: 1min 10s, total: 2min 35s
Wall time: 1min 19s
len(df['date'].unique()), len(df['date'].unique()[LOOKBACK:])
(3441, 3189)
len(cov_list), len(return_list)
(3189, 3189)
# hide
# cov_list[0]
# return_list[:1]
# 
# form a dataframe with the cov_list and return_list
df_cov = pd.DataFrame({'date':df.date.unique()[LOOKBACK:], 'cov_list':cov_list, 'return_list':return_list})
df_cov
date cov_list return_list
0 2008-12-31 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
1 2009-01-02 [[0.001366149596033732, 0.0004339373862598414,... tic AAPL AMGN AXP ... ...
2 2009-01-05 [[0.001352020512391899, 0.00042947012279806696... tic AAPL AMGN AXP ... ...
3 2009-01-06 [[0.0013523440717467432, 0.0004313704561627888... tic AAPL AMGN AXP ... ...
4 2009-01-07 [[0.001349261443536303, 0.00043429869784408895... tic AAPL AMGN AXP ... ...
... ... ... ...
3184 2021-08-25 [[0.00039332760434043605, 9.936966754755532e-0... tic AAPL AMGN AXP ... ...
3185 2021-08-26 [[0.00039284508209810654, 9.896441517027707e-0... tic AAPL AMGN AXP ... ...
3186 2021-08-27 [[0.000392348741590828, 9.967092175224526e-05,... tic AAPL AMGN AXP ... ...
3187 2021-08-30 [[0.0003957763867577117, 0.0001004288872067923... tic AAPL AMGN AXP ... ...
3188 2021-08-31 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...

3189 rows × 3 columns

# 
# merge df_cov with the main dataframe
df = df.merge(df_cov, on='date')
df
date open high low close volume tic day macd boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma cov_list return_list
0 2008-12-31 3.070357 3.133571 3.047857 2.613433 607541200 AAPL 2 -0.083547 3.128999 2.482334 42.254773 -80.488341 16.129793 2.780976 2.894368 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
1 2008-12-31 57.110001 58.220001 57.060001 45.031933 6287200 AMGN 2 0.168718 45.965630 43.970341 51.060625 51.616195 10.432018 44.190812 43.701893 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
2 2008-12-31 17.969999 18.750000 17.910000 14.988309 9625600 AXP 2 -0.961734 19.168083 13.014357 42.554857 -75.368777 25.776759 16.184140 18.108784 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
3 2008-12-31 41.590000 43.049999 41.500000 32.005886 5443100 BA 2 -0.279799 32.174385 28.867829 47.440239 156.994713 5.366299 30.327210 32.389913 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
4 2008-12-31 43.700001 45.099998 43.700001 30.925045 6277400 CAT 2 0.684760 31.697356 26.587391 51.205312 98.425681 26.331746 27.876156 27.598373 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
89287 2021-08-31 158.850006 160.800003 158.710007 158.824631 1568700 TRV 1 2.301368 163.977861 149.229152 56.038937 73.949247 27.768804 153.781405 152.358006 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89288 2021-08-31 417.000000 420.739990 414.220001 414.813965 4249100 UNH 1 1.795430 428.868267 402.859321 52.549565 11.116244 9.546535 415.015272 409.123889 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89289 2021-08-31 54.709999 55.139999 54.580002 54.354485 18405600 VZ 1 -0.250367 55.499877 53.860358 44.653841 -120.717132 26.176223 54.859158 55.087388 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89290 2021-08-31 48.459999 50.869999 48.389999 50.750000 11814400 WBA 1 0.388522 50.322940 46.070231 56.376587 155.229924 20.132924 47.624491 48.772953 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89291 2021-08-31 147.990005 148.440002 147.240005 148.100006 8359500 WMT 1 1.379987 152.957874 143.371276 57.798913 35.525528 14.285037 146.036804 142.410712 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...

89292 rows × 18 columns

df = df.sort_values(['date','tic']).reset_index(drop=True)
df
date open high low close volume tic day macd boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma cov_list return_list
0 2008-12-31 3.070357 3.133571 3.047857 2.613433 607541200 AAPL 2 -0.083547 3.128999 2.482334 42.254773 -80.488341 16.129793 2.780976 2.894368 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
1 2008-12-31 57.110001 58.220001 57.060001 45.031933 6287200 AMGN 2 0.168718 45.965630 43.970341 51.060625 51.616195 10.432018 44.190812 43.701893 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
2 2008-12-31 17.969999 18.750000 17.910000 14.988309 9625600 AXP 2 -0.961734 19.168083 13.014357 42.554857 -75.368777 25.776759 16.184140 18.108784 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
3 2008-12-31 41.590000 43.049999 41.500000 32.005886 5443100 BA 2 -0.279799 32.174385 28.867829 47.440239 156.994713 5.366299 30.327210 32.389913 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
4 2008-12-31 43.700001 45.099998 43.700001 30.925045 6277400 CAT 2 0.684760 31.697356 26.587391 51.205312 98.425681 26.331746 27.876156 27.598373 [[0.0013489679031036903, 0.0004284118293704584... tic AAPL AMGN AXP ... ...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
89287 2021-08-31 158.850006 160.800003 158.710007 158.824631 1568700 TRV 1 2.301368 163.977861 149.229152 56.038937 73.949247 27.768804 153.781405 152.358006 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89288 2021-08-31 417.000000 420.739990 414.220001 414.813965 4249100 UNH 1 1.795430 428.868267 402.859321 52.549565 11.116244 9.546535 415.015272 409.123889 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89289 2021-08-31 54.709999 55.139999 54.580002 54.354485 18405600 VZ 1 -0.250367 55.499877 53.860358 44.653841 -120.717132 26.176223 54.859158 55.087388 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89290 2021-08-31 48.459999 50.869999 48.389999 50.750000 11814400 WBA 1 0.388522 50.322940 46.070231 56.376587 155.229924 20.132924 47.624491 48.772953 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...
89291 2021-08-31 147.990005 148.440002 147.240005 148.100006 8359500 WMT 1 1.379987 152.957874 143.371276 57.798913 35.525528 14.285037 146.036804 142.410712 [[0.0003917981292589402, 0.0001000497739447042... tic AAPL AMGN AXP ... ...

89292 rows × 18 columns

3.3 Modeling

The portfolio within the market will be modeled by the OpenAI Gym framework. This is referred to as the environment. For the agent, we will use the FinRL framework. As the agent interacts with the environment it will gradually learn a trading strategy based on the reward function. The agent is rewarded according to the total value of the portfolio.

3.3.1 Training data

TRAIN_START, TRADE_START
('2009-01-01', '2020-07-01')
train = data_split(df, TRAIN_START, TRADE_START)
train
date open high low close volume tic day macd boll_ub boll_lb rsi_30 cci_30 dx_30 close_30_sma close_60_sma cov_list return_list
0 2009-01-02 3.067143 3.251429 3.041429 2.778782 746015200 AAPL 4 -0.070954 3.115322 2.480241 45.440198 -32.098530 2.140064 2.781834 2.895179 [[0.001366149596033732, 0.0004339373862598414,... tic AAPL AMGN AXP ... ...
0 2009-01-02 58.590000 59.080002 57.750000 45.998852 6547900 AMGN 4 0.249877 46.122334 43.932164 52.756868 93.201935 0.814217 44.259693 43.796766 [[0.001366149596033732, 0.0004339373862598414,... tic AAPL AMGN AXP ... ...
0 2009-01-02 18.570000 19.520000 18.400000 15.618546 10955700 AXP 4 -0.855306 18.979336 12.997874 43.957565 -42.761255 16.335101 16.182793 17.988662 [[0.001366149596033732, 0.0004339373862598414,... tic AAPL AMGN AXP ... ...
0 2009-01-02 42.799999 45.560001 42.779999 33.941090 7010200 BA 4 -0.002008 32.948626 28.452126 50.822027 272.812626 20.494464 30.469475 32.344128 [[0.001366149596033732, 0.0004339373862598414,... tic AAPL AMGN AXP ... ...
0 2009-01-02 44.910000 46.980000 44.709999 32.475796 7117200 CAT 4 0.870226 32.221776 26.565580 53.661249 129.733630 34.637448 28.123538 27.598980 [[0.001366149596033732, 0.0004339373862598414,... tic AAPL AMGN AXP ... ...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2892 2020-06-30 112.470001 114.489998 112.099998 110.630806 1464500 TRV 1 1.826300 121.687411 104.149844 52.925506 19.009457 1.892194 108.457671 102.220519 [[0.0006523831425009409, 0.0004061943197087559... tic AAPL AMGN AXP ... ...
2892 2020-06-30 288.570007 296.450012 287.660004 289.569763 2932900 UNH 1 -0.019600 305.819470 272.941248 52.413046 -25.789704 1.846804 289.815174 282.752193 [[0.0006523831425009409, 0.0004061943197087559... tic AAPL AMGN AXP ... ...
2892 2020-06-30 54.919998 55.290001 54.360001 51.604496 17414800 VZ 1 -0.447764 55.232498 49.916928 48.097035 -50.926677 8.508886 52.255363 52.718949 [[0.0006523831425009409, 0.0004061943197087559... tic AAPL AMGN AXP ... ...
2892 2020-06-30 42.119999 42.580002 41.759998 40.278057 4782100 WBA 1 -0.086659 43.965360 37.648307 48.830183 -14.417188 1.500723 40.380679 40.174252 [[0.0006523831425009409, 0.0004061943197087559... tic AAPL AMGN AXP ... ...
2892 2020-06-30 119.220001 120.129997 118.540001 117.465683 6836400 WMT 1 -0.896829 120.856465 114.824143 48.159675 -69.901687 3.847271 119.150817 121.108866 [[0.0006523831425009409, 0.0004061943197087559... tic AAPL AMGN AXP ... ...

81004 rows × 18 columns

3.3.2 Portfolio (Environment)

class Portfolio(gym.Env):
    """A portfolio/market environment
    Attributes
    ----------
        df: DataFrame
            input data
        stock_dim : int
            number of unique stocks
        hmax : int
            maximum number of shares to trade
        initial_amount : int
            start money
        transaction_cost_pct: float
            transaction cost percentage per trade
        reward_scaling: float
            scaling factor for reward, good for training
        state_space: int
            the dimension of input features
        action_space: int
            equals stock dimension
        tech_indicator_list: list
            a list of technical indicator names
        turbulence_threshold: int
            a threshold to control risk aversion
        day: int
            an increment number to control date
    Methods
    -------
    _sell_stock()
        perform sell action based on the sign of the action
    _buy_stock()
        perform buy action based on the sign of the action
    step()
        at each step the agent will return actions, then 
        we will calculate the reward, and return the next observation.
    reset()
        reset the environment
    render()
        use render to return other functions
    save_asset_memory()
        return account value at each time step
    save_action_memory()
        return actions/positions at each time step
    """
    metadata = {'render.modes': ['human']}

    def __init__(self, 
                df,
                stock_dim,
                hmax,
                initial_amount,
                transaction_cost_pct,
                reward_scaling,
                state_space,
                action_space,
                tech_indicator_list,
                turbulence_threshold=None,
                lookback=LOOKBACK,
                day=0):
        #super(StockEnv, self).__init__()
        #money = 10 , scope = 1
        self.day = day
        self.lookback = lookback
        self.df = df
        self.stock_dim = stock_dim
        self.hmax = hmax
        self.initial_amount = initial_amount
        self.transaction_cost_pct = transaction_cost_pct
        self.reward_scaling = reward_scaling
        self.state_space = state_space
        self.action_space = action_space
        self.tech_indicator_list = tech_indicator_list

        # action_space normalization and shape is self.stock_dim
        self.action_space = spaces.Box(low=0, high=1, shape=(self.action_space,)) 
        self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(self.state_space + len(self.tech_indicator_list), self.state_space))

        # load data from a pandas dataframe
        self.data = self.df.loc[self.day, :]
        self.covs = self.data['cov_list'].values[0]
        self.state =  np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0)
        self.terminal = False
        self.turbulence_threshold = turbulence_threshold
        # initalize state: inital portfolio return + individual stock return + individual weights
        self.portfolio_value = self.initial_amount

        # memorize portfolio value each step
        self.asset_memory = [self.initial_amount]
        # memorize portfolio return each step
        self.portfolio_return_memory = [0]
        self.actions_memory = [[1/self.stock_dim]*self.stock_dim]
        self.date_memory = [self.data.date.unique()[0]]

    def step(self, actions):
        self.terminal = self.day >= len(self.df.index.unique()) - 1

        if self.terminal:
            df = pd.DataFrame(self.portfolio_return_memory)
            df.columns = ['daily_return']
            plt.plot(df.daily_return.cumsum(), 'r')
            plt.savefig('results/cumulative_reward.png')
            plt.close()
            
            plt.plot(self.portfolio_return_memory, 'r')
            plt.savefig('results/rewards.png')
            plt.close()

            print("=================================")
            print("begin_total_asset:{}".format(self.asset_memory[0]))           
            print("end_total_asset:{}".format(self.portfolio_value))

            df_daily_return = pd.DataFrame(self.portfolio_return_memory)
            df_daily_return.columns = ['daily_return']
            if df_daily_return['daily_return'].std() !=0:
              sharpe = (252**0.5)*df_daily_return['daily_return'].mean()/ \
                       df_daily_return['daily_return'].std()
              print("Sharpe: ",sharpe)
            print("=================================")
            
            return self.state, self.reward, self.terminal, {}
        else:
            weights = self.softmax_normalization(actions) 
            self.actions_memory.append(weights)
            last_day_memory = self.data

            #load next state
            self.day += 1
            self.data = self.df.loc[self.day,:]
            self.covs = self.data['cov_list'].values[0]
            self.state =  np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0)
            portfolio_return = sum(((self.data.close.values / last_day_memory.close.values)-1)*weights)
            log_portfolio_return = np.log(sum((self.data.close.values / last_day_memory.close.values)*weights))
            # update portfolio value
            new_portfolio_value = self.portfolio_value*(1+portfolio_return)
            self.portfolio_value = new_portfolio_value

            # save into memory
            self.portfolio_return_memory.append(portfolio_return)
            self.date_memory.append(self.data.date.unique()[0])            
            self.asset_memory.append(new_portfolio_value)

            # the reward is the new portfolio value or end portfolo value
            self.reward = new_portfolio_value
        return self.state, self.reward, self.terminal, {}

    def reset(self):
        self.asset_memory = [self.initial_amount]
        self.day = 0
        self.data = self.df.loc[self.day,:]
        # load states
        self.covs = self.data['cov_list'].values[0]
        self.state =  np.append(np.array(self.covs), [self.data[tech].values.tolist() for tech in self.tech_indicator_list ], axis=0)
        self.portfolio_value = self.initial_amount
        #self.cost = 0
        #self.trades = 0
        self.terminal = False 
        self.portfolio_return_memory = [0]
        self.actions_memory=[[1/self.stock_dim]*self.stock_dim]
        self.date_memory=[self.data.date.unique()[0]] 
        return self.state
    
    def render(self, mode='human'):
        return self.state
        
    def softmax_normalization(self, actions):
        numerator = np.exp(actions)
        denominator = np.sum(np.exp(actions))
        softmax_output = numerator/denominator
        return softmax_output

    def save_asset_memory(self):
        date_list = self.date_memory
        portfolio_return = self.portfolio_return_memory
        #print(len(date_list))
        #print(len(asset_list))
        df_account_value = pd.DataFrame({'date':date_list,'daily_return':portfolio_return})
        return df_account_value

    def save_action_memory(self):
        # date and close price length must match actions length
        date_list = self.date_memory
        df_date = pd.DataFrame(date_list)
        df_date.columns = ['date']
        
        action_list = self.actions_memory
        df_actions = pd.DataFrame(action_list)
        df_actions.columns = self.data.tic.values
        df_actions.index = df_date.date
        #df_actions = pd.DataFrame({'date':date_list,'actions':action_list})
        return df_actions

    def _seed(self, seed=None):
        self.np_random, seed = seeding.np_random(seed)
        return [seed]

    def get_sb_env(self):
        e = DummyVecEnv([lambda: self])
        obs = e.reset()
        return e, obs
# hide
#. was 29 in original notebook !
stock_dimension = len(train['tic'].unique())
state_space = stock_dimension
print(f"Stock Dimension: {stock_dimension}, State Space: {state_space}")
Stock Dimension: 28, State Space: 28
# hide
config.TECHNICAL_INDICATORS_LIST
['macd',
 'boll_ub',
 'boll_lb',
 'rsi_30',
 'cci_30',
 'dx_30',
 'close_30_sma',
 'close_60_sma']
env_kwargs = {
    "hmax": 100, 
    "initial_amount": INITIAL_AMOUNT, 
    "transaction_cost_pct": TRANSACTION_COST_PCT, 
    "state_space": state_space, 
    "stock_dim": stock_dimension, 
    "tech_indicator_list": config.TECHNICAL_INDICATORS_LIST, 
    "action_space": stock_dimension, 
    "reward_scaling": REWARD_SCALING,
}
e_train_gym = Portfolio(df=train, **env_kwargs)
env_train, _ = e_train_gym.get_sb_env()
print(type(env_train))
<class 'stable_baselines3.common.vec_env.dummy_vec_env.DummyVecEnv'>

3.3.3 Portfolio Manager (Agent)

  • We investigate the performance of two models for the agent:

    • A2C
    • PPO

Both models are based on algorithm implementations in the OpenAI Baselines and Stable Baselines libraries.

# hide
# https://towardsdatascience.com/finrl-for-quantitative-finance-tutorial-for-portfolio-allocation-9b417660c7cd
# hide
# agent = DRLAgent(env=env_train)
# DDPG_PARAMS = {
#   # "n_steps": 10, 
#   # "ent_coef": 0.005, 
#   "learning_rate": 0.0004,
# }
# model_ddpg = agent.get_model(model_name="ddpg", model_kwargs=DDPG_PARAMS)
# model_ddpg
# hide
# %%time
# trained_ddpg = agent.train_model(model=model_ddpg, tb_log_name='ddpg', total_timesteps=1_000)
# hide
# agent = DRLAgent(env=env_train)
# SAC_PARAMS = {
#   # "n_steps": 10, 
#   # "ent_coef": 0.005, 
#   "learning_rate": 0.0004,
# }
# model_sac = agent.get_model(model_name="sac", model_kwargs=SAC_PARAMS)
# model_sac
# hide
# %%time
# trained_sac = agent.train_model(model=model_sac, tb_log_name='sac', total_timesteps=1_000)
# hide
# agent = DRLAgent(env=env_train)
# TD3_PARAMS = {
#   # "n_steps": 10, 
#   # "ent_coef": 0.005, 
#   "learning_rate": 0.0004,
# }
# model_td3 = agent.get_model(model_name="td3", model_kwargs=TD3_PARAMS)
# model_td3
# hide
# %%time
# trained_sac = agent.train_model(model=model_sac, tb_log_name='sac', total_timesteps=1_000)
# hide
# agent = DRLAgent(env=env_train)
# MADDDPG_PARAMS = {
#   # "n_steps": 10, 
#   # "ent_coef": 0.005, 
#   "learning_rate": 0.0004,
# }
# model_madddpg = agent.get_model(model_name="madddpg", model_kwargs=MADDDPG_PARAMS)
# model_madddpg
# hide
# %%time
# trained_madddpg = agent.train_model(model=model_madddpg, tb_log_name='madddpg', total_timesteps=1_000)
3.3.3.1 A2C
agent = DRLAgent(env=env_train)
A2C_PARAMS = {
  "n_steps": 10, 
  "ent_coef": 0.005, 
  "learning_rate": 0.0004,
}
model_a2c = agent.get_model(model_name="a2c", model_kwargs=A2C_PARAMS)
model_a2c
{'n_steps': 10, 'ent_coef': 0.005, 'learning_rate': 0.0004}
Using cuda device
<stable_baselines3.a2c.a2c.A2C at 0x7fa13b221d50>
%%time
trained_a2c = agent.train_model(model=model_a2c, tb_log_name='a2c', total_timesteps=50_000)
Logging to tensorboard_log/a2c/a2c_2
-------------------------------------
| time/                 |           |
|    fps                | 225       |
|    iterations         | 100       |
|    time_elapsed       | 4         |
|    total_timesteps    | 1000      |
| train/                |           |
|    entropy_loss       | -39.6     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 99        |
|    policy_loss        | 4.6e+08   |
|    reward             | 2207526.0 |
|    std                | 0.995     |
|    value_loss         | 1.69e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 227       |
|    iterations         | 200       |
|    time_elapsed       | 8         |
|    total_timesteps    | 2000      |
| train/                |           |
|    entropy_loss       | -39.6     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 199       |
|    policy_loss        | 8.55e+08  |
|    reward             | 4078324.2 |
|    std                | 0.994     |
|    value_loss         | 5.58e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6349662.786137318
Sharpe:  0.9646091662004085
=================================
-------------------------------------
| time/                 |           |
|    fps                | 224       |
|    iterations         | 300       |
|    time_elapsed       | 13        |
|    total_timesteps    | 3000      |
| train/                |           |
|    entropy_loss       | -39.6     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 299       |
|    policy_loss        | 2e+08     |
|    reward             | 1102288.1 |
|    std                | 0.994     |
|    value_loss         | 3.91e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 224       |
|    iterations         | 400       |
|    time_elapsed       | 17        |
|    total_timesteps    | 4000      |
| train/                |           |
|    entropy_loss       | -39.5     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 399       |
|    policy_loss        | 4.79e+08  |
|    reward             | 2496023.2 |
|    std                | 0.993     |
|    value_loss         | 2.16e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 225       |
|    iterations         | 500       |
|    time_elapsed       | 22        |
|    total_timesteps    | 5000      |
| train/                |           |
|    entropy_loss       | -39.5     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 499       |
|    policy_loss        | 8.02e+08  |
|    reward             | 3879689.0 |
|    std                | 0.991     |
|    value_loss         | 5.57e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5735779.5957364235
Sharpe:  0.9206384172273598
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 600       |
|    time_elapsed       | 26        |
|    total_timesteps    | 6000      |
| train/                |           |
|    entropy_loss       | -39.5     |
|    explained_variance | 5.96e-08  |
|    learning_rate      | 0.0004    |
|    n_updates          | 599       |
|    policy_loss        | 2.81e+08  |
|    reward             | 1340301.0 |
|    std                | 0.991     |
|    value_loss         | 6.44e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 223       |
|    iterations         | 700       |
|    time_elapsed       | 31        |
|    total_timesteps    | 7000      |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 699       |
|    policy_loss        | 5.18e+08  |
|    reward             | 2603395.5 |
|    std                | 0.99      |
|    value_loss         | 2.23e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 224       |
|    iterations         | 800       |
|    time_elapsed       | 35        |
|    total_timesteps    | 8000      |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 799       |
|    policy_loss        | 9.09e+08  |
|    reward             | 4535535.0 |
|    std                | 0.987     |
|    value_loss         | 7.03e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6065084.0163611695
Sharpe:  0.9408937296693319
=================================
-------------------------------------
| time/                 |           |
|    fps                | 223       |
|    iterations         | 900       |
|    time_elapsed       | 40        |
|    total_timesteps    | 9000      |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 899       |
|    policy_loss        | 3.08e+08  |
|    reward             | 1510501.5 |
|    std                | 0.988     |
|    value_loss         | 7.75e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 223       |
|    iterations         | 1000      |
|    time_elapsed       | 44        |
|    total_timesteps    | 10000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 999       |
|    policy_loss        | 6.13e+08  |
|    reward             | 2961423.2 |
|    std                | 0.988     |
|    value_loss         | 3.05e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 223       |
|    iterations         | 1100      |
|    time_elapsed       | 49        |
|    total_timesteps    | 11000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 1099      |
|    policy_loss        | 1.15e+09  |
|    reward             | 5204864.5 |
|    std                | 0.989     |
|    value_loss         | 1.08e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6632797.506983363
Sharpe:  0.9879877787373227
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1200      |
|    time_elapsed       | 53        |
|    total_timesteps    | 12000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 1199      |
|    policy_loss        | 2.77e+08  |
|    reward             | 1437981.6 |
|    std                | 0.987     |
|    value_loss         | 6.57e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1300      |
|    time_elapsed       | 58        |
|    total_timesteps    | 13000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 1299      |
|    policy_loss        | 5.96e+08  |
|    reward             | 2920604.0 |
|    std                | 0.988     |
|    value_loss         | 2.87e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1400      |
|    time_elapsed       | 62        |
|    total_timesteps    | 14000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 1399      |
|    policy_loss        | 1.09e+09  |
|    reward             | 5123196.5 |
|    std                | 0.988     |
|    value_loss         | 9.15e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5565781.931934857
Sharpe:  0.9071288326984139
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1500      |
|    time_elapsed       | 67        |
|    total_timesteps    | 15000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 1499      |
|    policy_loss        | 3.63e+08  |
|    reward             | 1731057.9 |
|    std                | 0.989     |
|    value_loss         | 1.03e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1600      |
|    time_elapsed       | 71        |
|    total_timesteps    | 16000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 1599      |
|    policy_loss        | 6.57e+08  |
|    reward             | 3219108.5 |
|    std                | 0.988     |
|    value_loss         | 3.75e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1700      |
|    time_elapsed       | 76        |
|    total_timesteps    | 17000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 1699      |
|    policy_loss        | 1.07e+09  |
|    reward             | 5349077.5 |
|    std                | 0.988     |
|    value_loss         | 9.44e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5975580.981813196
Sharpe:  0.9408061937036222
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1800      |
|    time_elapsed       | 80        |
|    total_timesteps    | 18000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 1799      |
|    policy_loss        | 3.64e+08  |
|    reward             | 1759525.8 |
|    std                | 0.987     |
|    value_loss         | 1.07e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 1900      |
|    time_elapsed       | 85        |
|    total_timesteps    | 19000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 1899      |
|    policy_loss        | 7.44e+08  |
|    reward             | 3488812.2 |
|    std                | 0.988     |
|    value_loss         | 4.28e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2000      |
|    time_elapsed       | 89        |
|    total_timesteps    | 20000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 1999      |
|    policy_loss        | 1.2e+09   |
|    reward             | 5854436.5 |
|    std                | 0.987     |
|    value_loss         | 1.15e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5753434.527233439
Sharpe:  0.9234199802759206
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2100      |
|    time_elapsed       | 94        |
|    total_timesteps    | 21000     |
| train/                |           |
|    entropy_loss       | -39.4     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 2099      |
|    policy_loss        | 3.63e+08  |
|    reward             | 1700565.9 |
|    std                | 0.987     |
|    value_loss         | 1.03e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2200      |
|    time_elapsed       | 98        |
|    total_timesteps    | 22000     |
| train/                |           |
|    entropy_loss       | -39.3     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 2199      |
|    policy_loss        | 6.94e+08  |
|    reward             | 3262350.8 |
|    std                | 0.986     |
|    value_loss         | 3.95e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2300      |
|    time_elapsed       | 103       |
|    total_timesteps    | 23000     |
| train/                |           |
|    entropy_loss       | -39.3     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 2299      |
|    policy_loss        | 1.22e+09  |
|    reward             | 5957404.0 |
|    std                | 0.986     |
|    value_loss         | 1.29e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5922463.637790388
Sharpe:  0.9372661191751825
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2400      |
|    time_elapsed       | 107       |
|    total_timesteps    | 24000     |
| train/                |           |
|    entropy_loss       | -39.3     |
|    explained_variance | 5.96e-08  |
|    learning_rate      | 0.0004    |
|    n_updates          | 2399      |
|    policy_loss        | 4.34e+08  |
|    reward             | 1917851.5 |
|    std                | 0.985     |
|    value_loss         | 1.43e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2500      |
|    time_elapsed       | 112       |
|    total_timesteps    | 25000     |
| train/                |           |
|    entropy_loss       | -39.2     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 2499      |
|    policy_loss        | 7.54e+08  |
|    reward             | 3441977.2 |
|    std                | 0.983     |
|    value_loss         | 4.45e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 223       |
|    iterations         | 2600      |
|    time_elapsed       | 116       |
|    total_timesteps    | 26000     |
| train/                |           |
|    entropy_loss       | -39.2     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 2599      |
|    policy_loss        | 1.23e+09  |
|    reward             | 5727213.5 |
|    std                | 0.982     |
|    value_loss         | 1.12e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5957219.06392782
Sharpe:  0.943274677798875
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2700      |
|    time_elapsed       | 121       |
|    total_timesteps    | 27000     |
| train/                |           |
|    entropy_loss       | -39.2     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 2699      |
|    policy_loss        | 4.51e+08  |
|    reward             | 2066869.2 |
|    std                | 0.981     |
|    value_loss         | 1.6e+14   |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2800      |
|    time_elapsed       | 125       |
|    total_timesteps    | 28000     |
| train/                |           |
|    entropy_loss       | -39.2     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 2799      |
|    policy_loss        | 7.63e+08  |
|    reward             | 3609586.8 |
|    std                | 0.98      |
|    value_loss         | 4.81e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5797086.842523102
Sharpe:  0.9274624882867446
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 2900      |
|    time_elapsed       | 130       |
|    total_timesteps    | 29000     |
| train/                |           |
|    entropy_loss       | -39.2     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 2899      |
|    policy_loss        | 2.01e+08  |
|    reward             | 976247.06 |
|    std                | 0.98      |
|    value_loss         | 2.99e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3000      |
|    time_elapsed       | 134       |
|    total_timesteps    | 30000     |
| train/                |           |
|    entropy_loss       | -39.1     |
|    explained_variance | -2.38e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 2999      |
|    policy_loss        | 4.52e+08  |
|    reward             | 2214999.5 |
|    std                | 0.978     |
|    value_loss         | 1.73e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 223       |
|    iterations         | 3100      |
|    time_elapsed       | 138       |
|    total_timesteps    | 31000     |
| train/                |           |
|    entropy_loss       | -39.1     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 3099      |
|    policy_loss        | 8.31e+08  |
|    reward             | 3969269.5 |
|    std                | 0.978     |
|    value_loss         | 5.84e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5631683.243449584
Sharpe:  0.9044595005152428
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3200      |
|    time_elapsed       | 143       |
|    total_timesteps    | 32000     |
| train/                |           |
|    entropy_loss       | -39.1     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 3199      |
|    policy_loss        | 2.63e+08  |
|    reward             | 1308491.2 |
|    std                | 0.977     |
|    value_loss         | 5.64e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3300      |
|    time_elapsed       | 148       |
|    total_timesteps    | 33000     |
| train/                |           |
|    entropy_loss       | -39.1     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 3299      |
|    policy_loss        | 5.3e+08   |
|    reward             | 2536331.2 |
|    std                | 0.977     |
|    value_loss         | 2.34e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3400      |
|    time_elapsed       | 152       |
|    total_timesteps    | 34000     |
| train/                |           |
|    entropy_loss       | -39       |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 3399      |
|    policy_loss        | 9.25e+08  |
|    reward             | 4471689.0 |
|    std                | 0.976     |
|    value_loss         | 7.31e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6170278.00294478
Sharpe:  0.9501501352416521
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3500      |
|    time_elapsed       | 157       |
|    total_timesteps    | 35000     |
| train/                |           |
|    entropy_loss       | -39       |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 3499      |
|    policy_loss        | 2.75e+08  |
|    reward             | 1398594.6 |
|    std                | 0.975     |
|    value_loss         | 6.67e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3600      |
|    time_elapsed       | 161       |
|    total_timesteps    | 36000     |
| train/                |           |
|    entropy_loss       | -39       |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 3599      |
|    policy_loss        | 5.7e+08   |
|    reward             | 2741481.8 |
|    std                | 0.974     |
|    value_loss         | 2.77e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3700      |
|    time_elapsed       | 166       |
|    total_timesteps    | 37000     |
| train/                |           |
|    entropy_loss       | -39       |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 3699      |
|    policy_loss        | 1.04e+09  |
|    reward             | 5317960.0 |
|    std                | 0.973     |
|    value_loss         | 9.53e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6114847.089741801
Sharpe:  0.9499645625356147
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3800      |
|    time_elapsed       | 170       |
|    total_timesteps    | 38000     |
| train/                |           |
|    entropy_loss       | -38.9     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 3799      |
|    policy_loss        | 2.75e+08  |
|    reward             | 1391995.9 |
|    std                | 0.973     |
|    value_loss         | 6.2e+13   |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 3900      |
|    time_elapsed       | 175       |
|    total_timesteps    | 39000     |
| train/                |           |
|    entropy_loss       | -38.9     |
|    explained_variance | 5.96e-08  |
|    learning_rate      | 0.0004    |
|    n_updates          | 3899      |
|    policy_loss        | 5.74e+08  |
|    reward             | 2949128.0 |
|    std                | 0.972     |
|    value_loss         | 3.06e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4000      |
|    time_elapsed       | 179       |
|    total_timesteps    | 40000     |
| train/                |           |
|    entropy_loss       | -38.9     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 3999      |
|    policy_loss        | 1.08e+09  |
|    reward             | 4953613.5 |
|    std                | 0.97      |
|    value_loss         | 9.38e+14  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6140003.597467672
Sharpe:  0.9495630019023644
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4100      |
|    time_elapsed       | 184       |
|    total_timesteps    | 41000     |
| train/                |           |
|    entropy_loss       | -38.9     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4099      |
|    policy_loss        | 3.08e+08  |
|    reward             | 1652238.6 |
|    std                | 0.969     |
|    value_loss         | 9.38e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4200      |
|    time_elapsed       | 188       |
|    total_timesteps    | 42000     |
| train/                |           |
|    entropy_loss       | -38.9     |
|    explained_variance | 1.19e-07  |
|    learning_rate      | 0.0004    |
|    n_updates          | 4199      |
|    policy_loss        | 6.67e+08  |
|    reward             | 3193442.5 |
|    std                | 0.97      |
|    value_loss         | 3.92e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4300      |
|    time_elapsed       | 192       |
|    total_timesteps    | 43000     |
| train/                |           |
|    entropy_loss       | -38.8     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4299      |
|    policy_loss        | 1.2e+09   |
|    reward             | 5570221.0 |
|    std                | 0.969     |
|    value_loss         | 1.12e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5994759.818762383
Sharpe:  0.9343812285364731
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4400      |
|    time_elapsed       | 197       |
|    total_timesteps    | 44000     |
| train/                |           |
|    entropy_loss       | -38.8     |
|    explained_variance | -1.19e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 4399      |
|    policy_loss        | 3.66e+08  |
|    reward             | 1744263.1 |
|    std                | 0.968     |
|    value_loss         | 1.12e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4500      |
|    time_elapsed       | 201       |
|    total_timesteps    | 45000     |
| train/                |           |
|    entropy_loss       | -38.8     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4499      |
|    policy_loss        | 7.06e+08  |
|    reward             | 3390206.2 |
|    std                | 0.968     |
|    value_loss         | 4.01e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4600      |
|    time_elapsed       | 206       |
|    total_timesteps    | 46000     |
| train/                |           |
|    entropy_loss       | -38.8     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4599      |
|    policy_loss        | 1.12e+09  |
|    reward             | 5508010.0 |
|    std                | 0.967     |
|    value_loss         | 1.15e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5825032.88963333
Sharpe:  0.9207004457658553
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4700      |
|    time_elapsed       | 211       |
|    total_timesteps    | 47000     |
| train/                |           |
|    entropy_loss       | -38.8     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4699      |
|    policy_loss        | 3.31e+08  |
|    reward             | 1754341.0 |
|    std                | 0.966     |
|    value_loss         | 9.28e+13  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4800      |
|    time_elapsed       | 215       |
|    total_timesteps    | 48000     |
| train/                |           |
|    entropy_loss       | -38.8     |
|    explained_variance | -2.38e-07 |
|    learning_rate      | 0.0004    |
|    n_updates          | 4799      |
|    policy_loss        | 6.75e+08  |
|    reward             | 3174122.0 |
|    std                | 0.966     |
|    value_loss         | 3.39e+14  |
-------------------------------------
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 4900      |
|    time_elapsed       | 219       |
|    total_timesteps    | 49000     |
| train/                |           |
|    entropy_loss       | -38.7     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4899      |
|    policy_loss        | 1.2e+09   |
|    reward             | 5869549.0 |
|    std                | 0.966     |
|    value_loss         | 1.25e+15  |
-------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6069476.213875138
Sharpe:  0.9416732517124944
=================================
-------------------------------------
| time/                 |           |
|    fps                | 222       |
|    iterations         | 5000      |
|    time_elapsed       | 224       |
|    total_timesteps    | 50000     |
| train/                |           |
|    entropy_loss       | -38.7     |
|    explained_variance | 0         |
|    learning_rate      | 0.0004    |
|    n_updates          | 4999      |
|    policy_loss        | 4.14e+08  |
|    reward             | 2024892.1 |
|    std                | 0.965     |
|    value_loss         | 1.43e+14  |
-------------------------------------
CPU times: user 3min 42s, sys: 2.41 s, total: 3min 45s
Wall time: 3min 44s
3.3.3.2 PPO
agent = DRLAgent(env = env_train)
PPO_PARAMS = {
  "n_steps": 2048,
  "ent_coef": 0.005,
  "learning_rate": 0.001,
  "batch_size": 128,
}
model_ppo = agent.get_model("ppo",model_kwargs = PPO_PARAMS)
model_ppo
{'n_steps': 2048, 'ent_coef': 0.005, 'learning_rate': 0.001, 'batch_size': 128}
Using cuda device
<stable_baselines3.ppo.ppo.PPO at 0x7fa139f3e950>
%%time
# train PPO agent
trained_ppo = agent.train_model(model=model_ppo, tb_log_name='ppo', total_timesteps=50_000)
Logging to tensorboard_log/ppo/ppo_2
----------------------------------
| time/              |           |
|    fps             | 278       |
|    iterations      | 1         |
|    time_elapsed    | 7         |
|    total_timesteps | 2048      |
| train/             |           |
|    reward          | 4205996.5 |
----------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6471950.428373658
Sharpe:  0.9810113167249571
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 254          |
|    iterations           | 2            |
|    time_elapsed         | 16           |
|    total_timesteps      | 4096         |
| train/                  |              |
|    approx_kl            | 8.993084e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 8.1e+14      |
|    n_updates            | 10           |
|    policy_gradient_loss | -5.78e-06    |
|    reward               | 2551239.2    |
|    std                  | 1            |
|    value_loss           | 1.82e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5671570.88193258
Sharpe:  0.9191388907320074
=================================
-----------------------------------------
| time/                   |             |
|    fps                  | 245         |
|    iterations           | 3           |
|    time_elapsed         | 25          |
|    total_timesteps      | 6144        |
| train/                  |             |
|    approx_kl            | 9.95351e-09 |
|    clip_fraction        | 0           |
|    clip_range           | 0.2         |
|    entropy_loss         | -39.7       |
|    explained_variance   | 0           |
|    learning_rate        | 0.001       |
|    loss                 | 2.41e+15    |
|    n_updates            | 20          |
|    policy_gradient_loss | -4.06e-06   |
|    reward               | 1314939.5   |
|    std                  | 1           |
|    value_loss           | 3.97e+15    |
-----------------------------------------
------------------------------------------
| time/                   |              |
|    fps                  | 244          |
|    iterations           | 4            |
|    time_elapsed         | 33           |
|    total_timesteps      | 8192         |
| train/                  |              |
|    approx_kl            | 9.604264e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | -2.38e-07    |
|    learning_rate        | 0.001        |
|    loss                 | 2.08e+15     |
|    n_updates            | 30           |
|    policy_gradient_loss | -3.18e-06    |
|    reward               | 5233401.5    |
|    std                  | 1            |
|    value_loss           | 4.18e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5817540.606434593
Sharpe:  0.9306042990926048
=================================
-------------------------------------------
| time/                   |               |
|    fps                  | 241           |
|    iterations           | 5             |
|    time_elapsed         | 42            |
|    total_timesteps      | 10240         |
| train/                  |               |
|    approx_kl            | 8.4983185e-09 |
|    clip_fraction        | 0             |
|    clip_range           | 0.2           |
|    entropy_loss         | -39.7         |
|    explained_variance   | 0             |
|    learning_rate        | 0.001         |
|    loss                 | 1.34e+15      |
|    n_updates            | 40            |
|    policy_gradient_loss | -4.26e-06     |
|    reward               | 3246761.5     |
|    std                  | 1             |
|    value_loss           | 2.75e+15      |
-------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5695902.7291096095
Sharpe:  0.9176930266699396
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 239          |
|    iterations           | 6            |
|    time_elapsed         | 51           |
|    total_timesteps      | 12288        |
| train/                  |              |
|    approx_kl            | 9.778887e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 1.19e+15     |
|    n_updates            | 50           |
|    policy_gradient_loss | -4.9e-06     |
|    reward               | 1709994.4    |
|    std                  | 1            |
|    value_loss           | 3e+15        |
------------------------------------------
------------------------------------------
| time/                   |              |
|    fps                  | 240          |
|    iterations           | 7            |
|    time_elapsed         | 59           |
|    total_timesteps      | 14336        |
| train/                  |              |
|    approx_kl            | 9.371433e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | -1.19e-07    |
|    learning_rate        | 0.001        |
|    loss                 | 1.92e+15     |
|    n_updates            | 60           |
|    policy_gradient_loss | -3.62e-06    |
|    reward               | 6413815.5    |
|    std                  | 1            |
|    value_loss           | 3.92e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5773825.456382013
Sharpe:  0.9292986010034127
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 8            |
|    time_elapsed         | 68           |
|    total_timesteps      | 16384        |
| train/                  |              |
|    approx_kl            | 8.527422e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 1.79e-07     |
|    learning_rate        | 0.001        |
|    loss                 | 2.27e+15     |
|    n_updates            | 70           |
|    policy_gradient_loss | -3.54e-06    |
|    reward               | 3521137.8    |
|    std                  | 1            |
|    value_loss           | 4.2e+15      |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5913896.907793608
Sharpe:  0.9398449534658818
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 9            |
|    time_elapsed         | 77           |
|    total_timesteps      | 18432        |
| train/                  |              |
|    approx_kl            | 9.080395e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 9.56e+14     |
|    n_updates            | 80           |
|    policy_gradient_loss | -5.98e-06    |
|    reward               | 2331254.8    |
|    std                  | 1            |
|    value_loss           | 1.79e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5906144.2790909065
Sharpe:  0.936100873233639
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 236          |
|    iterations           | 10           |
|    time_elapsed         | 86           |
|    total_timesteps      | 20480        |
| train/                  |              |
|    approx_kl            | 9.109499e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | -1.19e-07    |
|    learning_rate        | 0.001        |
|    loss                 | 1.84e+15     |
|    n_updates            | 90           |
|    policy_gradient_loss | -3.91e-06    |
|    reward               | 1304201.6    |
|    std                  | 1            |
|    value_loss           | 3.56e+15     |
------------------------------------------
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 11           |
|    time_elapsed         | 94           |
|    total_timesteps      | 22528        |
| train/                  |              |
|    approx_kl            | 8.440111e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 1.19e-07     |
|    learning_rate        | 0.001        |
|    loss                 | 2.29e+15     |
|    n_updates            | 100          |
|    policy_gradient_loss | -3.78e-06    |
|    reward               | 5044326.5    |
|    std                  | 1            |
|    value_loss           | 4.41e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5613406.579829459
Sharpe:  0.9137180403563157
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 12           |
|    time_elapsed         | 103          |
|    total_timesteps      | 24576        |
| train/                  |              |
|    approx_kl            | 8.934876e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 1.15e+15     |
|    n_updates            | 110          |
|    policy_gradient_loss | -4.32e-06    |
|    reward               | 3345747.8    |
|    std                  | 1            |
|    value_loss           | 2.18e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6564181.124094705
Sharpe:  0.9917371900900754
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 13           |
|    time_elapsed         | 112          |
|    total_timesteps      | 26624        |
| train/                  |              |
|    approx_kl            | 8.643838e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | -1.19e-07    |
|    learning_rate        | 0.001        |
|    loss                 | 1.62e+15     |
|    n_updates            | 120          |
|    policy_gradient_loss | -4.75e-06    |
|    reward               | 1763981.2    |
|    std                  | 1            |
|    value_loss           | 3.2e+15      |
------------------------------------------
-----------------------------------------
| time/                   |             |
|    fps                  | 237         |
|    iterations           | 14          |
|    time_elapsed         | 120         |
|    total_timesteps      | 28672       |
| train/                  |             |
|    approx_kl            | 8.58563e-09 |
|    clip_fraction        | 0           |
|    clip_range           | 0.2         |
|    entropy_loss         | -39.7       |
|    explained_variance   | 5.96e-08    |
|    learning_rate        | 0.001       |
|    loss                 | 2.7e+15     |
|    n_updates            | 130         |
|    policy_gradient_loss | -3.39e-06   |
|    reward               | 5603086.5   |
|    std                  | 1           |
|    value_loss           | 5.08e+15    |
-----------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5739353.082453819
Sharpe:  0.9239509454536167
=================================
-------------------------------------------
| time/                   |               |
|    fps                  | 237           |
|    iterations           | 15            |
|    time_elapsed         | 129           |
|    total_timesteps      | 30720         |
| train/                  |               |
|    approx_kl            | 9.6915755e-09 |
|    clip_fraction        | 0             |
|    clip_range           | 0.2           |
|    entropy_loss         | -39.7         |
|    explained_variance   | 1.79e-07      |
|    learning_rate        | 0.001         |
|    loss                 | 1.61e+15      |
|    n_updates            | 140           |
|    policy_gradient_loss | -3.99e-06     |
|    reward               | 3180577.8     |
|    std                  | 1             |
|    value_loss           | 3.38e+15      |
-------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6121677.119323213
Sharpe:  0.9518584052880635
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 16           |
|    time_elapsed         | 137          |
|    total_timesteps      | 32768        |
| train/                  |              |
|    approx_kl            | 8.876668e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 1.27e+15     |
|    n_updates            | 150          |
|    policy_gradient_loss | -4.17e-06    |
|    reward               | 2041366.0    |
|    std                  | 1            |
|    value_loss           | 2.38e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6194900.095902682
Sharpe:  0.9533504685776558
=================================
-----------------------------------------
| time/                   |             |
|    fps                  | 237         |
|    iterations           | 17          |
|    time_elapsed         | 146         |
|    total_timesteps      | 34816       |
| train/                  |             |
|    approx_kl            | 9.57516e-09 |
|    clip_fraction        | 0           |
|    clip_range           | 0.2         |
|    entropy_loss         | -39.7       |
|    explained_variance   | -1.19e-07   |
|    learning_rate        | 0.001       |
|    loss                 | 1.97e+15    |
|    n_updates            | 160         |
|    policy_gradient_loss | -3.3e-06    |
|    reward               | 1066660.9   |
|    std                  | 1           |
|    value_loss           | 4.18e+15    |
-----------------------------------------
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 18           |
|    time_elapsed         | 155          |
|    total_timesteps      | 36864        |
| train/                  |              |
|    approx_kl            | 9.516953e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 1.19e-07     |
|    learning_rate        | 0.001        |
|    loss                 | 2.44e+15     |
|    n_updates            | 170          |
|    policy_gradient_loss | -3.2e-06     |
|    reward               | 4368200.5    |
|    std                  | 1            |
|    value_loss           | 4.91e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6256386.665746946
Sharpe:  0.9645291056516293
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 19           |
|    time_elapsed         | 163          |
|    total_timesteps      | 38912        |
| train/                  |              |
|    approx_kl            | 9.662472e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 1.16e+15     |
|    n_updates            | 180          |
|    policy_gradient_loss | -4.02e-06    |
|    reward               | 2860747.2    |
|    std                  | 1            |
|    value_loss           | 1.96e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6374538.443520849
Sharpe:  0.9745580669847568
=================================
-------------------------------------------
| time/                   |               |
|    fps                  | 237           |
|    iterations           | 20            |
|    time_elapsed         | 172           |
|    total_timesteps      | 40960         |
| train/                  |               |
|    approx_kl            | 1.0186341e-08 |
|    clip_fraction        | 0             |
|    clip_range           | 0.2           |
|    entropy_loss         | -39.7         |
|    explained_variance   | -1.19e-07     |
|    learning_rate        | 0.001         |
|    loss                 | 1.94e+15      |
|    n_updates            | 190           |
|    policy_gradient_loss | -3.91e-06     |
|    reward               | 1601631.9     |
|    std                  | 1             |
|    value_loss           | 3.69e+15      |
-------------------------------------------
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 21           |
|    time_elapsed         | 180          |
|    total_timesteps      | 43008        |
| train/                  |              |
|    approx_kl            | 8.556526e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 1.79e-07     |
|    learning_rate        | 0.001        |
|    loss                 | 2.34e+15     |
|    n_updates            | 200          |
|    policy_gradient_loss | -3.22e-06    |
|    reward               | 5244387.0    |
|    std                  | 1            |
|    value_loss           | 4.63e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:6116456.21387306
Sharpe:  0.9532338438249139
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 22           |
|    time_elapsed         | 189          |
|    total_timesteps      | 45056        |
| train/                  |              |
|    approx_kl            | 9.516953e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 1.66e+15     |
|    n_updates            | 210          |
|    policy_gradient_loss | -3.82e-06    |
|    reward               | 3210717.2    |
|    std                  | 1            |
|    value_loss           | 3.24e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5553792.098235709
Sharpe:  0.9092045718318931
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 23           |
|    time_elapsed         | 198          |
|    total_timesteps      | 47104        |
| train/                  |              |
|    approx_kl            | 9.720679e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 0            |
|    learning_rate        | 0.001        |
|    loss                 | 1.4e+15      |
|    n_updates            | 220          |
|    policy_gradient_loss | -5.03e-06    |
|    reward               | 2039149.5    |
|    std                  | 1            |
|    value_loss           | 2.71e+15     |
------------------------------------------
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 24           |
|    time_elapsed         | 206          |
|    total_timesteps      | 49152        |
| train/                  |              |
|    approx_kl            | 9.604264e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 1.19e-07     |
|    learning_rate        | 0.001        |
|    loss                 | 1.8e+15      |
|    n_updates            | 230          |
|    policy_gradient_loss | -3.74e-06    |
|    reward               | 5600381.5    |
|    std                  | 1            |
|    value_loss           | 3.64e+15     |
------------------------------------------
=================================
begin_total_asset:1000000
end_total_asset:5790217.426083512
Sharpe:  0.9278784245395951
=================================
------------------------------------------
| time/                   |              |
|    fps                  | 237          |
|    iterations           | 25           |
|    time_elapsed         | 215          |
|    total_timesteps      | 51200        |
| train/                  |              |
|    approx_kl            | 9.051291e-09 |
|    clip_fraction        | 0            |
|    clip_range           | 0.2          |
|    entropy_loss         | -39.7        |
|    explained_variance   | 1.79e-07     |
|    learning_rate        | 0.001        |
|    loss                 | 2.42e+15     |
|    n_updates            | 240          |
|    policy_gradient_loss | -2.67e-06    |
|    reward               | 3792370.0    |
|    std                  | 1            |
|    value_loss           | 4.5e+15      |
------------------------------------------
CPU times: user 3min 34s, sys: 2.72 s, total: 3min 37s
Wall time: 3min 36s

3.4 Evaluation

We now use the most recent data to evaluate the performance of the two models of the agent. This is also referred to as back-testing or simply trading. This data has never been seen by the training process. The start date of this data is captured in the parameter TRADE_START.

TRADE_START
'2020-07-01'
# hide
env_kwargs
{'action_space': 28,
 'hmax': 100,
 'initial_amount': 1000000,
 'reward_scaling': 0.1,
 'state_space': 28,
 'stock_dim': 28,
 'tech_indicator_list': ['macd',
  'boll_ub',
  'boll_lb',
  'rsi_30',
  'cci_30',
  'dx_30',
  'close_30_sma',
  'close_60_sma'],
 'transaction_cost_pct': 0}
# trade = data_split(df,'2020-07-01', '2021-09-02')
trade = data_split(df, TRADE_START, DATA_END)
e_trade_gym = Portfolio(df=trade, **env_kwargs)
e_trade_gym
<__main__.Portfolio at 0x7fa139521c10>
baseline_df = get_baseline(
        ticker="^DJI", 
        start=TRADE_START,
        end=DATA_END)
baseline_df_stats = backtest_stats(baseline_df, value_col_name = 'close')
baseline_returns = get_daily_return(baseline_df, value_col_name="close")
dji_cumpod =(baseline_returns + 1).cumprod() - 1
[*********************100%***********************]  1 of 1 completed
Shape of DataFrame:  (295, 8)
Annual return          0.311845
Cumulative returns     0.374034
Annual volatility      0.140762
Sharpe ratio           2.006165
Calmar ratio           3.491806
Stability              0.950106
Max drawdown          -0.089308
Omega ratio            1.397014
Sortino ratio          2.988706
Skew                        NaN
Kurtosis                    NaN
Tail ratio             1.094883
Daily value at risk   -0.016614
dtype: float64
df_daily_return_a2c, df_actions_a2c = DRLAgent.DRL_prediction(model=trained_a2c, environment = e_trade_gym)
df_daily_return_ppo, df_actions_ppo = DRLAgent.DRL_prediction(model=trained_ppo, environment = e_trade_gym)
time_ind = pd.Series(df_daily_return_a2c.date)
a2c_cumpod =(df_daily_return_a2c.daily_return + 1).cumprod() - 1
ppo_cumpod =(df_daily_return_ppo.daily_return + 1).cumprod() - 1
DRL_strat_a2c = convert_daily_return_to_pyfolio_ts(df_daily_return_a2c)
DRL_strat_ppo = convert_daily_return_to_pyfolio_ts(df_daily_return_ppo)

perf_func = timeseries.perf_stats
perf_stats_all_a2c = perf_func(returns=DRL_strat_a2c, factor_returns=DRL_strat_a2c, positions=None, transactions=None, turnover_denom="AGB")
perf_stats_all_ppo = perf_func(returns=DRL_strat_ppo, factor_returns=DRL_strat_ppo, positions=None, transactions=None, turnover_denom="AGB")
=================================
begin_total_asset:1000000
end_total_asset:1406004.7584590872
Sharpe:  2.107539731140808
=================================
hit end!
=================================
begin_total_asset:1000000
end_total_asset:1379799.1786619413
Sharpe:  2.0438193360059196
=================================
hit end!
# hide
len(df_actions_a2c.columns)
28

3.4.1 Inspect actions

For the sake of interest, we inspect some the actions taken by the A2C agent.

# 
# Inspect the actions taken by the A2C agent (for interest sake)
# A2C actions
df_actions_a2c
AAPL AMGN AXP BA CAT CRM CSCO CVX DIS GS HD HON IBM INTC JNJ JPM KO MCD MMM MRK MSFT NKE PG TRV UNH VZ WBA WMT
date
2020-07-01 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714 0.035714
2020-07-02 0.022851 0.021544 0.021544 0.054664 0.058563 0.021544 0.058563 0.033766 0.021544 0.021544 0.021544 0.047575 0.029315 0.037148 0.058563 0.021544 0.021544 0.058563 0.021544 0.021544 0.058563 0.021544 0.038039 0.027352 0.058563 0.021544 0.058563 0.040820
2020-07-06 0.023122 0.023122 0.023122 0.023122 0.023122 0.023122 0.062852 0.062852 0.034828 0.062852 0.023122 0.062852 0.062852 0.062852 0.023122 0.023122 0.023122 0.023122 0.053111 0.023122 0.032871 0.023122 0.024343 0.023122 0.047578 0.023122 0.060211 0.023122
2020-07-07 0.022395 0.022395 0.022395 0.060876 0.044717 0.029106 0.037972 0.060876 0.022395 0.026222 0.026109 0.060876 0.041859 0.051377 0.025097 0.022395 0.033752 0.031678 0.022395 0.035332 0.060876 0.031227 0.060876 0.053564 0.022395 0.022395 0.026053 0.022395
2020-07-08 0.026839 0.035866 0.021922 0.059589 0.059589 0.022608 0.021922 0.024016 0.021922 0.059589 0.021922 0.049543 0.030209 0.021922 0.021922 0.021922 0.021922 0.045135 0.034305 0.021922 0.057705 0.046461 0.059589 0.026816 0.035204 0.021922 0.059589 0.048134
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2021-08-25 0.023468 0.041688 0.023468 0.037213 0.047170 0.023468 0.023468 0.063792 0.023468 0.044747 0.048257 0.029479 0.023468 0.063792 0.023468 0.023468 0.050654 0.041948 0.023468 0.023468 0.058281 0.023468 0.023468 0.023468 0.063792 0.023468 0.035619 0.045020
2021-08-26 0.052309 0.019244 0.019244 0.052309 0.052309 0.052309 0.050473 0.052309 0.019244 0.033189 0.019244 0.052309 0.052309 0.044344 0.052309 0.019244 0.019244 0.019244 0.019244 0.019244 0.019244 0.041529 0.040483 0.019244 0.052309 0.019244 0.052309 0.035965
2021-08-27 0.022473 0.022473 0.022473 0.022473 0.054423 0.061087 0.022473 0.027179 0.022473 0.022473 0.061087 0.061087 0.049859 0.022473 0.059373 0.022473 0.061087 0.022473 0.053641 0.022473 0.022473 0.022473 0.049426 0.022473 0.061087 0.022473 0.022473 0.041102
2021-08-30 0.042140 0.025380 0.025380 0.025380 0.055952 0.025380 0.040324 0.068990 0.025380 0.068990 0.025380 0.028750 0.025957 0.025380 0.025380 0.025380 0.025380 0.025380 0.025380 0.025380 0.025380 0.025380 0.068990 0.025380 0.068990 0.025380 0.030466 0.068990
2021-08-31 0.024780 0.024780 0.024780 0.056376 0.067358 0.024780 0.045034 0.067358 0.024780 0.024780 0.024780 0.066868 0.024780 0.024780 0.024780 0.024780 0.028545 0.041410 0.051628 0.024780 0.044195 0.024780 0.048336 0.024780 0.043857 0.024780 0.042559 0.024780

295 rows × 28 columns

Here is a visualization of the A2C agent’s actions specifically on AAPL:

fig = go.Figure()
fig.update_layout(width=900, height=600)
fig.add_trace(go.Scatter(x=time_ind, y=df_actions_a2c['AAPL'], mode='lines', name='AAPL A2C'))
# fig.add_trace(go.Scatter(x=time_ind, y=-df_actions_ppo['AAPL'], mode='lines', name='AAPL PPO'))
fig.update_layout(
    legend=dict(
        x=0,
        y=1,
        traceorder="normal",
        font=dict(
            family="sans-serif",
            size=20,
            color="black"
        ),
        bgcolor="White",
        bordercolor="white",
        borderwidth=2))
fig.update_layout(title={
        'text': "AAPL actions of A2C & PPO",
        'y': 0.87,
        'x': 0.48,
        'xanchor': 'center',
        'yanchor': 'top'})
fig.update_layout(
    paper_bgcolor='rgba(1, 1, 0, 0)',
    plot_bgcolor='rgba(1, 1, 0, 0)',
    xaxis_title="Date",
    yaxis = dict(titlefont=dict(size=26), title="Daily Actions"),
    font=dict(size=15))
# fig.update_layout(font_size = 20)
fig.update_traces(line=dict(width=2))
fig.update_xaxes(showline=True, linecolor='black', showgrid=False, gridwidth=1, gridcolor='Black', mirror=True)
fig.update_yaxes(showline=True,linecolor='black', showgrid=False, gridwidth=1, gridcolor='Black', mirror=True)
fig.update_yaxes(zeroline=True, zerolinewidth=1, zerolinecolor='Grey')
fig.show()

3.4.2 Inspect daily return

Here is a visualization of the daily return of the portfolio:

fig = go.Figure()
fig.update_layout(width=900, height=600)
# fig.add_trace(go.Scatter(x=time_ind, y=df_actions_a2c['AAPL'], mode='lines', name='AAPL'))
fig.add_trace(go.Scatter(x=time_ind, y=DRL_strat_a2c, mode='lines', name='A2C'))
fig.add_trace(go.Scatter(x=time_ind, y=DRL_strat_ppo, mode='lines', name='PPO'))
fig.update_layout(
    legend=dict(
        x=0,
        y=1,
        traceorder="normal",
        font=dict(
            family="sans-serif",
            size=20,
            color="black"
        ),
        bgcolor="White",
        bordercolor="white",
        borderwidth=2))
fig.update_layout(title={
        'text': "Daily Return of A2C & PPO",
        'y': 0.87,
        'x': 0.48,
        'xanchor': 'center',
        'yanchor': 'top'})
fig.update_layout(
    paper_bgcolor='rgba(1, 1, 0, 0)',
    plot_bgcolor='rgba(1, 1, 0, 0)',
    xaxis_title="Date",
    yaxis = dict(titlefont=dict(size=26), title="Daily Return"),
    font=dict(size=15))
# fig.update_layout(font_size = 20)
fig.update_traces(line=dict(width=2))
fig.update_xaxes(showline=True, linecolor='black', showgrid=False, gridwidth=1, gridcolor='Black', mirror=True)
fig.update_yaxes(showline=True,linecolor='black', showgrid=False, gridwidth=1, gridcolor='Black', mirror=True)
fig.update_yaxes(zeroline=True, zerolinewidth=1, zerolinecolor='Grey')
fig.show()

3.4.3 Inspect cumulative return

Finally, we inspect the cumulative return of the portfolio brought about by each of the agents. The DJIA index is used as a baseline for reference. Both agents end up with a larger cumulative return than the DJIA.

fig = go.Figure()
fig.update_layout(width=900, height=900)
fig.add_trace(go.Scatter(x=time_ind, y=dji_cumpod, mode='lines', name='DJIA', line=dict(color="#a9a9a9")))
fig.add_trace(go.Scatter(x=time_ind, y=a2c_cumpod, mode='lines', name='A2C'))
fig.add_trace(go.Scatter(x=time_ind, y=ppo_cumpod, mode='lines', name='PPO'))
fig.update_layout(
    legend=dict(
        x=0,
        y=1,
        traceorder="normal",
        font=dict(
            family="sans-serif",
            size=16,
            color="black"
        ),
        bgcolor="White",
        bordercolor="white",
        borderwidth=2))
fig.update_layout(title={
        'text': "Cumulative Return of A2C & PPO against DJIA",
        'y': 0.92,
        'x': 0.48,
        'xanchor': 'center',
        'yanchor': 'top'})
fig.update_layout(
    paper_bgcolor='rgba(1, 1, 0, 0)',
    plot_bgcolor='rgba(1, 1, 0, 0)',
    xaxis_title="Date",
    yaxis = dict(titlefont=dict(size=26), title="Cumulative Return"),
    font=dict(size=15))
# fig.update_layout(font_size = 20)
fig.update_traces(line=dict(width=2))
fig.update_xaxes(showline=True, linecolor='black', showgrid=False, gridwidth=1, gridcolor='Black', mirror=True)
fig.update_yaxes(showline=True,linecolor='black', showgrid=False, gridwidth=1, gridcolor='Black', mirror=True)
fig.update_yaxes(zeroline=True, zerolinewidth=1, zerolinecolor='Grey')
fig.show()

Disclaimer

This content is only meant for research purposes and is not meant to be used in any form of trading. Past performance is no guarantee of future results. If you suffer losses from making use of this content, directly or indirectly, you are the sole person responsible for the losses. The author will not be held responsible in any way.