Case study: Accelerated backtesting for deep learning HFT models
Backtesting the performance of potential trading models is a key step in the research and development of new HFT (High Frequency Trading) strategies. However, it is also one of the most computationally intensive tasks, requiring compromises to be made in the scope of the backtesting simulations or to delay the time to deploy the new strategy until the simulations are complete.
Our Hyper-Efficient Training technology drastically reduces the time to backtest models, without requiring any additional hardware. In this case study, we show that with our hyper-efficient training approach we can gain 40x acceleration, completing a 15 month backtest in only 12 days on the same hardware. The gained time can be used to test the models over a wider range of scenarios, investigate more potential strategies and/or bring the chosen approach into production sooner.
Background
Although past performance is no guarantee of future returns, backtesting a potential trading model helps to quantify the expected performance and provides key insights to make decisions on whether to deploy a particular strategy into production. A realistic backtest simulation requires a huge amount of computational resource. Depending on the model rollout policy and the number of trading days the simulation should account for, thousands of model trainings can be required, thus making inevitable a trade-off between the number of financial scenarios covered or the time depth of the simulation, and the time to complete.
Backtest simulation
For this case study we utilise L1 quotes dataset of eleven different instruments gained over more than 1000 trading days from beginning of 2017 to end of 2021. The instruments are from different markets and total more than 920 million ticks (an average of 887,000 ticks per day).
In this simulation we have chosen a one-day model rollout policy, where a new model, trained on the latest available data, is deployed each trading day. The training protocol is window based and considers the latest six trading days, where the most recent day is used for validation and the five remaining for training.
The following figure represents the evolution of the size of the backtest data over time. On average there are almost 4.5million training samples per day.
With this policy and training protocol, a full simulation over the dataset corresponds to 1037 different training tasks to be executed.
The model to test comprises 17 parametric layers, totalling almost 300,000 parameters, and it is organized into three major blocks:
- An input block which takes raw L1 quotes vectors and consolidates them into low level features where data normalization and minimal pre-processing is applied.
- A feature extraction backbone where six sequential convolutional layers and a three branches inception block are used to create high-level features.
- A classification head where a seq2seq LSTM network is responsible to transform high-level features into probabilities of the input data being a buy, a sell or a neutral pattern.
Results
With standard training techniques, the whole backtest requires more than 15 months to complete on a single Nvidia 1080Ti GPU, with an average of 2.2 backtested days per wall-clock day.
But on the very same hardware, our Hyper-Efficient Training technology completes the entire backtest task in only 12 days, with an average of 82.3 backtested days per wall-clock day. An acceleration of 40x. As shown in the figure below, with our approach, we can complete the training on the 4.6bn samples in the windowed dataset in the same time it takes for standard techniques to train on only 2.5% of that dataset.
Unlocking potential
What could 40x acceleration mean in backtesting AI trading models?
The figure The below illustrates the acceleration potential. When the Hard Sums Technologies powered backtest finished training on the last backtest day of the 29th Dec 2021 (green dashed line in the figure), the backtest running with standard neural network training technologies had only processed data up to 21st Feb 2017 (red dashed line).
What additional performance benefits could be unlocked with the 454 computation days saved?
Get updates
Join our mailing list to keep updated on Hard Sums news and insights
By signing up you agree to our Privacy Policy
Thank you for signing up to our mailing list