Endeavoring to Build a Trading System
In my effort to always be learning, I have set myself a rather ambitious goal. Over the course of the next year, I want to delve into the fascinating world of high and mid frequency trading systems.
The foremost ‘expert’ on the subject and just about every subject ever (up until 2021) ChatGPT concisely summarizes:
“High-frequency trading (HFT) refers to the use of algorithmic systems to execute trades on financial markets at high speeds, with low latency and high frequency. It emerged in the 2000s with advancements in technology and data connectivity, providing new opportunities for algorithmic trading. HFT seeks to exploit market inefficiencies, such as price differences between exchanges or discrepancies in information, by using algorithms to execute trades in milliseconds. The market opportunity for HFT arises from the increasing efficiency and speed of financial markets, allowing traders to take advantage of price discrepancies in real-time.” - OpenAI's language model, ChatGPT, who kindly informed me on the practice of citing large-language models
Now, I am not delusional. I know that I am not an expert in software development, networking, or FPGA programming. But, I believe that by dedicating time to learning about these systems, I can learn enough to be dangerous.
Luckily, I am not starting this journey from scratch. My background in quantitative finance and machine learning has given me a good understanding of the tools required for this task. I am proficient in the Rust programming language, which I plan to use for market data handling and order management. Additionally, I have experience developing trading models using Python, and I may continue using it or even port Python systems to Rust. To kick off my journey, I did extensive research this month and came up with a plan to start building my own trading system, which I will outline below.
So what do I plan to do? I am endeavoring to develop a real-time trading system for high and mid-frequency trading that can handle high volumes of financial data, execute trades, and provide up-to-date market information. The system should be scalable, robust, and have low latency to ensure fast and accurate trade execution. Additionally, the system must be secure and have robust risk management features to prevent unauthorized access and financial losses.
That statement was quite a mouthful, but succinctly I want to achieve the following goals:
Improve my understanding of exchange design and high-performance trading
Build a successful real-time trading system.
Become a proficient Rust programmer, and develop my overall programming skillset.
System Design
“If I was starting a HFT system from scratch, I would treat the development of the trading system as if I was writing an API — as opposed to a focused system for a specific task–to allow myself maximum flexibility during the process of model development.” - wkselph, Building a Trading System - General Considerations (2011)1
The algorithmic trading process consists of two components: the dynamic system and the static system. The dynamic system includes strategies such as determining the fair price of an asset, responding to market events, and making predictions about future returns. These are the trading models. The static system, on the other hand, includes applications that handle connections to markets, databases, order management systems, and many other processes (described in detail by Max Dama in his Quantitative Trading Summary writeup2). These are the constant parts of algorithmic trading, as they are the system in which your trading models live.
Why separate the dynamic and static systems? Combining the two into one high-performance application may seem like a good idea, but it can lead to technical debt and slow down development. I’ve experienced it personally, trying to manage a brokerage connection in the same code where you’re running even the simplest strategies will only hold back the performance of both your models and your trading. Keeping the strategies and orders separate allows you to easily implement new strategies or respond to short-lived alpha without being bogged down by technical issues.
Development
Breaking apart these systems creates several logical sections of development:
Market Data Handler (MDH) - Primarily a static system, the market data handler’s job is to collect, process, and store data from a variety of sources and markets. This data informs all further work within a trading system and is a fundamental first step for any algorithmic trading solution.
Order Management System (OMS) - wkselph has a great blog post detailing the importance of designing an order management system as a fixed construction within the overall trading architecture. He describes the idea of an intelligent OMS, which sounds like an intriguing problem to figure out, as much of my current strategy execution and risk management code exists as parameterized models.
Strategies - Given that markets are by definition large stochastic processes, our strategies will fall under the dynamic section of our development. Strategies are inherently not part of the trading system but are trading models. Ever-changing, adapting to changes across a number of market factors.
The architecture above covers the flow of our trading system described thus far: Exchange posts a price update, the market data handler processes and stores the update, our strategies listen for the update and creates a trigger if a trade condition is met, the order management system is notified of the trigger and sends out the order after checking risk parameters, then we get notified of our trade (hopefully)3
Simple enough right!?
Of course not! In my limited research and experience, I’ve compiled a short list of considerations and challenges that already stand in my way:
Funding management across exchanges: From what I hear, big firms have whole teams dedicated to managing capital distribution across exchanges.
Data quality and consistency: Crypto exchanges were built by full-stack engineers in California, and don’t have the same reliability constraints as Wall Street.
System reliability and stability: I no nothing about reliability engineering and very little about release stability testing and benchmarking. Furthermore, high-performance systems need to be engineered to handle massive data feeds (millions of orders per second, per some sources).
Personal skill gaps: My experience in physics, machine learning, and quantitative trading cover only a small portion of the expertise needed to truly create success in high-performance trading systems.
Burnout: I have a full-time job. All of this will be for “fun”. An undertaking like this certainly poses a burnout risk.
Conclusion
With all the above in mind, I am moving forward. I have a “never finished” mentality in life, especially with regard to learning. My suspicion is that this undertaking will never truly feel complete to me, taking many years before I am satisfied with the breadth of the system. As for a timeline I’m comfortable detailing on my blog, let’s see if I can start posting write-ups on trading models by mid-year with all the data I am working on collecting.
Upcoming
Through the Firehouse - Exchanges, WebSockets, and Rust
My next blog post will be a walkthrough of how the Rust programming language can be used to create a WebSocket processing application for a variety of cryptocurrency exchanges and is the first step in the market data receiver for my trading system.
What I am Reading
Advances in Financial Machine Learning, Marcos López de Prado - Comprehensive review on financial machine learning. This will probably always be linked down here, as I come back to it routinely.
Rust in Action, Tim McNamara - A systems programming playbook for the Rust programming language. Covers not only Rust but concepts in computer architecture and systems programming as well.
Rust in Blockchain Newsletter - Always interesting to read about how people much smarter than me are applying Rust to the blockchain ecosystem and in the crypto world.
Building a Trading System - General Considerations, wkselph (2011)
Quantitative Trading Summary, Max Dama (2017)
Trading at light speed: designing low latency systems in C++, David Gross (2022)