Through the Firehose - Exchanges, WebSockets, and Rust
Rewrite it in Rust!
In this shorter write-up, I will walk through a spattering of relatively complex topics at a high level. The objective is to build my first programmatic data feed for my systematic trading engine. I plan to use Coinbase’s Advanced Trade API as an entry point for data and eventually trading. In building a market data handler, we need a base of knowledge covering technical subjects like financial exchanges and how they operate, WebSocket protocols for data transfer, and the Rust programming language (the language I chose for the job). My last blog post covers much of the motivation for what I’ll be discussing today so feel free to give that a read if you missed it:
Where does the data come from and how do we get it?
In short, financial exchanges are where the pricing data for financial instruments originates. In the old days, people yelled back and forth to make trades but now the only sound an exchange makes is the subtle hum of a server rack’s fans. Since exchanges have a fee-based business model, taking a small percentage of all trades settled on their exchange, it’s in their best interest to facilitate as many trades for as many people as possible.
This brings us to the next point. Exchanges typically provide several retail-grade access options for their technical audience via FIX Protocols, REST APIs, or WebSocket Endpoints. For my use case, WebSocket made the most sense. The ability to simplify the data streaming by sending a handshake and subscription method makes the coding overhead much lower.
This takes us to the next topic, interacting with exchange data feeds programmatically. There’s been a bit of a stir online regarding the “rewrite it in Rust” propaganda, so last year I decided to get with the program and start learning Rust. As put by Adrian Kosmaczewski, “Rust shows many, if not all, signs of modernity”1, which is a strong argument for the language being in almost any developer's toolkit. Additionally, it doesn’t hurt that Rust topped StackOverflow’s 2022 Developer Survey list as the most-loved language2.
Software Plan
In figure 1 I show a basic software diagram for our market data handler. The objective is to create an infinite event loop that will search for all, or some, tradable products, then subscribe to that product’s WebSocket for real-time price data. A simplified walkthrough of the process:
Authenticate a connection to the Coinbase Advanced Trade REST API
On a “200 - Ok” response we will connect to the Advanced Trade WebSocket
Afterward, we send a GET request for some or all tradable products (e.g. “BTC-USD”, “ETH-USD”, etc.)
Once we receive the product availability, we then send subscription requests to each product’s WebSocket
Incoming messages (prices) are parsed within the event loop
Now I just need to code all this up…
Let’s get Rusty
While I probably could have used any number of programming languages for an application this simple, Rust stood out to me as the right tool for the job. The language’s speed, reliability, and productivity gave me the confidence to start building one of the trading system’s most critical applications.
After a long battle with the borrow checker, I began to find aspects of Rust that I will whole-heartedly miss when I use other languages.
Rust’s package manager
cargomade setting up my environment easier than it has ever been. Python’scondaand JavaScript’snpmshould take note of cargo’s simplicity.Another great thing about Rust is that I really wanted to document my code — something I never thought I’d say as someone who mainly hacks together “science code”. The built-in documentation builder with test parsing is incredible.
Rust has a spectacular
matchkeyword that lets you do rather complex case-matching across your applications. Came in handy for parsing the variety of incoming REST and WebSocket messages.Additionally, I want to shout out the awesome cargo crates serde and anyhow. Serde made serialization/deserialization a breeze, and anyhow’s error handling and
Resultsignificantly reduced the error handling overhead.
Setting up the REST client was rather simple and leagues faster than my previous Python implementation. I was overcome with joy seeing the DEBUG: Found 549 available symbol. for the first time.
Next was the more difficult part. Building out a whole WebSocket client and Advanced Trade Websocket client. I had to get comfortable with WebSockets, error handling, serialization/deserialization, connections, async, and thread safety. In other words, it was a big bite. The keen eye might see there is a three-week delta between the screenshot above and below.
Though this is the first iteration of the market data handler, for the first exchange I am concerned with, I learned quite a lot and along the way built a highly modular code system that will be easier to extend to other data feeds. My future plans are to:
Integrate my cloud time-series database to store this data. This probably looks something like passing incoming messages into batches and uploading those batches to TimescaleDB via sqlx.
Write a file-flattener and unpacker for data that ages out, a la Apache Iceberg. This is a bit absurd, but I am really interested in how it works and even more interested in reducing my cloud bill. This will also come into play when I start working on building my backtesting engine and need to efficiently extract price histories.
Conclusion and Future Plans
Thanks for taking the time to read my ramblings about a statically-typed language that no one IRL cares to listen to me drone on about. I’m hoping to get out another post this month talking about how I integrate a cloud-based time series database into the data feed. We’ll talk about equally exciting things like optimal batching and type safety there I’m sure.
What I am Reading
Considering Rust, John Gjengset - A video lecture that details high-level comparisons between Rust and Java, C++, and Python. Also gives some great insight into Rust’s selling points, drawbacks, and long-term viability. This video was a big motivator for me to use the language in my systematic trading.
The technology behind GitHub’s new code search, Timothy Clem - In the vein of “rewrite it in Rust”, GitHub rewrote their code search engine in Rust. I’ve certainly noticed the difference in usability and speed when searching for code, and the article is a good summary of how a large-scale team planned and executed building a Rust application for the largest public code search index.
Living with a SEAL: 31 Days Training with the Toughest Man on the Planet, Jesse Itzler - This was a fun read (listen) for me. I’ve been on a bit of a fitness journey for the past year and found the book’s commentary on disrupting your comfort and controlling your mind to be motivational. Though the book does say “motivation is bullshit”. I’ve been running about 100+ miles a month and this book encouraged me to run those miles a little bit faster.
The Great Rewriting in Rust, Adrian Kosmaczewski (2021)
Great post, Thanks!
Do you plan to continue with posting your progress?