HFT and market manipulation
Discussion about HFT started as people noticed recurring patterns in the BTC charts. It looked a little like automated HFT patterns to some.
What is HFT
So what is high frequency trading? In the original sense of the word it’s automated trading done by organizations that gain faster access to market information than anyone else. It’s about detecting that someone placed a big limit order and then jumping ahead the queue.
It cannot be done by human at all, it needs to be servers with the fastest possible data link to the exchange.
Does this happen on Bitcoin markets? To some extent.
Here you have a fine explanation from Reddit:
One possible (shady) way happens when you are offered a third party service that has a relationship with the exchange that brags you will get your coins cheaper than any maximum you set. In reality, you will always get your coins just a little bit cheaper than your maximum was but you would be able to get them way cheaper if you had access to the same sell offers as the service that buys the coins for you. (oversimplified, read the original comment)
The title image on this article comes from this order book visualization. They explain the patterns are not what one should call HFT. These are trading bots and automated strategies and people who are making the market. It’s not something that occurs in microseconds. So far it’s probably not feasible yet to even attempt HFT. The market is not deep enough, it wouldn’t work smoothly.
But indeed, it’s a different thing if you are partnered with an exchange. If you buy 1 bottle of beer it might cost 1 quid but buying 100 will have either much lower cost per bottle because wholesale or much higher because more demand than supply.
If you Google HFT in Bitcoin…
…One of the first results will be an old Quora thread that talks about HFT being impossible in Bitcoin because a Bitcoin transaction last long. That’s clearly bullshit since within a single exchange you don’t get “normal” transactions. There are no coins transfered, not even when trading on margin - these days most exchanges implemented more secure and faster internal solutions and there is only a symbolic transfer of ownership (keys).
Another argument was liquidity. That is true on some exchanges for scalping algorithms. Scalping is a very short term trading, often against the trend or sideways - when you are shorting local tops for pennies of profit, etc. Bots must be configured particularly for this, and some traders only turn them on for the time the close to ideal conditions last - when there are no abrupt changes. Another very important thing when scalping is zero or very low fees, because you need to make a lot of trades. And indeed, someone who will take the other side of your coin.
On Chinese exchanges none of this is an issue: trades tend to be free or very cheap and volume is largely artificial thanks to volume bots that run on behalf of the house.
See below the volume of Huobi - when volume bots took a break.
Whales and frontrunning is another matter. Every trader knows the hopeless feeling when all the TA was absolutely clear, until the whale came:
As WSJ reports, some of the U.S.’s biggest proprietary HFT traders say they see potential for big profits in trading bitcoin. Translation: unprecedented volatility, which essentially crushed all interest in Bitcoin after its peak in late 2013, is about to come back with a vengeances as such conventional HFT market rigging techniques as frontrunning, momentum ignition, quote stuffing, spoofing, subpennying and everything else in the vacuum tube arsenal is about to be unleashed on the world’s geekiest traders. (Zerohedge in April 2015)
Most awkward things tend to happen when new quarterly futures open, for instance, or even on the weekly settlement. The dangerous thing really is to follow logic and reason, at least if you trade with high leverage.
Unless you trade algorithmically, your bot is fast enough (if in nothing else then in data fetching and canceling orders) and get an instantaneous data feed. (In which 3rd party apps like cryptowatch or bitcoinwisdom tend to suck and go laggy when things get wild.)
One thing that you will need to figure out for a successful HFT setup is where are the servers of the exchanges. When you just look at the apparent IP it will be different depending on your location because of load balancing and anti-DDOS systems.
What you need to do is to find out the location and provider of the actual server and get a VPS in the same datacenter. Not all exchanges will get you faster connect this way though. Ideally you also figure out their ISP (carrier) and also use that carrier. That should result in faster connect times.
The find-out can be as simple as asking on reddit. Or elsewhere.
If you know the IP address of the servers though you can easily tell which carrier they’re on. Put your servers on the same carrier should get you on faster connect to them.
You can actually tell the carrier NOC techs when you’re purchasing the connect that you want to be as close as possible to their public IP. A lot of times they’ll build you custom routes, just takes a phone call.
Also, if you’re ballsy call up the data center and make friends with the engineers. You’d be shocked how much you can get done with a phone call or a case of beer shipped to the break room.
When a carrier applies for a block of IP addresses, a group called ARIN hands them out. There is a publicly searchable resource to determine who owns those blocks. For a candidate IP you simply search it and likely you’ll see WHOIS data showing you the owner. Example: http://mxtoolbox.com/ARIN.aspx Let’s you punch in the IP you’re searching for. In this case I’ll use the Public IP “18.104.22.168”. The tool returns Google as the owner.
A carrier is the long distance company that sells you or a data center an internet connection. It’s just like your local cable company selling you a wire for you to connect your cable modem to, only typically a much faster wire.
If you’d like to get a competitive advantage for your bots via fast/stable API access you might just put on a white hat and call up bitfinex and ask if they have a problem with you putting your trade bots in the same data center. They’ll likely sell you premium access for a fee and make your life a lot easier. They’re not Disney villains, just dudes trying to make a dollar for their families. They make money when you make money. A rising tide raises all boats etc.
Barring that, hire a network engineer for a short term contract. /r/networkengineering crawling with sharp dudes.
BitMEX server location
They are hosted at AWS Ireland.
Bitfinex server location
It appears they have a datacenter in Italy.
Bitstamp server location
Websockets - the API for HFT
The reason websockets are so fast is that it’s a single connection (ideally with a test for interruptions) and that it goes in both directions within the single connection. There still is some delay, but lower than with traditional REST API which uses separate POST vs GET requests.
Websockets are used quite a lot in the Bitcoin space:
- Bitfinex websocket API
- OkCoin websockets API on GitHub or on their homepage
- Bitstamp websocket API
- BitMEX websocket API
- Blockchain info websockets API (to track large transfers)
Some exchanges only have REST APIs but gradually most platforms are getting to set up a websockets server.
Users can set up websockets clients and get info in both directions as fast as possible. On most exchanges you get WS normally, on CEX you need to ask for it, but it is also otherwise available in mainstream trading bots.
More Reading on HFT in Bitcoin
ZeroHedge, Apr 6 2015 (print-friendly version with no ads so that your browser won’t crash) - “SkyNet Is Almost Sentient: HFTs To Start Trading Bitcoin”
CoinDesk, May 16 2016. - “Coinbase HFT-ish trading with whale frontrunning bot”
Stocksharp program and data collection (Github) - free programs for trading at any markets of the world (American, European, Asian, Russian, stocks, futures, options, Bitcoins, forex, etc.). You will be able to trade manually or automated trading (algorithmic trading robots, conventional or HFT).
Bitfinex Referral Code 5egV78YtlC
- Bitstamp realtime trading platform (Github) - Python, Redis pubsub, REST API to Bitstamp but websockets to the trader app (Bitstamp is not soo liquid so this could be good enough)
HFT on stock markets
VPro Backlight on High Frequency Trading
There is a couple of documentaries by vpro on high frequency trading on YouTube. This one is a basic one, explaining how big stock trading has moved to data centers.
Here you get some more information if you are interested.
In case you really have a lot of time and want to watch even more, this video features real quants that were around the subprime mortgage crash in the US. It explains the CDO derivative a little bit better than The Big Short movie: The issues with the mathematical model was that it didn’t count with different borrowers defaulting could be correlated events.
What does it mean for the little guy?
It looks like everyone is running a bot in the Bitcoin space already. There’s a great trust in anything automated, described also in this fintech article - if there’s no human there’s no one to screw you. But it’s BS if you think whenever it’s computed it cannot work against you.
That’s why we read scripts before we run them (or at least we should). Scripts are faster ad smarter than people but they were written by people, these people either might have had some intentions or they might have also been careless.
Take a look at Poloniex lending right now. A month ago the daily rates were above 0.5%, now it’s 0.02% and dropping. Why? Look at the order book: Because everyone is running the bot that sets your order a tiny bit below the market so that it gets executed as soon as possible. As everyone is doing that now (look at the order sizes, it’s the chunks that a bot makes) the rates keep dropping.
So what are you to do with that now, as the little guy?
Go with the flow.
Margin trade, it’s cheap now.
The same goes for sudden dips, parabolic spikes and pump&dump scheme that comes with large volume transaction. Just trade it.