Primer on "high frequency trading" -- AKA stockbots

Ars Technica gives us a look into the world of "High Frequency Trading" (HFT), a lightspeed quick automated style of stock-market trading that uses semiautonomous systems to buy and sell assets in a fraction of a second, for a fraction of a cent more than they paid, a billion times, and make real money:

Some categories of "predatory algos" closely monitor the markets in order to sniff out exactly these types of hidden large orders, so that the algo can trade against them. For instance, if a predatory algo detects that someone is trying to hide a large sell order for INTC by trickling it out into the market in small blocks, it might work to bid down the price of INTC just a bit so that it can pick up those blocks at a discount and then sell them for a profit when the share price floats back up to the market's earlier, non-manipulated valuation...

The final animal in the HFT menagerie that I'll point out on this brief tour is the automated market maker (AMM), which is a subtype of what is often called "dark pools," or "dark liquidity." AMMs like Citadel always stand ready to buy and sell large quantities of assets, and they don't publish price quotes to other market participants via exchanges.

To find out what assets a dark pool will either sell or buy and at what price, you first have to ping it. Once you ping the pool with a request to, say, buy a specific asset, the pool will reply with the price that it's willing to sell you that asset for. You can either accept the price and complete the transaction, or turn it down and ping again later to see if the price has moved in your direction.

The Matrix, but with money: the world of high-speed trading (via Futurismic)

• [Takuan]

  all descendants of “salami slicing”.

• [IWood]

  Heh. I did say a decade or so.That there is “wiggle room.” :-D

  No, I hear what you’re saying. When I said “recoup pre-bubble losses” I meant something like “when the sting has sufficiently faded from memory and I get enough money back I shall then move it into something boring and safe, like the gasoline tanker at the center of the compound.”

  And, it’s really a matter of attitude–if you’re greedy and want big stupid dot-com profits, you want another Dow-14,000 bubble inflated with something so you can exploit it. Me, I’m just concerned with a reasonable return, which means dull things like potash and steel and bearings and the occasional cool shiny tech thing like nano, which also feeds into alt energy. The danger with the potential alt-energy bubble is that if it comes (and it’s already here, to a certain extent–see First Solar’s overvalued, overhyped stock price, even after the crash), it has the potential to fuck up something really important when it pops. If there are no alt-energy equivalents of Google after that process, we’re kind of boned…

• [Anonymous]

  I recommend everyone should read the book “Devil Take The Hindmost” which looks at the history of financial speculation. Read that then re-read the article above and in my opinion it puts a different spin of how we, as individuals with disposable income looking at ‘capital preservation’ as opposed to pure ‘investment income’ should approach our finances. Financial speculation and investing has been a cut throat business since the 1500′s and is not going to change because someone feels that someone else has an advantage – technical advantages are not long lived but human greed and stupidity is forever. The trick is knowing when to get off the ferris wheel!
  However, i am also mindful of how, whether we like it or not; the global markets, governments and our overall level of prosperity are so intertwined as to be inseparable. If you cannot change it (i live in South Africa, how am i supposed to influence the US government to change legislation around SPV’s and US accounting standards??) – how best can i crash proof my life? Now thats a whole different qeustion!

  Something to also thing about – where can the ‘man in the street’ go to ‘play the market’. Many of you are right in saying that its almost impossible to get a fair go as a small player when you are up against the like of Goldmans. But keep an eye out on the rest of the world – Africa and Asia are new big thing. Im involved in financial markets in the South Africa and i think it wouldnt be unfair to say there is new global interest in Africa. Watch this space!

• [JeffF]

  There are two parts main of this.

  First is that these operations are about an outside party getting a cut of a transaction without all the other parties consenting to it and without adding value.

  The second part is that it is about a new way for the financial high fliers to make money without risk. As long as their models are accurate and behave nicely under all conditions… Uh… Yea.

  Another part of the story I read about elsewhere was that there is an element of corruption. The HFT operations are buying special access to that regular buyers and sellers don’t have allowing them to take get information and take actions the others could not. So the stock exchange is not treating everyone equally. It is possible to buy “better” access and exploit other customers of the exchange. Do the other customers have a choice to go to an exchange that doesn’t allow such exploitation to be purchased? No, not really.

  Probably what is needed is some special case regulation to prevent/make dramatically less profitable such behavior.

  It seems to me that something which has been proposed many times to deal with many different problems of this general sort might deal well with this instance: a small financial transaction tax. It would be tiny, something like 0.01%, but would make many schemes like this unprofitable forcing finance to focus on more useful activities, and if the government extracts a bit of money from the stream rather than wall street it goes to public priorities (schools, for exmaple) rather than to build the modern Versailles (helicopter accessible ski resorts for the ultra rich, for example).

• [alowishus]

  So, what this means is fairly simple, yet far-reaching: technology and greed have effectively broken the stock market. It is no longer possible for anyone except high-powered, uber-resourced brokers to make money there. The best – and only sane – thing for investors to do is get the hell away from Wall Street and never go back; find some other way to use your money to make more money. It’s a dinosaur now.

  You’re assuming the market was functional in the first place.

  I can’t help feeling that the stock market is now just a giant wealth vacuum, sucking money out of the middle class to stuff the pockets of the wealthy. Then again, the whole economy is kinda like that, huh?

• [bwcbwc]

  Isn’t this basically a high-tech form of front-running? Capture incoming orders and if they diverge too far from normal trading volumes, start manipulating the market prices so that your arbitrage difference is maximized.

• [IWood]

  About five years ago, I wrote the technical manual for one of the leading stock trading software products. It wasn’t HFT, but it was obvious that’s where things were going: financial combat algorithms fighting in a virtual arena with billions at stake.

  I’m not sure that The Matrix is quite the right reference, though–I think of what we’ve got now as the economic equivalent of Skynet.

• [legion]

  So, what this means is fairly simple, yet far-reaching: technology and greed have effectively broken the stock market. It is no longer possible for anyone except high-powered, uber-resourced brokers to make money there. The best – and only sane – thing for investors to do is get the hell away from Wall Street and never go back; find some other way to use your money to make more money. It’s a dinosaur now.

• [Thac0]

  I want to be able to use this to trade penny stocks while I’m at work!

• [mn_camera]

  This is why we need securities transaction fees.

• [Bevatron Repairman]

  All this does is smooth out the curve, it hardly negates the value of long-term investing in the stock market. There may be other reasons to not invest in the stock market, but that a company buys a stock at 1.9996 and sells it at 1.9999 before you can read the name on the stock ticker isn’t one of them.

• [Crashproof]

  More like the precursor to Charles Stross’ “Economy 2.0″.

• [IWood]

  #2 posted by legion:

  So, what this means is fairly simple, yet far-reaching: technology and greed have effectively broken the stock market. It is no longer possible for anyone except high-powered, uber-resourced brokers to make money there. The best – and only sane – thing for investors to do is get the hell away from Wall Street and never go back; find some other way to use your money to make more money. It’s a dinosaur now.

  I’m not so sure about that. The trading software I documented was used by brokerage houses…and individuals. They could make market just like a brokerage house, using the same tools and algorithms. Modern HFT hardware is basically gaming hardware, and there’s every possibility of developing open-source HFT algorithms. The true limiting factor is the same as it’s always been: capital.

  I’m a (very) small investor, and I’ve doubled my money since first buying in at the beginning of April, without using market-making software or complex algorithms. It’s still quite possible to make money on Wall Street.

• [SKR]

  I don’t see the doom and gloom that some are espousing. Sure the end of the article paints a picture of a market crash that could be huge, but it is erroneously based on a circumstance that isn’t going to occur again. That being everyone using the same algo. Let’s not forget that market crashes have occurred without the help of computers (1929). The computers do take out a cause of market crashes by eliminating human emotion or more specifically, panic. They are also just doing what traders would be doing. They just do it faster and more reliably. Actually, until

  As to the comment in the article about unwary Etrade account holders, if you read almost any book on trading you will be informed about market makers that are just waiting to take advantage, in this way, of an unwary newbie issuing market orders. What does it matter that a computer is taking advantage of your stupidity instead of an actual person? The solution is to not be stupid.

  I’m going to agree with IWOOD about individual investors being able to make money even in this choppy market. You have to carefully manage your emotion and risk.

• [IWood]

  (To clarify in light of my first comment, though–there is a portion of the market that’s inaccessible to mortal humans, but that “arena” isn’t the whole space, by any means)

• [SKR]

  Doh, I forgot to finish a sentence.

  Actually, until the computers were introduced, the market makers had a much higher margin when they capitalized on trades in this way. Now the margin is so low they need the volume afforded by computerized speed.

• [hep cat]

  A friend of mine was the head of IT in NYC for a major japanese [ bank I think, he was forbidden to name his employer ] about 10 years ago, and a major part of the job was reducing latency in the network. An extra tenth of a second for trades could cost millions if they didn’t catch it.
  Once they got switched to a satellite link and it really really cost them a lot.

• [zikzak]

  I hadn’t heard about HFT until recently, and it’s very freaking interesting. Thanks for posting about it.

  I’m reminded of “Dr. Strangelove”, in which the case is made that the logical conclusion of the nuclear arms race is to put computers in control of pressing “the button”, because of the fallibility of having humans in the loop.

  In situations like this, we’d like to retain control over how our computers operate…and in a very coarse sense we have that control by pulling the plug and altering the program.

  But in order to have them function /effectively/ we have no choice but to let them make unsupervised decisions for us. And the market clearly favors those who confer the most of their own autonomy and responsibility to the machine.

• [Keeper of the Lantern]

  Legion:
  I agree with you, except where I don’t. For a long, long time anyone (except for big fund managers) who thought they had some kind of insight was full of crap: You’d see all sorts of people using silly golden ratios or doing these absurd trend analyses and claiming to be good at investing, whereas they were just another boat on an ocean where high tide was coming in. Any “insight” has long been cancelled by those with the kind of access implied by the big trading systems.

  HOWEVER, what Wall Street really lacks (usually) is specific insight into early, pre-trend growing companies that have a real advantage that hasn’t manifested itself in trendable numbers yet. Individual investors who are really close to an industry can sometime make out like bandits in such an environemt, though such opportunities are rare.

  For example, because I went to CUNY in NYC I knew about Qualcomm in the early 90s and could see that CDMA was eventually going to take over, which it did. That’s the kind of thing that Wall Street isn’t good at spotting.

• [Caroline]

  Iwood, if you bought in at the beginning of April, you’ve just been riding a short-term trend up. I started my IRA around the same time and have seen 30%+ gains, but that’s so short term that I consider it meaningless. I don’t intend to touch that money for 40 years, so what happens over one summer isn’t a big thing.

  Over the long term, talking decades, you’ll do about as well as the market as a whole. Market timing is a fool’s game.

  I highly recommend the book Naked Economics.

• [Roy Trumbull]

  I suggest reading Richard Bookstabers “A Demon of Our Own Design”. He is a card carrying Quant and has worked in brokerage houses and for Hedge funds. He points out it is nearly impossible to parse a trade made for liquidity from a trade made from new information. He compares the trading systems in place in 1987 to Three Mile Island. Both were tightly coupled sequences. I found interesting that the one thing that hadn’t been considered in “portfolio insurance” was that the Chicago trading in futures moves at the rate of an eye blink whereas the stock market is very slow by comparison. As he puts it: Broadband vs dialup. The offset in trading speed became part of the problem.He also covers the various temporary price differences of like investments and the bets placed on convergence. They don’t last for long. Too many players kill the game.

• [Stefan Jones]

  I second #4 call for transaction fees. 0.25% should do it nicely. That would cost me under $10 per year.

  Proceeds to reimburse the government for bank bailouts.

• [SKR]

  I highly recommend the book Naked Economics.

  I might read it, but Chapter 3 “Government and the Economy: Government is your friend”, had better be satire.

• [Anonymous]

  I know we have all become almost fully desensitized to the evils of this world, but wake up folks, this is absolutely one of the worst evils taking place out there. Have we become blind to just how disgustingly out-of-control this greed is? How much we’re all being raped from it, especially the poorest in the world. I think we’ve become braindead, lusting after these stupid iPhones as their toxic components pollute every last corner (think about the sales figures for any device over the past decade — 90% of those precious ‘units’ are now in a landfill!) HFT made $21 billion last year ($666 a second, haha) — skimmed right off the top. As Hunter Thompson says, it’s a Nixonian cleverness, shitting in your own nest. Meaning it’s only an increasingly messy situation for all of us. Real smart.

  Mark my words: As long as billions of people don’t have clean drinking water or adequate food, this world will continue to be a hell! “LIFE IS SUFFERING”

  As Agent Smith in the Matrix says, human beings are the disease, plain & simple. Now go play with your electronic gizmo du jour, and as Rage Against the Machine says, “Forget about yourselves, and forget about your culture, and forget about your history and just buy. JUST BUY!”

  You know what you doing. For great justice, baby.

• [IWood]

  #20 posted by Caroline:

  Iwood, if you bought in at the beginning of April, you’ve just been riding a short-term trend up. I started my IRA around the same time and have seen 30%+ gains, but that’s so short term that I consider it meaningless. I don’t intend to touch that money for 40 years, so what happens over one summer isn’t a big thing.

  Over the long term, talking decades, you’ll do about as well as the market as a whole. Market timing is a fool’s game.

  I highly recommend the book Naked Economics.

  I bought stock in a molybdenum mining company that (as of this afternoon) is up 228%, a wind energy ETF that’s up 18%, and a nanotech-focused venture capital firm that’s up 59%. The total portfolio value has doubled, but obviously that’s mostly due to one good pick. And of course these gains are paper gains, but that’s only a concern if you’re *actually* trying to time the market. I’m not. I’m just buying cheap. I’ll sell when and if I need to, not before.

  The two companies are carrying almost no debt, the ETF is a trend rider (alternative energy is probably going to be the next bubble) composed of a broad range of wind energy-related companies (turbine manufacture, generation, and distribution), all of which remain down. Stop-loss orders are in at purchase price plus a a couple of percentage points, so I won’t lose any money if and when things tank again this Fall. And if they do, I will happily sell what I’ve got and buy more of the exact same stocks at cheaper prices, or keep what I’ve got and buy more of the same at slightly above my original buy-in, then watch them do what they’ve just done over the next 3-5 years instead of three months.

  My 401(K) existed before the bubble popped, and I dumped a bit more money into a mix of domestic and foreign index funds immediately after the pop. I don’t pay attention to it. In a decade or so when I’ve recouped the losses from the pre-bubble funds, I’ll move them into dividend-paying stocks or interest-protected bonds.

  My long term plays (i.e., 401(k))will match the market. My short term plays are based on buying well-managed, low- or zero-debt companies that make things people need or make things that people need to make *other* things that people need (e.g., you can’t make steel without molybdenum). My most speculative play is in something I think is cool (nano).

  If lose any of these bets it will mean that a) no one needs any more steel b) no one needs wind energy and c) no one believes that any further nanotech research is needed. At that point, my main concerns will mostly likely be canned food and clean water.

  You either actually believe that civilization is about to collapse and capitalism is over, or you believe that in five years, a decade, or two, people will readjust and move on. If you buy mostly practical things when they’re cheap and undervalued, as opposed to ephemeral crap that contributes nothing and only offers the shiny plastic prospect of illusory profit, you’ll do OK. I’m scraping together a little cash in anticipation of a Fall price collapse to snap up a few shares of a well-run company that makes (among many other useful things) low-friction bearings used in turbines and motors (as in: wind generators and electric car motors).

  It’s not about timing the market. It’s about maintaining a base of up-to-date research, identifying value, and making educated decisions.

• [mdh]

  the bubble gets twice as large every 18 months then?