Will Big Data Destroy the Stock Market?

When you buy a stock, you place a bet on how that stock will perform in the future. In a perfect world, where market insiders and manipulators are removed from the equation, the market is a terrific tool for determining the true value of companies being invested in.

But what happens when the volume of data used to make decisions increases 100 million times, and trading volumes increase 100 million times, and trades can be transacted over 100 million times a second?

The proliferation of mobile phones, social media, machine data, and web logs has led to massive amounts of data being generated, stored, and processed, and this volume is increasing exponentially with the digital shift from offline to online. Inside these rapidly expanding data pools are millions of tiny little “tells” that can be extracted and combined with the emerging science of anticipatory computing into very predictable movement indicators.

At the same time, these “tells” can be fabricated, manipulated, and auto-generated. In less time than it takes for a human to blink an eye, the entire value of the markets can fluctuate over $100 trillion and back again without leaving any humanly understandable trace of what just happened.

How long will it be before a series of trading algorithms go terribly wrong, causing the markets to simply implode? Here are a couple thoughts.

Moving into the Big Data Era

First a few jaw dropping stats on the growth of structured and unstructured data. Here are a few examples:

• RFID (radio frequency ID) systems generate up to 1,000 times the data of conventional bar code systems.
• Smart dust products and similar “Internet of Things” devices connected to the Internet will reach 50 billion by 2020, each with its own 24/7 data generating capability.
• 10,000 payment card transactions are made every second around the world.
• Wal-Mart averages more than 1 million customer transactions every hour.
• 340 million tweets are sent per day. That’s nearly 4,000 tweets per second.
• More than 5 billion people are calling, texting, tweeting and browsing websites on mobile phones.
• 247 billion emails are sent every day (80% is spam!)
• It now only costs $600 to buy a disk drive large enough to store all of the world’s music.
• 30 billion pieces of content are shared on Facebook every month.
• Projected growth in global data generated annually is 40%. By 2020, the production of data will be 44 times greater than what we produced in 2009

Big Data is indeed …big! And getting bigger.

The Future Stock Market

When it comes to the stock market, humans are already the outsiders. The number of actual trades transacted by real humans is only a small fraction of a day’s total trading volume.

Yes, there are still people who buy stocks because they “like” the company, and yes, companies who are attempting to recruit key employees will still give stock options as part of their compensation package.

But for those who are serious traders, buying and selling stock on a daily basis, the game is constantly amping up to new levels where automated systems do most of the grunt work.

In a trash-talking society with quants boasting “my algorithm can beat your algorithm,” the entire stock exchange has deteriorated into a kind of bot vs. bot warfare that has lost all semblance of being a marketplace of human values.

One Possible Scenario

As a way to help you understand how this may unfold, here is a short scenario.

With the stock market, every trade creates a butterfly effect, a tiny ripple of influence, with the size of the ripple being directly proportional to the size of the trade. When trades happen at lightening speed, the size of the ripples can be manipulated through a series of external distortion signals designed to mislead all of the other bots, leading to ever-greater buy-sell patterns until they reach a point of maximum distortion for profit taking.

If, as example, a quant were to set up a specific trading pattern starting with 2 shares, followed over a set period of time by a series of doublings, 4 shares, then 8, 16, 32, 64, 128, 256, 512, and so on, the exponentially growing signal produces a “resonancing effect.”

Whenever a pattern like this is repeated multiple times, other bots will pick up on it, and start formulating strategies around it. For the puppet masters this becomes a relatively easy way to manipulate the market by first establishing a pattern, and then skipping or altering the final trade.

Much like the cracking of a whip, where all of the attention is focused on the final snap, the market positions itself for the grand finale. Predictability creates manipulability and its games like these that polymaths live for.

So what can be done to keep the destructive forces of our hacker culture at bay? Can the controllers and regulators ever outsmart the unleashers?

Reaching the Breaking Point – One Additional Scenario

People who are earning a living off the stock market have a vested interest in preserving it.

Even though the markets are being heavily abused by “the takers,” virtually all of the current players will go out of their way to keep the current system operational. No one wants to kill the goose that lays the golden eggs.

However, there are three key variables where complexity issues are spiraling out of control – volume, speed, and scale. With the size and scale of the problem growing exponentially faster than the regulatory ability of those trying to keep the genie in the box, given enough time, I would place the odds of total collapse at 100%.

So what does failure mean, and exactly what will it look like?

Consider this scenario.

The first of those to master the big data stock market era will become the wealthiest people on the planet in a very short period of time, literally over night. Their rise to prominence will be silent and without fanfare.

This new class of the super elite will quickly move to shore up their power by demanding prominent board positions in some of the world’s top corporations, and in short order, start converting public companies to private ones as a way to insure ongoing control.

Once a few major companies have been privatized, others will begin to panic and begin similar steps to isolate themselves from the public markets.

In just a few months we will see a massive migration towards privatization, with thousands of companies around the world seeking refuge from similar hijackings.

With huge numbers of businesses leaving the marketplace, this black swan event will cause an implosion of global stock exchanges.

So at what point do the existing stock exchanges begin to fail? To put it another way, at what point do the too-big-to-fail banks with all their hands in the cookie jars find it necessary to step in to try to regain control of a system that has gotten away from them?

The answer: When someone new comes along, a new breed of “big data super-quants,” that learns to play the game on a whole new level, leaving today’s existing power elite one step behind on the investing chessboard.

The initial battles will be over rule changes designed to keep “intruder techniques” at bay.

The system will be officially broken when people lose faith in its integrity. When the trust is gone, businesses will begin to abandon it.

So then what?

Given enough time, all industries will eventually end. When the stock market ends, how will it affect society?

The businesses that the market has been investing in will undoubtedly figure out other ways to compensate. The traders that made their living in the market will find themselves looking for new skills to do other work. And the news media that has dedicated unusual levels of attention to this high-end chess game will be forced to find real news to report on.

Final Thoughts

Admittedly, I’m not an expert on the stock market or the nuanced theories that govern the system. And I’m quite certain the scenarios above are overly simplistic and simply wrong.

But big data itself is a game changer, and once combined with the scaling of speed and volume, people heavily invested in today’s game will undoubtedly be blindsided by something new.

Much like adding steroids and performance enhancers to professional sports, the human vs. human marketplace for stocks is about to be overtaken by a wave of cyborg quants driving next generation supercomputers to play an entirely new game of Moneyball.

So how long before everything begins to unravel? I’m not sure of the year, but it will happen on the 2nd Wednesday after a full moon. In fact, I’m so certain of it, that I’m willing to bet all my Wachovia, Blockbuster Video, Enron, and Circuit City stock on it.