The next gold rush in financial markets will be driven by Big Data

L S Subramanian

The next gold rush in Financial Markets will be driven by Big Data. The key driver for this is the low cost to create, manage, mine and deploy Big Data using the pay-as-you-use solutions in the cloud. Cloud computing has democratized Big Data usage and analytics.

The differentiators for the winners will be the traders who use Big Data analytics and algorithmic trading engines to churn profits from the markets.

All markets work on information whether it is from government, corporate, industry, social media, economical trends, sales figures, people, historical statistics and future projections and more. All this data -- both structured and unstructured will have to be brought together under one umbrella and made to work in tandem to provide analytics which can be piped into an algorithmic  engine which in turn will create the upside in the markets for the bold investors who get this technology usage right.

Here is a scenario on how could this happen. Let us consider the US Bond market trader who wants to use this combination of big data analytics and an algorithmic engine, wall street data over the years, data on government spending, SEC guidelines, the various bond guru prophesies, the news -- both digital and printed, plus the market prices and the historic data. This list is not complete and there are more pieces in this big puzzle including the various trading strategies that will need to be adopted.

The trader will have an almost infinite information set which needs to be mined with the right tools and analyzed to get the winning formula; we will need some of the best mathematicians and statisticians in our team to prioritize and map this data into the trading strategy and predicted outcome.

The trader will need Big Data to hold the humongous volume of structured and unstructured data and his analysts will need the ability to toss and turn the data. The trader will need a team to help managing this jumbo canister of digital data. There will be a need for a team to scrub and clean this data before it can be used by the customized tools for analyzing this data. The team has to be adept at building and deploying tools to clean and scrub data in real time.  There will be static data streams and real time data streams which have to be merged by the Big Data algorithms to give the trader an useful output to feed the algorithmic trading engine which will connect into the digital bonds market.

Remember all this activity of data production in the real market will happen in nano seconds or less in a fast moving and frenetic digital market and will be more complex in the equity and commodity markets which are like quicksilver. If the trader decided on hedging, cross border correlation with other markets and also within the domestic markets the data flows and information engines become more complex but the upside for the winner will be take it all.

How do we put a team together to make this happen? In my mind it will be a global project.  Big data and algo engines are best today in the USA. So, we will use the computing infrastructure in that country and also work in the markets in the USA. Asia could be the ideal location for data cleaning and preparation. It has a competitive and educated work force which has shown that it can deliver. India could be a leader in this space given its Y2k migration track record. But other countries like China, Korea are also excellent sources.  The algorithmic engines complemented with mathematics and statistical engines may come from the team in Eastern Europe who have wonderful teams who have made significant contributions in this area.

India could provide the expertise in building the solutions need for data analytics and algo trading engines and other related technology engineering. The funding for this gold rush would not be from the European nobility but from ordinary citizens who believe in the power of Big Data and its positive outcome.

Once we have the ingredients together, we will need this team of Americans, Asians, Europeans to be connected virtually to make our gold mining expedition a reality and to take on the global markets.

What will be the time needed for such a project? I predict that it may need not be more than three months if we use cloud computing.  We can start small and keep improving our Big Data analytical engine and our algo trading engine as we see an upside in our technology.  The downside could be that we lose our capital if this does not work.  That said, we only have to make a small investment to make on cloud computing, and with the right team there will be no looking back.

The biggest worry in this strategy is whether the global exchanges will be able to handle the tsunami created by the marriage of Big Data analytics and algo trading. The barrier for this project  could be the exchanges and regulators  who may play spoil sport and close this window of opportunity even before the party starts.

But the early bird catches the worm, so get your tools sharpened, put your tools together and start your gold mining expedition. If you are lucky you may even strike platinum. Markets are all about slices of opportunities like a surfer finding the perfect wave, so get your boards out and ride the waves of this new opportunity. Nerds can change the global markets and Big Data will be the technology engine that they can be harnessed to make the difference.

- The author , L S Subramanian, is CEO of NISE.  LS (as he is called) is also a respected thought leader known for his innovation in information technology for business. He is the architect of the OTC Exchange of India, India’s first online trading national stock exchange and has advised NCDEX, Power Exchange, CRISIL, Exchange Next  and other companies in trading solutions and processes.