What Happened When I Tried Market Prediction With ML

Right around the time you get your first basic regression or classification model going, it will at least cross your mind. The vast piles of time series data, coupled with the possibility of retiring young has the irresistible pull of finding an old treasure map in your grandfather’s attic. How can you NOT think about it? Can you use machine learning to predict the market?

There are plenty of small scales tutorials on the web that are a great place to start. They show you how to pull down the history of a stock, perhaps calculate a few indicators, and feed it to a regression algorithm and try to predict the next day’s value. Alternatively, they use a classifier to predict whether the stock will rise or fall, without predicting a value.

I had two ideas on where to go from here. First, I wanted to go bigger. I theorized that there might be hidden relationships between some stocks, currencies, and financial indicators that were just too subtle to be found by eye. I figured a machine learning algorithm might be able to pick them out.

Second, I wasn’t going to pick a stock that I wanted to predict. I was going to to train models for all of them, and see which stocks performed best. The idea was that some companies might be more predictable than others, so I needed to find them.

......