Buyer Beware

You might be interested in using a quantitative approach for your investing but you do not have the background or the time to develop your own quantitative strategies.  If you do an online search you will come across many third party solutions that seem to address your needs.

Can I Trust A Black Box System?

As I mentioned in another post, the investment arena is fraught with tricksters and scammers trying to get hold of your monies. Knowing how to find a good strategy and vendor whom you can trust with your monies can be a challenge since the pitfalls might not be obvious.  Having been in the quantitative investment business for many years I’ve learnt to see beyond the surface and spot the dangers in advance. Here are some warning signs that you need to look out for:

  1. Fool’s Gold.  This is when a vendor shows historical results and makes claims with no independent third party verification to substantiate those claims.  In today’s world anybody can publish anything, including studies, endorsements and test results.  How can you trust any of it if you have no way of verifying any of it?
  2. The Mirage.  This is when the trades are presented individually and not as a strategy or a portfolio with a measure of the average annual return and also the maximum loss over a period of time or drawdown.  Not knowing what the historical drawdown was is like driving a car with your eyes closed.  Sooner or later the road is going to curve and you are going to drive straight into a wall.
  3. The Perfect Fit.  This is when a developer, intentionally or unintentionally, uses too many parameters in the system development.  As discussed in the post The Quantitative Process Part I, given enough parameters one can make any data set fit.  Although the numbers look great they are meaningless.
  4. The Fancy Lingo.  A lot of times you will come across fancy terminology such as out of sample testing, Montecarlo simulation and walk forward analysis that seem to highlight the scientific rigor of the process.  Although these tests are commonly accepted as the antidote to The Perfect Fit problem listed above, they’re not all they are cracked out to be.  They are indeed helpful to verify the robustness of a system but they are by no means sufficient.  A system can pass all these tests and still fail miserably.
  5. The Shotgun Approach.  This is when a developer lists a substantial amount of strategies in the hope that some of them will have a good run and make it to the top. By virtue of luck alone, if you have enough strategies, then one or more of them are going to hit a suitable market and have a good run.  The problem with this approach is that there is no merit to the strategy and you might as well put money blindly in several different markets.  You never know which of the strategies to trust and if you split your monies among all of them you will end up in the red.
  6. The First Place Finisher.  This is when you come across a strategy or a developer that placed well in an investing or trading competition.  A strategy that wins a trading competition is trying to make the most amount of money possible in a pre-defined short period of time.  On the other hand, as an investment strategy you want one that makes money consistently over a long period of time.  In addition, many trading competitions allow competitors to use the shotgun approach just mentioned above.  Thus luck plays a much bigger role in competitions and this is not suitable for an investment portfolio.
  7. The Bait and Switch.  This is when a developer is running the shotgun approach in the background and then constantly updating or changing the presented strategies such that the historical performance looks great.
  8. The Theoretical Results.  If a developer does not include the full costs of commission, slippage and any other fees then the strategy results usefulness is limited.  Real world application costs can easily turn a winning theoretical concept into a looser as discussed in The Quantitative Process Part II .
  9. The Iceberg.  Last but not least, one very common problem with a lot of published strategies is the very short performance history.  It is much easier to find a strategy that does well over a short period of time.  However in general, the shorter the time period, the less likely the strategy is to continue to deliver into the future.

Why Quantitative Strategies Fail

In the post The Quantitative Process : Part II we saw an example where a simple quantitative strategy stopped working after performing well for more than 45 years.  Fundamental market changes like this example do happen.  However,  in my opinion, the main reason why the majority of published quantitative strategies fail is because when developing the strategy a lot of vendors have, intentionally or unintentionally, not used a sufficiently large sample of data that tests the performance over different market conditions.  Thus, as discussed in the post A Strategy For All Markets, when markets conditions change the strategy breaks down.

Changing market conditions not only expose a strategy’s lack of fundamental basis but they shine a light on the hidden part of the iceberg and expose over-fitting and the strategy’s true poor statistical characteristics.  Sophisticated statistical approaches such as Monte Carlo simulations and out of sample testing are useless if the sample size is small.  Thus the astute investor should not give much regard to strategies showing great or excellent results over a couple of months or even a couple of years.  Over such a short period one cannot discriminate between a solid strategy and a frail one having a lucky run.

The Road To Success

Strategy research and development is a tedious and arduous process as most roads lead to nowhere.  Only a very few strategies have a real expectation to generate profits.  Most of these viable strategies are never published.  Thus finding a good useful strategy can  be like finding the needle in a haystack.  In the post What makes a good strategy? I will offer some suggestions of how to go about finding a good strategy and avoid the pitfalls mentioned above.

Leave a Reply

Your email address will not be published. Required fields are marked *