The Success Equation: Untangling Skill and Luck

November 6, 2022

Michael Mauboussin wrote a great book called The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing (Harvard Business Press, 2012).

Here’s an outline for this blog post:

    • Understand were you are on the skill-luck continuum
    • Assess sample size, significance, and swans
    • Always consider a null hypothesis
    • Think carefully about feedback and rewards
    • Make use of counterfactuals
    • Develop aids to guide and improve your skill
    • Have a plan for strategic interactions
    • Make reversion to the mean work for you
    • Know your limitations

 

UNDERSTAND WHERE YOU ARE ON THE SKILL-LUCK CONTINUUM

If an activity is mostly skill or mostly luck, it’s generally easy to classify it as such.  But many activities are somewhere in-between the two extremes, and it’s often hard to say where it falls on the continuum between pure skill and pure luck.

An activity dominated by skill means that results can be predicted reasonably well.  (You do need to consider the rate at which skill changes, though.)  A useful statistic is one that is persistent—the current outcome is highly correlated with the previous outcome.

An activity dominated by luck means you need a very large sample to detect the influence of skill.  The current outcome is not correlated with the previous outcome.

Obviously the location of an activity on the continuum gives us guidance on how much reversion to the mean is needed in making a prediction.  In an activity that is mostly skill, the best estimate for the next outcome is the current outcome.  In an activity that is mostly luck, the best guess for the next outcome is close to the base rate (the long-term average), i.e., nearly a full reversion to the mean.

Our minds by nature usually fail to regress to the mean as much as we should.  That’s because System 1—the automatic, intuitive part of our brain—invents coherent stories based on causality.  This worked fine during most of our evolutionary history.  But when luck plays a significant role, there has to be substantial reversion to the mean when predicting the next outcome.

 

ASSESS SAMPLE SIZE, SIGNIFICANCE, AND SWANS

Even trained scientists have a tendency to believe that a small sample of a population is representative of the whole population.  But a small sample can deviate meaningfully from the larger population.

If an activity is mostly skill, then a small sample will be representative of the larger population from which it is drawn.  If an activity is mostly luck, then a small sample can be significantly different from the larger population.  A small sample is not reliable when an activity is mostly luck—we need a large sample in this case in order to glean information.

In business, it would be an error to create a sample of all the companies that used a risky strategy and won, without also taking into account all the companies that used the same strategy and lost.  A narrow sample of just the winners would obviously be a biased view of the strategy’s quality.

Also be careful not to confuse statistical significance with economic significance.  Mauboussin quotes Deirdre McCloskey and Stephen Ziliak: “Tell me the oomph of your coefficient; and do not confuse it with mere statistical significance.”

Lastly, it’s important to keep in mind that some business strategies can produce a long series of small gains, followed by a huge loss.  Most of the large U.S. banks pursued such a strategy from 2003-2007.  It would obviously be a big mistake to conclude that a long series of small gains is safe if in reality it is not.

Another example of ignoring black swans is Long-Term Capital Management.  The fund’s actual trades were making about 1% per year.  But LTCM argued that these trades had infintessimally small risk, and so they levered the trades at approximately 40:1.  Many banks didn’t charge LTCM anything for the loan because LTCM was so highly regarded at the time, having a couple of Nobel Prize winners, etc.  Then a black swan arrived—the Asian financial crisis in 1998.  LTCM’s trades went against them, and because of the astronomically high leverage, the fund imploded.

 

ALWAYS CONSIDER A NULL HYPOTHESIS

Always compare the outcomes to what would have been generated under the null hypothesis.  Many streaks can easily be explained by luck alone.

Mauboussin gives the example of various streaks of funds beating the market.  Andrew Mauboussin and Sam Arbesman did a study on this.  They assumed that the probability a given fund would beat the S&P 500 Index was equal to the fraction of active funds that beat the index during a given year.  For example, 52 percent of funds beat the S&P 500 in 1993, so the null model assigns the same percentage probability that any given fund would beat the market in that year.  Mauboussin and Arbesman then ran ten thousand random simulations.

They determined that, under the null model—pure luck and no skill—146.9 funds would have a 5-year market-beating streak, 53.6 funds would have a 6-year streak, 21.4 funds would have a 7-year streak, 7.6 funds would have an 8-year streak, and 3.0 funds would have a 9-year streak.  They compared these figures to the actual empirical frequencies:  206 funds had 5-year streaks, 119 had 6-year streaks, 75 had 7-year streaks, 23 had 8-year streaks, and 28 had 9-year streaks.

So there were many more streaks in the empirical data than the null model generated.  This meant that some of those streaks involved the existence of skill.

 

THINK CAREFULLY ABOUT FEEDBACK AND REWARDS

Everybody wants to improve.  The keys to improving performance include high-quality feedback and proper rewards.

Only a small percentage of people achieve expertise through deliberate practice.  Most people hit a performance plateau and are satisfied to stay there.  Of course, for many activities—like driving—that’s perfectly fine.

The deliberate practice required to develop true expertise involves a great deal of hard and tedious work.  It is not pleasant.  It requires thousands of hours of very focused effort.  And there must be a lot of timely and accurate feedback in order for someone to keep improving and eventually attain expertise.

Even if you’re not pursuing expertise, the keys to improvement are still focused practice and high-quality feedback.

In activities where skill plays a significant role, actual performance is a reasonable measure of progress.  Where luck plays a strong role, the focus must be on the process.  Over shorter periods of time—more specifically, over a relatively small number of trials—a good process can lead to bad outcomes, and a bad process can lead to good outcomes.  But over time, with a large number of trials, a good process will yield good outcomes overall.

The investment industry struggles in this area.  When a strategy does well over a short period of time, quite often it is marketed and new investors flood in.  When a strategy does poorly over a short period of time, very often investors leave.  Most of the time, these strategies mean revert, so that the funds that just did well do poorly and the funds that just did poorly do well.

Another area that’s gone off-track is rewards for executives.  Stock options have become a primary means of rewarding executives.  But the payoff from a stock option involves a huge amount of randomness.  In the decade of the 1990’s, even poor-performing companies saw their stocks increase a great deal.  In the decade of the 2000’s, many high-performing companies saw their stocks stay relatively flat.  So stock options on the whole have not distinguished between skill and luck.

A solution would involve having the stock be measured relative to an index or relative to an appropriate peer group.  Also, the payoff from options could happen over longer periods of time.

Lastly, although executives—like the CEO—are much more skillful than their junior colleagues, often executive success depends to a large extent on luck while the success of those lower down can be attributed almost entirely to skill.  For instance, the overall success of a company may only have a 0.3 correlation with the skill of the CEO.  And yet the CEO would be paid as if the company’s success was highly correlated with his or her skill.

 

MAKE USE OF COUNTERFACTUALS

Once we know what happened in history, hindsight bias naturally overcomes us and we forget how unpredictable the world looked beforehand.  We come up with reasons to explain past outcomes.  The reasons we invent typically make it seem as if the outcomes were inevitable when they may have been anything but.

Mauboussin says a good way to avoid hindsight bias is to engage in counterfactual thinking—a careful consideration of what could have happened but didn’t.

Mauboussin gives an example in Chapter 6 of the book: MusicLab.  Fourteen thousand people were randomly divided into 8 groups—each 10% of the total number of people—and one independent group—20% of the total number of people.  There were forty-eight songs from unknown bands.  In the independent group, each person could listen to each song and then decide to download it based on that alone.  In the other 8 groups, for each song, a person would see how many other people in his or her group had already downloaded the song.

You could get a reasonable estimate for the “objective quality” of a song by looking at how the independent group rated them.

But in the 8 “social influence” groups, strange things happened based purely on luck—or which songs were downloaded early on and which were not.  For instance, a song “Lockdown” was rated twenty-sixth in the independent group.  But it was the number-one hit in one of the social influence worlds and number forty in another.

In brief, to maintain an open mind about the future, it is very helpful to maintain an open mind about the past.  We have to work hard to overcome our natural tendency to view what happened as having been inevitable.  System 1 always creates a story based on causality—System 1 wants to explain simply what happened and close the case.

If we do the Rain Dance and it rains, then to the human brain, it looks like the dance caused the rain.

But when we engage System 2 (the logical, mathematical part of our brain)—which requires conscious effort—we can come to realize that the Rain Dance didn’t cause the rain.

 

DEVELOP AIDS TO GUIDE AND IMPROVE YOUR SKILL

Depending on where an activity lies on the pure luck to pure skill continuum, there are different ways to improve skill.

When luck predominates, to improve our skill we have to focus on learning the process for making good decisions.  A good process must be well grounded in three areas:

  • analytical
  • psychological
  • organizational

In picking an undervalued stock, the analytical part means finding a discrepancy between price and value.

The psychological part of a good process entails an identification of the chief cognitive biases, and techniques to mitigate the influence of these cognitive biases.  For example, we all tend to be wildly overconfident when we make predictions.  System 1 automatically makes predictions all the time.  Usually this is fine.  But when the prediction involves a probabilistic area of life—such as an economy, a stock market, or a political situation—System 1 makes errors systematically.  In these cases, it is essential to engage System 2 in careful statistical thinking.

The organizational part of a good process should align the interests of principals and agents—for instance, shareholders (principals) and executives (agents).  If the executives own a large chunk of stock, then their interests are much more aligned with shareholder interests.

Now consider the middle of the continuum between luck and skill.  In this area, a checklist can be very useful.  A doctor caring for a patient is focused on the primary problem and can easily forget about the simple steps required to minimize infections.  Following the suggestion of Peter Pronovost, many hospitals have introduced simple checklists.  Thousands of lives and hundreds of millions of dollars have been saved, as the checklists have significantly reduced infections and deaths related to infections.

A checklist can also help in a stressful situation.  The chemicals of stress disrupt the functioning of the frontal lobes—the seat of reason.  So a READ-DO checklist gets you to take the concrete, important steps even when you’re not thinking clearly.

Writes Mauboussin:

Checklists have never been shown to hurt performance in any field, and they have helped results in a great many instances.

Finally, anyone serious about improving their performance should write down—if possible—the basis for every decision and then measure honestly how each decision turned out.  This careful measurement is the foundation for continual improvement.

The last category involves activities that are mostly skill.  The key to improvement is deliberate practice and good feedback.  A good coach can be a great help.

Even experts benefit from a good coach.  Feedback is the single most powerful way to improve skill.  Being open to honest feedback is difficult because it means being willing to admit where we need to change.

Mauboussin concludes:

One simple and inexpensive technique for getting feedback is to keep a journal that tracks your decisions.  Whenever you make a decision, write down what you decided, how you came to that decision, and what you expect to happen.  Then, when the results of that decision are clear, write them down and compare them with what you thought would happen.  The journal won’t lie.  You’ll see when you’re wrong.  Change your behavior accordingly.

 

HAVE A PLAN FOR STRATEGIC INTERACTIONS

The weaker side won more conflicts in the twentieth century than in the nineteenth.  This is because the underdogs learned not to go toe-to-toe with a stronger foe.  Instead, the underdogs pursued alternative tactics, like guerrilla warfare.  If you’re an underdog, complicate the game by injecting more luck.

Initially weaker companies almost never succeed by taking on established companies in their core markets.  But, by pursuing disruptive innovation—as described by Professor Clayton Christensen—weaker companies can overcome stronger companies.  The weaker companies pursue what is initially a low-margin part of the market.  The stronger companies have no incentive to invest in low-margin innovation when they have healthy margins in more established areas.  But over time, the low-margin technology improves to the point where demand for it increases and profit margins typically follow.  By then, the younger companies are already ahead by a few of years, and the more established companies usually are unable to catch up.

 

MAKE REVERSION TO THE MEAN WORK FOR YOU

Mauboussin writes:

We are all in the business of forecasting.

Reversion to the mean is difficult for our brains to understand.  As noted, System 1 always invents a cause for everything that happens.  But often there is no specific cause.

Mauboussin cites an example given by Daniel Kahneman: Julie is a senior in college who read fluently when she was four years old.  Estimate her GPA.

People often guess a GPA of around 3.7.  Most people assume that being precocious is correlated with doing well in college.  But it turns out that reading at a young age is not related to doing well in college.  That means the best guess for the GPA would be much closer to the average.

Mauboussin adds:

Reversion to the mean is most pronounced at the extremes, so the first lesson is to recognize that when you see extremely good or bad results, they are unlikely to continue that way.  This doesn’t mean that good results will necessarily be followed by bad results, or vice versa, but rather that the next thing that happens will probably be closer to the average of all things that happen.

 

KNOW YOUR LIMITATIONS

There is always a great deal that we simply don’t know and can’t know.  We must develop and maintain a healthy sense of humility.

Predictions are often difficult in many situations.  The sample size and the length of time over which you measure are essential.  And you need valid data.

Moreover, things can change.  If fiscal policy has become much more stimulative than it used to be, then bear markets may—or may not—be shallower and shorter.  And stocks may generally be higher than previously, as Ben Graham pointed out in a 1963 lecture, “Securities in an Insecure World”: http://jasonzweig.com/wp-content/uploads/2015/04/BG-speech-SF-1963.pdf

If monetary policy is much more stimulative than before—including a great deal of money-printing and zero or negative interest rates—then the long-term average of stock prices could conceivably make another jump higher.

The two fundamental changes just mentioned are part of why most great value investors never try to time the market.  As Buffett has said:

  • Forecasts may tell you a great deal about the forecaster;  they tell you nothing about the future.
  • I make no effort to predict the course of general business or the stock market.  Period.
  • I don’t invest a dime based on macro forecasts.

Henry Singleton—who has one of the best capital allocation records of all time—perhaps put it best:

I don’t believe all this nonsense about market timing.  Just buy very good value and when the market is ready that value will be recognized.

 

BOOLE MICROCAP FUND

An equal weighted group of micro caps generally far outperforms an equal weighted (or cap-weighted) group of larger stocks over time.  See the historical chart here:  https://boolefund.com/best-performers-microcap-stocks/

This outperformance increases significantly by focusing on cheap micro caps.  Performance can be further boosted by isolating cheap microcap companies that show improving fundamentals.  We rank microcap stocks based on these and similar criteria.

There are roughly 10-20 positions in the portfolio.  The size of each position is determined by its rank.  Typically the largest position is 15-20% (at cost), while the average position is 8-10% (at cost).  Positions are held for 3 to 5 years unless a stock approaches intrinsic value sooner or an error has been discovered.

The mission of the Boole Fund is to outperform the S&P 500 Index by at least 5% per year (net of fees) over 5-year periods.  We also aim to outpace the Russell Microcap Index by at least 2% per year (net).  The Boole Fund has low fees.

If you are interested in finding out more, please e-mail me or leave a comment.

My e-mail: jb@boolefund.com

Disclosures: Past performance is not a guarantee or a reliable indicator of future results. All investments contain risk and may lose value. This material is distributed for informational purposes only. Forecasts, estimates, and certain information contained herein should not be considered as investment advice or a recommendation of any particular security, strategy or investment product. Information contained herein has been obtained from sources believed to be reliable, but not guaranteed. No part of this article may be reproduced in any form, or referred to in any other publication, without express written permission of Boole Capital, LLC.