This was an interesting paper shared by Ethan Mollick on innovation experimentation. Larger scale A/B tests are often done (notably in tech businesses but in many other sectors as well, particularly given the proliferation of services) to screen ideas. But with limited resources to allocate to experimentation, should a business concentrate on testing a few ideas with large scale tests or actually consider running a broader range of smaller scale tests to increase the chances of discovering potential outlier or breakthrough ideas?
The researchers looked at this question, using experiments conducted by Microsoft Bing. Like many businesses, the Bing team were running a smaller number of large scale experiments but the researchers also considered the potential for a ‘fat tail’ of ideas vs impact – accounting for the possibility of outlier ideas with unexpectedly significant impacts. The study observed a very ‘fat tail’, meaning that changing innovation strategy to run many more small-scale experiments looking for outlier levels of impact (like a start-up would) would significantly increase innovation productivity.
The researchers concluded that even beyond a digital-based scenario, ‘fat-tails’ can be important to optimise learning, but that context is all important in defining an appropriate innovation strategy.
‘The key insight of the model is that the optimal experimentation strategy depends on whether most gains accrue from typical innovations, or from rare and unpredictable large successes that can be detected using tests with small samples.’
In my books I talk about the idea of the ambidextrous organisation that is able to exploit existing advantage whilst continuously exploring new territories or models. To be truly ambidextrous a business needs to get good at both optimisation, and breakthrough innovation (or transformation). It seems that many businesses limit the scope and number of experiments which, as the research suggests, may work fine in cases where the most gains come from ‘typical’ innovations. But limiting experimentation in this way risks missing out on the significant value that can often be created by newer, outlier ideas (at Bing, just the top 2% of ideas accounted for 75% of the gains). Understanding how fat the tail of ideas vs impact really is can transform the effectiveness of organisational experimentation. As Ethan notes: ‘When tails are thin: “perform thorough prior screening of potential innovations & run a few high-powered precise experiments” Where tails are thick: “run many small experiments, and test a large number of ideas in hopes of finding a big winner”’.
Photo by Alex Kondratiev on Unsplash
Leave a Reply