Analytics and perfection

Good is better than best

In a recent interview Rayid Ghani, Chief Scientist of the Obama Campaign 2012 was asked:

Q: Can you share some interesting examples of non-trivial analytics findings ? For example, it was reported that the most effective email subject line for Obama Campaign was “Hey”.

His answer might suprise you:

A: I think the key to the analytics operation was that we weren’t really going after ‘insights’ (but) focusing on operationalizing our models and analysis. What made us successful was that we embedded analytics in processes across the campaign and in tools that people were using to get their jobs done. By completing the ‘loop’ and increasing automation, we were able to deploy fairly sophisticated analytics that improved over time and were used by the campaign.

Think about that for a moment. Their strategy was to avoid seeking particularly novel solutions and instead spend resources integrating the solution into their business process, where it would be gradually improved over time.

Netflix Agrees

One prominent example of this is the $1m worth of research which Netflix chose not to put into production:

We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then.

You should be asking yourself how large an opportunity cost you are forgoing by not putting your good models into production.