I’m often astonished at the extent to which people who run record companies can be bad at maths.

Yesterday I sat in a meeting where it was suggested that the success of a promotion might be measured be checking the sales of the records in it before, during and after the promotion. It’s not a totally barmy approach, but without comparing the sales of the records in the promotion with sales of some other records over the same period, any increase or decrease might be the result of normal sales patterns or overall fluctuations in the market.

Still, that’s not nearly as daft as this thing I found today. The headline is “Which music services are growing, which are shrinking” which would make for a pretty interesting article if it actually told you the answer to that question.*

It’s not easy to measure this unless you sell music on these all the services** so the author has instead used Google Trends to find out which services people are searching for. In the medical world, this is known as a surrogate outcome. The thing you want to measure is difficult to measure, so you measure something related instead. It can go wrong.

The trouble is that a related variable isn’t necessarily strongly correlated to the outcome you care about. If you’re gong to use a surrogate like this, it’s a good idea to check if it’s really easy to find examples of it not working at all well. The examples below are literally the first things I searched for.

According to Google Trends, Steve Jobs was never more in demand than the day he died, and is still not as unpopular as  when he was alive (that last bit might be true).

Perhaps its fairer to compare Google Trends with a better-known indicator of company performance: share price. Here, Berkshire Hathaway’s search popularity has been in steady decline for the best part of a decade.

Don’t tell their shareholders that, though. According to the NYSE, business has never been better.

Meanwhile, at Yahoo, search interest hit a high plateau in 2010:

And the stock price is doing almost the exact opposite.

Pretty much the entire foundation of all scientific progress is to come up with an idea and then try to prove it wrong, because if you can’t prove it wrong, you might just be on to something. If the author had made even a cursory attempt to check how poorly search popularity was correlated with company performance, I think he might have offered a stronger disclaimer than:

“Of course, these search trends are not the same as having an actual measure of activity. Millions of people play music on Spotify or iTunes every day without performing a search. However, until we can get raw user numbers from every music service, this is probably about the closest we can get to understanding which services are growing and which are shrinking.”

When I wrote “almost everything you read about the state of the record industry is, at best, totally useless” this is just the sort of thing I was talking about.

Footnotes:

* Perhaps I should avoid reading articles with questions in the headlines. If they ever answer the question, it’s because the answer is “probably not”. This is often the way with rhetorical questions.***

** If you don’t sell music, why do you even care which ones are growing?

*** And did those feet, in ancient time, walk upon England’s mountains green?**** And was the holy Lamb of God, On Englands pleasant pastures seen?**** And did the Countenance Divine, Shine forth upon our clouded hills?**** And was Jerusalem builded here, Among these dark Satanic Mills?****

**** Probably not.