Perhaps no-one has done more for the cause of data-driven decision-making in the minds of the public than Nate Silver. His book, The Signal and the Noise, explains how stat i st ica l modelling improves our predictions about everything from the weather to sports to the stock market. Data science is the hottest f ield to be in right now, and Silver is its poster child.
But for most people, the gulf between recognising the importance of data and actually beginning to analyse it is massive. How do those without extensive training in statistics equip themselves with the skills necessary to thrive (or even just survive) in our age of ‘ big data’?
Last month I had the chance to put that question to Silver, and his answers may surprise you. Far from counselling that everyone must major in statistics, in the edited conversation below he advises students and executives alike to roll up their sleeves – no matter their statistical literacy – and get their hands dirty with data.
I think the best training is almost always going to be hands-on training. In some ways the book is fairly abstract, partly because you’re trying to look at a lot of different fields. You’re trying not to make crazy generalisations across too many spheres.
But my experience is all working with baseball data, or learning game theory because you want to be better at poker, right? Or [you] want to build better election models because you’re curious and you think the current products out there aren’t as strong as they could be. So, getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth.
Again, I think the applied experience is a lot more important than the academic experience. It probably can’t hurt to take a stats class in university.
But it really is something that requires a lot of different parts of your brain. I mean the thing that’s toughest to teach is the intuition for what are big questions to ask – that intellectual curiosity. That bullsh*t detector, for lack of a better term, where you see a data set and you have at least a first approach on how much signal there is there. That can help to make you a lot more efficient.
That stuff is kind of hard to teach through book learning. So it’s by experience. I would be an advocate if you’re going to have an education, then have it be a pretty diverse education so you’re f lexing lots of different muscles.
You can learn the technical skills later on, and you’ll be more motivated to learn more of the technical skills when you have some problem you’re trying to solve or some financial incentive to do so. So, I think not specialising too early is important.
I mean my path has been kind of sui generis in some ways, right? Probably an online course could work, but I think actually when people are self-taught with occasional guidance, with occasional pushes here and there, that could work well.
An ideal situation is when you’re studying on your own and maybe you have some type of mentor who you talk to now and then. You should be alert that you’re going to make some dumb mistakes at first. And some will take a one-time correction. Others will take a lifetime to learn. But yes, people who are motivated on their own, I think, are always going to do better than people who are fed a diet of things.
Say an organisation brings in a bunch of ‘stat heads’, to use your terminology. Do you silo them in their own department that serves the rest of the company? Or is it important to make sure that every team has someone who has the ana-