Thursday, March 23, 2017

The significance of Statistical Significance

So you've started A/B testing and you're cranking in results. Great. Now what? 

You could just pick the better performer and start A/B testing off that version, but statistics gurus would tell you to grow up. Seriously. Grow up.  Now I'm pointing at myself here too! Read on, and we'll all grow up together.

I recently listened to the gurus and started growing up. "Just because you got more hits on B doesn't mean B is going to be statistically relevant and should change the course of your business!" the gurus said. (I paraphrased. And called the voices of amalgamated articles and writings in my head "gurus".)

"Ok fine." I responded. Let's check this out then. 

I read "Statistics for Dummies." I checked out countless articles on how to calculate. I tried pumping stuff in to spreadsheets. And, let's face it folks, I'm not a statistician. I'd argue I'm better than most at crunching data in Excel, Tableau and other tools, but I am not a died-in-the-wool data scientist. I'm guessing that you aren't either. 

One of my favorite tools I found was Hub Spot's terribly (well, mostly) simple explanation and statistical significance calculator

Statistical significance is easy to calculate
SO easy people. Seriously.

Well I'll be danged. Now when I say "this just isn't statistically relevant," I can actually back it up with math

And now I'm testing statistical significance like the guy from Beautiful Mind seeing math bouncing aroun everywhere. 

So go on. Grow up with me and use those tools above and check out the books below to get yourself up to speed on your A/B testing and statistical significance. 

No comments:

Post a Comment